Our Data Journey to Serve Children in Poverty and Double Our Impact
Like many organizations, Compassion International, has dozens of data sources and thousands of data consumers eager to get answers to their business questions.
The difference with Compassion International is that good data directly impacts how well we serve children in poverty around the world. Compassion partners with over 7,000 churches in 26 countries to deliver a holistic child development program to over 1.9 million babies, children and young adults. This approach includes providing for the physical, educational, social and spiritual care of our beneficiaries. Our data has many uses in helping us to achieve these outcomes.
Data Virtualization in the Service of Children in Poverty
In 2004, we were serving over 500,000 children. In 2009, this grew to 1 million. Earlier this decade a vision was put forth to minister to 4 million children in the near future. As we march toward this goal, our data needs have increased significantly and range from consistent enterprise reporting on one end to rapid answers with high agility on the other. Our goal is to enable quick answers and harmonious stories – all rooted in meaningful, consistent data.
Our initial approach was to use traditional, circa-2008 data integration methods. It quickly became apparent that there were several problems to this approach, including:
- The brittleness of ETL routines that are difficult to change
- Loss of visibility into business rules
- Unreliability because of load dependencies (e.g. one load depends on another load)
With the help of some answers to prayer and strategic relationships we started a journey towards a virtual enterprise single source of truth. It began with a prototype that gave a quick win in three weeks that solved a problem we’d been struggling with for over a year. In 2011, the doors really began to open as we revamped to an Agile development process and started incorporating enterprise-defined, core business entities into our logical data warehouse. At the heart of the new architecture was Cisco Data Virtualization. We worked closely with our Enterprise Information Management (EIM) group, and a series of cross-functional business teams, to revamp our Ministry Information Library (MIL) onto the new virtualized platform.
At Compassion, we strive to make our users’ access to the information they need as easy as possible. We hide complexity by abstracting and federating the source data, and making the data sources independent of data consumers. As a corollary to this, we also provide (secured) access to the raw data for full transparency and lineage.
With data virtualization, we’re able to ensure that definitions and business rules are applied consistently to improve information quality. (No more “where did you get your numbers?”) This also allows us to surface data at any level of latency required by the customers – ranging from near real-time to daily, weekly, monthly and yearly.
The overall architecture can be summed up in the acronym ELvT – Extract, Load, virtually Transform. ELvT is based on three principles:
- EL: Leave data where it is or copy it with minimal (preferably no) transformation. This allows us to leverage proven technologies such as replication, etc. The goal is dial-tone loads.
- vT: Transform/conform the data virtually in views (not ETL routines). The goal is agreement with agility.
- Minimize performance tuning by investing in good hardware. The goal is for people’s creative gifts to solve business problems not performance problems.
One of the most powerful features of an information factory based on virtualization is providing data in consumer-agnostic way. This frees business users to use whatever tools make sense for their data needs. “Any read only need at any latency” is the mantra.
Another benefit is the reduction of spreadmarts. Instead of having data scavengers trying to scrounge data from anywhere they can get it, we bring it all together into areas called “BI workspaces.” These workspaces include secure access to conformed data, raw data, and the ability to augment with “BYOD – bring your own data.” The combination of these three becomes a powerful information force in the hands of business users, data scientists and other consumers.
From an IT perspective, we simplified data integration – making us more responsive to new requests. By standardizing our enterprise business logic and terms, with views instead of ETL, we are able to simplify maintenance and provide significant agility in new solution development.
Furthermore, we greatly improved data quality and a shared consciousness of trust and transparency. With data freely flowing to our customers, we are actively helping our organization’s goal of serving four million children in the near future.
In summary, great data is not the end game. For Compassion International, it represents one of many means to a much more significant goal of “Releasing Children from Poverty in Jesus’ Name”. . We invite you to join with us and change the life of one child.