By Dr. Teri Behrens, director of special projects and editor in chief of The Foundation Review at the Johnson Center for Philanthropy
Big data is everywhere. The explosion of tools for analyzing massive amounts of data and displaying it in visually appealing ways is revolutionizing our ability to make data-informed decisions. This is great news, right? Well, maybe not completely. The downside is that easy access to data can lead to its misuse. A recent op-ed in the New York Times highlighted some major concerns about big data.
In the philanthropic sector, there’s a particular problem when we try to combine the push to measure outcomes with the accessibility of big data. The temptation is to use those readily accessible population level datasets as outcome measures for philanthropic efforts. If you are a large foundation trying to reduce health care costs, why not use the rate of emergency room visits to measure the success of your portfolio of grants? The temptation is to use the data that’s already there — at no additional evaluation cost! — to measure outcomes.
Stick with me on this… A teen was caught on video urinating into a Portland water reservoir, prompting the city, eager to assure the public about the quality of their water, to drain 38 million gallons of water from it.
Slate Magazine did an analysis of how much urine would have to be added to the reservoir to make the water unsafe. Their conclusion: the teen would have to urinate nonstop for 44 days before the levels of nitrates in the water would reach potentially harmful levels.
How many emergency room visits would a clinic have to prevent before the rate of emergency room usage in a city showed a decline? More than most foundations are able to fund or any clinic could carry out.
On the flip side, I heard of one organization that hoped to combine a number of state-level indicators of citizens’ wellbeing and to track improvements as a way of measuring the impact of foundations. What’s wrong with this? These outcomes (health, education, etc.) are complex and are influenced by many factors. Improving economic conditions will have a positive impact on many of these indicators of wellbeing — regardless of whether foundations are even doing grantmaking or other work in that area! Without being able to make the logical connection between effort and outcomes, big data is no help in assessing overall foundation impact.
On a final note, it is also important to remember where big data comes from — lots of little data, aggregated. As in cooking, the quality of big data is only as good as the quality of the ingredients. Anyone who has tried to do research using 990 forms as a data source, understands this issue. In cases where big data IS appropriate, it is always important to understand the underlying data sources and how reliable and valid they are.
So, go ahead and use big data. But use it responsibly.