Valuing What We Can Measure or Measuring What We Value?

(Michael Edwards is a leading expert on global civil society and the author of Small Change: Why Business Won't Save the World. This is the third in a series of posts in which he looks at different aspects of the Bellagio Initiative, an effort funded by the Rockefeller Foundation to produce a new framework for philanthropic and international development collaboration in pursuit of human well-being. Click here to read the first post and here to read the second. To read/download the Bellagio paper on which these posts are based --and from which the quotation below is taken -- click here.)

"There are no data or moral arguments to prove that one of these approaches is better than the others, so when interrogating the effectiveness of foundations the key question is always effective in doing what -- by what criteria, over which timescale, and who decides?..."

It is often said that we value what we can measure, but it might be more accurate to say that we measure what we value, and different people value very different things, an insight that is central to debates about the meaning of development and the need for broader indicators such as well-being. What, how and when we measure, how we use and interpret the results, and who decides these things are crucial questions in development -- and serve to highlight not just the methods, but the politics and power relations involved.

Do we measure long-term systems change as well as the numbers of lives saved and jobs created in a two- or three-year project cycle? Is democracy calculated as a cost or a benefit when we want to fix problems quickly and "get things done"? Should education be evaluated using test scores or the skills we need to live well together? As Robert Chambers, a modern icon of development research, puts it, "Whose reality counts?" Outside the science laboratory, all measurement means negotiation between different views and voices.

In the development sector, this debate has been accepted as legitimate for many years (though it remains to be resolved) -- perhaps because the exploitation of poor countries, their experience of colonialism, and/or their greater reliance on outside assistance from donor agencies have placed power at the heart of any meaningful analysis of what to do. It's a conversation that has generated whole new philosophies of knowledge and methods of participatory research to explore alongside randomized control trials and other quantitative techniques. Some have called this a shift from "using data" to "cultivating wisdom," by which they mean the ability to use information of different kinds to foster collective action and analysis, and to "co-produce" a vision of what matters most with those who are being "developed" at the center of that vision.

In philanthropy, by contrast, those who challenge the assumptions embedded in measurement and data-driven change are often labeled as idiots or reactionaries, guilty of the "hubris that measurement is the technocratic tool of the devil," as Fay Twersky from the Hewlett Foundation puts it in a recent op-ed in the Chronicle of Philanthropy. No one I know actually holds this view, and attacks like these do little except distract us from the need to ask tough questions about the influence of measurement over the kinds of work we prioritize for support.

The lesson to be learned from development is that opening ourselves to these questions is the best way to find a reasonable path forward -- a path that shifts the conversation away from the belief that measurement will tell us which are the "best" organizations or projects to support. That's a subjective evaluation at the best of times and empirically unsound, given that most of the data we collect are not strong enough to determine either causality or attribution. First and foremost, measurement is a contribution to learning and to sharing -- one input among many into well-informed but imperfect decision-making processes that continue to rely on human judgment.

In fact, arguments over measurement are not really about measurement at all. What separates enthusiasts and skeptics is not that one group wants more impact or believes in evaluation while the other group does not -- in my experience we all value those things -- but that we have substantive differences in how we approach these questions and which aspects of social change we are prepared to fight for and defend. That's why this debate generates so much heat -- and why it would be a much healthier conversation if we worked from that premise and were honest with each other about the dilemmas it creates.

Counting and measuring are only one part of a process of continuous, collective learning and communication that should be driven by the diverse needs and beliefs of all those who are working for and funding development and social change. There is no "best" approach or model or organization to be revealed by "the data," outside of a particular set of criteria and circumstances that are themselves contested.

Therefore, what we really need, as I'll explore in my fourth and final post, is an "ecosystem" of funders and funding styles and not a monoculture that uses measurement to drive resources to the causes which it, and it alone, finds of value.

-- Michael Edwards

Comments

[Bradford Smith]

Another great post Mike raising the deep issues about philanthropy, development, evaluation and social change. I particularly liked your point about arguments over measurement not really being about measurement at all, rather more about differences around what constitutes impact and social change. It can be difficult to settle an argument without addressing the underlying issues. Your point about "cultivating wisdom" is an interesting one. Increasingly I am convinced that the next evolutionary cycle for philanthropy (at least as represented by the larger, staffed foundations) will involve managing information to generate knowledge (and wisdom).

[Niki Jagpal]

I agree completely with Brad's comment above and your blog post, Mike. I think that one of the biggest challenges we face in terms of measurement is our inability to look beyond metrics and to truly seek out collective/mutual learning. As you note, this is especially important for organizations focused on social change or some kind of systemic reform effort in our sector. I hope that we can find common ground as grantmakers and grantees to build our mutual learning and capacity to develop meaningful measurement and break down the presumed power differential between funder and grantee.

[Bradford Smith]

Metrics metrics can play a huge part in that Niki, but alone are not sufficient. Metrics provide some of the data we need to answer larger questions.