Does Measurement = Randomized Control Trials?

(The following post by Andrew Wolk, CEO of Root Cause, a nonprofit that works to advance solutions to social and economic problems by supporting social innovators and educating social impact investors, originally appeared on Andrew's blog. Click here to read a recent op-ed piece by Wolk in PND.)

There is a debate going on about how to identify and invest in the highest-performing nonprofits -- with a great emphasis on randomized control trial (RCT) studies.

The RCT, a research methodology that involves randomly selecting subjects from a larger test group to receive an experimental product or service, is undoubtedly a rigorous way of determining whether a cause-and-effect relationship exists between a given service and a desired outcome.

The current interest in RCTs is an encouraging sign of the growing momentum for linking nonprofit funding to proven results and investing in what works, particularly if we can also disseminate the information far and wide.

At the same time, RCTs raise concerns for me, as they could end up stifling rather than encouraging social innovation.

The trouble with randomized control trials is that they are extremely expensive to carry out and so only the more developed and/or the best able to access resources participate. Most nonprofits struggle with finding the funds for general operations, let alone conducting an RCT.

So, we are stuck between encouraging social innovation and the strong desire to invest in what works. I believe we need to develop a spectrum that links an organization’s stage of development, with measurement requirements, to levels of investment. The stages could look something like this:

The Social Impact Measurement Spectrum

Stage 1 - Start-up: At the earliest stage, start-ups with promising ideas could receive minimal support in exchange for a commitment to developing and tracking an initial set of measures that align with standards in the organization’s field.

Stage 2 - Proof of concept: At the next stage, the organization would have a simple, internally driven performance measurement system that would drive internal continuous improvements while generating initial data on the impact of the organization’s model. Funding would be increased to match the increased rigor of the organization’s measurement, but it would still be far less than that given to organizations at later stages.

Stage 3 - Promising, not proven: At this stage, organizations would be required to operate a full performance measurement system that is integrated with their business models and publicly share the results. They might also do an external evaluation at this stage to further elucidate their impact.

Stage 4 - Spread of proven model: This final stage would occur when an organization has conducted an RCT study with positive results. They would be able to receive substantial investment in exchange for a commitment to share the details of their model so it spread and others could learn and adopt the organization’s successful practices. The Coalition for Evidence Based Policy is building a site that will identify social interventions shown in rigorous studies to produce sizable, sustained benefits to participants and/or society.

A spectrum like this would do a few important things. It would provide a framework that would inform how a social innovation might spread; it would provide a roadmap to the kind of measurement system an organization should develop over time; and, lastly, it would give an idea as to the amount of funding an organization could expect as an innovation moved toward greater proven social impact.

-- Andrew Wolk

What do you think? Is Wolk's concept helpful and/or workable? Share your comments below....

Comments

[Jeff Mason]

This is a great post! I couldn’t agree more. The model makes a lot of sense and syncs with the work of David Hunter and Steve Butz in their Guide to Effective Social Investment and the related Social Value Risk Assessment tool. Both can be found on the Alliance for Effective Social Investing website (http://www.alleffective.org).

I like to think of high-performing orgs as those that have clear goals, established indicators to monitor their progress toward achieving goals, collect quality data that relates their efforts to desired outcomes so they can understand what works and what doesn’t, and then make appropriate adjustments to continuously improve. This is an essential part of the daily work an org must do to stand a significant chance of achieving impact.

It’s a high-impact org that has been shown via RCT to have been the cause of the social change. But even high-impact orgs need to be high-performing on a daily basis since any impact shown via RCT is due to past efforts. We need to invest in future efforts. What will you do tomorrow? This is all about an orgs ability to effectively manage their performance.