P/PV President Nadya Shmavonian on Evaluative Learning

Evaluation in a nonprofit context -- what it is and isn't; who should bear the costs; and whether it's even possible -- remains a much-debated topic. Earlier today, for example, Charity Navigator president Ken Berger responded to skeptics of his organization's plans to revamp its oft-criticized rating system. You might not agree with all of the arguments Berger puts forward in his post, but most people would find it hard to disagree with his contention that a charity ought to be able to answer the following questions:

1. Is it using outcomes in the design, management, and measurement of its efforts?
2. Are the targets it sets "reasonable" outcomes? In other words, are they, at a minimum, meaningful, sustainable, and verifiable?
3. Is the organization achieving those outcomes?

Evaluation also is the subject of the latest Mott Conversation between the Mott Foundation's Duane Elling and Nadya K. Shmavonian, president of Public/Private Ventures, a nonprofit social research and policy organization based in Philadelphia, about the critical role of program evaluation in designing, testing, and sharing high-quality solutions that help low-income communities create meaningful and lasting change. Here's an excerpt:

Mott: How can the nonprofit sector best approach evaluation activities to ensure that they lead to genuine learning and innovation?

Nadya Shmavonian: Evaluation is not a thing so much as it's a way of thinking. Frankly, I believe if you are not asking what you want to learn from a given intervention or program at the beginning, then it isn't necessarily structured as strategically or thoughtfully as it could be. And if it’s not a shared value with the grantees who understand -- who are really trying to advance their own knowledge and best practices -- then I think the attention, use and consumption of evaluative information can be somewhat limited, and there is not the imagination for the next time.

In a place where you have continuity of programs, where you know that you're going to be in it long term, where you’re field-building and advancing, there's a sense of history and of how you can work evaluation in at the front end, but also at the back end. I think there are too many situations where foundations and their grantees have not reaped the collective benefit at the back end to reinforce that.

But, certainly, if you don't do evaluation, then there are enormous wasted resources. You don't ever systematically build an evidence base. What I also think is quite important is that in a good evaluation you can pull apart the component elements -- and the data will surprise you in many instances -- and then you can ask the next generation questions about, "Okay, so why did that work?" and think about the next intervention to advance that knowledge.

Particularly in these times, which are so resource-constrained, we should be focusing on these issues and getting smarter about what it means to evaluate, grow and potentially even replicate programs. But I worry that in the press for resources, we are actually limiting our targeting of thoughtful evaluative inquiry....

To view a video of the conversation, click here.

Or click here to read an edited transcript of the conversation.

Where do you stand? Is Ken Berger right in asserting that every charity should use meaningful, verifiable outcomes in the design, management, and measurement of its efforts? Do you agree with Shmavonian that a failure to do evaluation invariably results in wasted resources? And which nonprofits are using evaluation effectively to advance their mission and/or build a field of practice/knowledge? Use the comments section below to share your thoughts....

-- Mitch Nauffts

Comments

[Mae Watson Grote]

Shmavonian is dead on: If you don't know where you're going, how will you know you got there? Much less have the ability to share your lessons with others. Measuring expectations is just as critical as measuring experiences.