Making Sense: Reflecting on Evaluations at the Jim Joseph Foundation
August 23, 2017
A core part of the Jim Joseph Foundation's relational approach to grantmaking is supporting the efforts of grantees to evaluate their programs — either through engaging an external evaluator or by collecting and analyzing data internally. The foundation has always believed this is a key part of good grantmaking, in that it builds the capacity of organizations to ask questions, to collect data, and to reflect on findings in a way that then enables them to make changes that increase the chances of success.
In this period of transition at the foundation, the grantmaking team has asked some pertinent questions regarding our own evaluation program: "What are we learning from the evaluation work we have supported over the past eleven years?” And, "Are there common lessons and emerging themes that we should recognize and reflect upon?"
To begin exploring these and other questions, the entire foundation team gathered for a full day earlier this year to share and discuss learnings and common themes discovered from a comprehensive review of nearly all the key evaluations and reports commissioned by the foundation since its inception.
To make the day as productive as possible, the foundation grantmaking team completed "homework" in the weeks leading up to the day-long session, dividing up the responsibility for reviewing a sample of forty-two evaluation reports, capacity-building and business plans, and field-building research reports — all commissioned and completed in the foundation's first eleven years — among team members and asking them to summarize the challenges, outcomes, and successes they identified in their respective documents.
This "day of evaluation reflection" (as we called it) turned out to be well worth the collective time and effort and, importantly, offered space for the team to discuss how the information and lessons that surfaced in our conversations might guide our future work. The summary below includes highlights from those discussions.
The foundation's effect on Jewish life and learning
How has the Jim Joseph Foundation influenced Judaism and Jewish peoples' approach to Jewish life and learning? This overarching question speaks directly to the foundation's mission. A common theme across many of the grants we have funded and evaluated is fostering community and positive relationships within the Jewish community. With few exceptions, evaluations show that participants in foundation-supported programs report feeling more connected to their Jewish identity and to Israel when those are the intended outcomes of the program. Since the DNA of the foundation includes a broad interpretation of and approach to Jewish learning, these programs encompass every kind of setting and activity, from camps, to schools, to service experiences, to Jewish outdoor food and environmental education. And, almost without exception, they have all proved to be effective while remaining aligned with our mission and values.
Lessons learned that have potential to inform foundation grantmaking
Several key themes emerged from the day's discussions that highlight opportunities for reflection, focus, and improvement:
- Young adult communities can be brought together successfully through different interests and avenues that resonate and are relevant to the lives of young adults. Social justice and service increasingly are reasons for young Jews to engage in Jewish life. And follow-on programming after an immersive experience is critical to deepening programmatic impact, creating community, and achieving positive outcomes.
- Successful programs vary in cost and scale, and while immersive programs can be expensive and reach a relatively small number of people, they also tend to have a deep and lasting effect on participants. Other programs, such as doctoral programs in Jewish studies or education, are a longer play, with a relatively high cost per student or participant.
- Mentorship and time for reflection are key elements in the success of many programs, particularly those focused on educator training. In addition, students value a reputable university program and also desire flexibility and diversity in their program options.
- Capacity building with respect to evaluation, development, and growth planning can be important investments for grantees. As a relational grantmaker, the Jim Joseph Foundation is in a position to help an organization pivot and/or engage in long-term strategic planning. These plans must be right-sized, however, with realistic revenue targets and investments.
- Relationships among organizations and people matter. There is value in collaboration and strength in building networks; both also are integral components of successful culture-change initiatives.
- Some grants are designed to leave a system in place so as to create impact long after the grant period ends. Admittedly, this is an ideal scenario, but local and national funding partners with aligned interests can leverage their resources to both widen and deepen the impact of their grant dollars.
Challenges grantees often encounter
The day also brought to the fore some of the common challenges grantee partners experience.
- The majority of challenges experienced by the foundation's grantees were related to marketing, recruitment, and retention. Retaining current participants can be just as valuable as bringing in new participants to a program/initiative. Another common challenge relates to hiring and retaining the right personnel — at all levels.
- Fundraising for sustainability and growth frequently is a challenge — and many effective programs end up being not "sexy" enough for donors.
- Whole school and/or organizational culture change is an effective way to create impact, but it often involves a lengthy process that requires significant staff capacity and buy-in.
Reflections on evaluation
In discussions about our evaluation support moving forward, the team discussed the importance of elevating the following concepts:
- Asking good questions and being data informed in our decision-making. Related: evaluations help tell a story for newer foundation staff members about what is working and what is not.
- It's important to create opportunities for funding to follow what is working — and evaluations can help inform both the "if" and "how" with respect to scaling a pilot program.
- We should "celebrate failure" in appropriate ways and for the purposes of learning. It's also important to acknowledge that some "failures" actually turned into partial successes years after the grant and evaluation periods had ended. In other words, sometimes an evaluation simply captures a moment in time that may not be representative of the true impact of the program.
- Field-building research reports frequently raise the profile of certain programs and certain issues — and dissemination is a very important part of the process.
- Assessing return-on-investment from a grant or series of grants is a daunting challenge. Numbers (e.g., program participants) do not tell the entire story about the long-term effects or how someone's experience influenced their worldview and connection to their faith and community. As a result of its experience, the team reaffirmed our commitment to understand more deeply how Jewish life and learning is experienced and fostered.
Our team viewed the Day of Evaluation Reflection as a productive, enjoyable time for learning. And staff expressed positive sentiments toward the day itself in terms of the structure, presentations, and team-building environment — as well as the preparation process outlined in advance. The conversations we had were open and honest, and signaled that the current grantmaking team is willing to critically examine the foundation's past, current, and future work in a manner that emphasizes transparency, trust, and patience.
The exercise also raised a number of interesting and important questions that we will continue to explore. As is our tradition, we will continue to ask new questions and encourage dialogue as a means to advance our work and deepen our understanding of the most effective ways to practice and evaluate philanthropy.