Evaluation in Foundations Welcome! Evaluation in Foundations October 26, 2012 AEA 2012

Panel Ruth Bowman: Ruth is an independent consultant and an affiliate of the Minnesota Evaluation Studies Institute at the University of Minnesota. Most recently, Ruth has held leadership positions within the group of Margaret A. Cargill Philanthropies. She has also served as adjunct faculty at the Hubert H. Humphrey School of Public Affairs (U of M). She served as the Assistant Editor of New Directions for Evaluation from 2004 – 2006. Becky Stewart: Becky has been Research and Evaluation Director with the Improve Group for eight years, a consulting firm in the Twin Cities, MN. Her client history involves programs ranging from small, local pilots, to statewide projects, to national initiatives and international programs. Randi Ilyse Roth: Randi Ilyse Roth has been the executive director of the Otto Bremer Foundation since May 2008. Before joining the Foundation, Roth served as the independent, court-appointed monitor in a civil rights lawsuit brought by African-American farmers against the U.S. Department of Agriculture. Prior to that, she served as a staff attorney and then as executive director of the Farmers’ Legal Action Group.

Foundation Evaluation Systems: A Case Study Designing Evaluation Systems for a Small Family Foundation

Case Study: The Setting Family foundation with about $3m in annual grant-making Hired to help the Foundation determine its approach to evaluating its impact Products: Grantee report Foundation “metrics” Framing questions guided the process Mission is to supporting efforts that address the unmet human and social needs of individuals, families, and communities that have the least access to resources. Additional context: Foundation had recently been through a planning period; strategies were pre-determined

Grantee Reports and Foundation Metrics Framing Questions Values Grantee Reports and Foundation Metrics Scope Resources Goal

What are key VALUES for the foundation? Why is the foundation doing evaluation? What is the relationship with grantees? What is the foundation’s desired role in the community?

What are the GOALS of evaluation efforts? Accountability: To whom? What are their mandates and expectations? Learning: By whom? About what? Planning: Over what time frame? What decisions need to be supported with data? Funding decisions: To inform ongoing funding decision? Streamlining grantee obligations? Accountability: To whom? What is mandated? Expected? Needed? Learning: By whom? What is the learning agenda for the Foundation? TO what extent does the Foundation desire to share learning? Planning: What information does the Foundation need to do planning? Annual? Longer term? What information is needed? Continuing support: Can/should the Foundation use information to determine ongoing support? Can the information help the Foundation understand when to STOP funding? Should application/reporting systems be streamlined/coordinated?

What are the RESOURCES for evaluation efforts? What data collection points or systems are available? What is staff capacity to gather, analyze and report on findings? What available data is relevant and accessible? What would be an acceptable burden on grantees?

What is the SCOPE of evaluation efforts? Where does the foundation expect to see change because of their support? What impact is realistic to expect, given grant amount and term? What level of evidence is needed? Do different aspects of the portfolio demand different levels of evaluation? What, if anything, makes sense to measure across different programs in the portfolio? Unit of analysis – Where do you expect to see change because of your funding? Community? Organization? Program? Participant? What is the Foundation’s theory of change? Funding/Expected Impact – What can realistically be achieved with the Foundation’s level of support/the scope of the program? Evidence needed – Is the Foundation support seeking to establish best practices? Conclusively attribute impact to a program model? Program portfolio - What kind of evaluation questions are appropriate for the kinds of program the Foundation funds? Confounding factors, environmental trends

Grantee Report Content Monitoring Milestones/Elements Organization Information Financial strength Partnerships Environmental scanning/Trends Needs Monitoring Milestones/Elements From this section, suggested elements to inform a board report are: Percent of programs meeting targets Average scores for program monitoring items Highlight informative, interesting select success stories Highlight major challenges to inform (1) environmental review and (2) funding strategy discussions Organization Information Financial strength: Average scores for financial health item to provide a summary of how organizations are faring in their finances Provide some examples of how Phillips Family Foundation funding allows grantees to leverage further support; will help Phillips Family Foundation to (1) clarify its understand of impact and (2) understand the role the Foundation plays in the program life cycle Percent of organizations ranking funding items as (1) hinder ability to work towards goals and (2) accelerating ability to work towards goals Partnerships: Average score on the partnership item Select examples from top and bottom categories to illustrate differences Note any staff observations on themes of partnership impacts on program effectiveness Percent of partnerships rated at “accelerating ability to work toward goals” Use overall results to determine if there if Phillips interest in convening could help strengthen partnerships Environmental Scan: Average score on the item regarding emerging trends to describe organizations’ strength in environmental scans Foundation staff selection of interesting/commonly described trends Suggestions to inform work planning: Compilation of ways that the Foundation can be helpful aside from funding (list of interesting ideas and themes) Extent to which the Foundation is already doing some of the suggested items Alignment of suggestions to Foundation resources and strengths   Program Model Development Notes about program model adjustments, best practices and target populations for the Foundation’s ongoing learning Average score for this section Note any trends in populations for whom program models are not working well Common themes of responses to the items around sustainability Alignment with Foundation resources and strengths Extent to which these needs call for the participation of other partners and consideration of Foundation role in supporting that involvement

Foundation Metrics (excerpt) The Foundation will spark innovation in mission-aligned programs Strategy Support programs that are trying promising new approaches Indicators Programs are trying new ideas based on strategic thinking Foundation is one of few, early funders Programs show signs of successful implementation Foundation shares lessons learned and new ideas

Randi Ilyse Roth Executive Director Otto Bremer Foundation www.ottobremer.org