Community and Grantee Voice in Evaluation Design Leslie J. Cooksy, Evaluation Director Presentation at Grantmakers In Health March 7, 2014

Context: Respite Partnership Collaborative Purpose of the overall project: Test of community-driven process for grantmaking for respite services for people in or at risk of mental health crisis. Respite Partnership Collaborative – 22 community members representing mental health providers, consumers, family members and others

Stakeholders County External Evaluator RPC Sierra Health Grantees County of Sacramento – source of funding (through state) Oversight of state $$ Technical assistance Respite Partnership Collaborative – developing RFPs for respite services; reviewing proposals; making decisions; looking at grantee performance data Foundation: Manage and facilitate RPC processes – convening meetings, build and support infrastructure, guide proposal review and funding, manage the external evaluation Intended is a partnership among RPC, County, Foundation Reality is a shifting, complicated set of relationships – Public-private partnership between county and foundation that supports the RPC The RPC’s power is primarily around the grant decisions The grantees are not equal partners: Grantees are reviewed by the RPC, which has the power to defund them if they feel performance is not up-to-expected Foundation gathers performance data from the grantees County sets certain parameters on what can be funded In this picture, the external evaluation team is expected to be independent, not a collaborative or empowerment evaluation, but still expected to incorporate the voices of the grantees and community members

Role of the External Evaluation Learn about: Public-private partnership Community-driven process Grantee services Evaluation: Because of state $$ and the experimental nature of the community-driven process, evaluation is high stakes. Note: The foundation itself is being evaluated as a partner and as a facilitator of the community-driven process

Strategies for Incorporating Community and Grantee Voice

Logic Model Development RPC members developed a logic model –

Participation in RFP Development The logic model informed the evaluation questions in the RFP; and RPC members were involved in reviewing and refining the RFP

Strategies from RPC & Grantee Voices Participated in selection of external evaluation team Provided input to external evaluation design Reviewed final design Provide oversight of evaluation process and findings

RPC and Grantee Voices

Goals for Engaging RPC & Grantees Relevant evaluation questions Feasible, less disruptive evaluation methods Commitment to participating in evaluation activities Relevant evaluation questions Logic model RFP RPC member interviews Feasible evaluation methods RPC and grantee interviews Commitment to participating in evaluation activities High quality, useful results – what comes from the above…

The Downside It’s work It can raise expectations It doesn’t satisfy everyone It’s work Lyn’s perspective – work, commitment, but worth the result: An unusually motivated participant Paul’s perspective – overwhelming, balancing the different kinds of evaluation, but worth having the evaluation team know what will work in their context It can raise expectations Leslie’s perspective – the part that Leslie was happy with was about increased consumer perspective, but she was also unhappy with the overall result It doesn’t satisfy everyone Iffat’s perspective – even with the engagement, not everyone will buy in…Leslie’s disappointment; plus those people who are not that engaged

“Infrastructure of Trust” Create authentic opportunities for voice The Sweet Spot Authentic opportunities for voice Realistic Expectations “Infrastructure of Trust” Create authentic opportunities for voice Framing questions Selecting evaluation teams and/or tools Providing context and input so evaluation is relevant and feasible Engaging in evaluation activities Managing expectations – through building capacity about evaluation – which happens through engagement (logic model teaches trade-offs; RFP teaches need for focus; proposal review informs about budgets…) Multiple audiences Limited budgets …Include relevant perspectives and interests of the full range of stakeholders …maintain a balance between client needs and other needs.

Questions for you What strategies have you used to engage grantees in evaluation? What challenges have you faced? Have you been able to find “the sweet spot”?

Thank you Information about the RPC: Paul Cecchettini Lyn Corbett Iffat Hussain Leslie Napper Information about the RPC: http://www.shfcenter.org/rpc