1. Community and Grantee Voice in Evaluation Design
Leslie J. Cooksy, Evaluation Director
Presentation at Grantmakers In Health
March 7, 2014

2. Context: Respite Partnership Collaborative
Purpose of the overall project: Test of community-driven process for grantmaking for respite services for people in or at risk of mental health crisis. Respite Partnership Collaborative – 22 community members representing mental health providers, consumers, family members and others

3. Stakeholders County External Evaluator RPC Sierra Health Grantees
County of Sacramento – source of funding (through state)
Oversight of state $$
Technical assistance
Respite Partnership Collaborative – developing RFPs for respite services; reviewing proposals; making decisions; looking at grantee performance data
Foundation: Manage and facilitate RPC processes – convening meetings, build and support infrastructure, guide proposal review and funding, manage the external evaluation
Intended is a partnership among RPC, County, Foundation
Reality is a shifting, complicated set of relationships –
Public-private partnership between county and foundation that supports the RPC
The RPC’s power is primarily around the grant decisions
The grantees are not equal partners:
Grantees are reviewed by the RPC, which has the power to defund them if they feel performance is not up-to-expected
Foundation gathers performance data from the grantees
County sets certain parameters on what can be funded
In this picture, the external evaluation team is expected to be independent, not a collaborative or empowerment evaluation, but still expected to incorporate the voices of the grantees and community members

4. Role of the External Evaluation
Learn about:
Public-private partnership
Community-driven process
Grantee services
Evaluation: Because of state $$ and the experimental nature of the community-driven process, evaluation is high stakes.
Note: The foundation itself is being evaluated as a partner and as a facilitator of the community-driven process

5. Strategies for Incorporating Community and Grantee Voice

6. Logic Model Development
RPC members developed a logic model –

7. Participation in RFP Development
The logic model informed the evaluation questions in the RFP; and RPC members were involved in reviewing and refining the RFP

8. Strategies from RPC & Grantee Voices
Participated in selection of external evaluation team
Provided input to external evaluation design
Reviewed final design
Provide oversight of evaluation process and findings

9. RPC and Grantee Voices

10. Goals for Engaging RPC & Grantees
Relevant evaluation questions
Feasible, less disruptive evaluation methods
Commitment to participating in evaluation activities
Relevant evaluation questions
Logic model
RFP
RPC member interviews
Feasible evaluation methods
RPC and grantee interviews
Commitment to participating in evaluation activities
High quality, useful results – what comes from the above…

11. The Downside It’s work It can raise expectations
It doesn’t satisfy everyone
It’s work
Lyn’s perspective – work, commitment, but worth the result: An unusually motivated participant
Paul’s perspective – overwhelming, balancing the different kinds of evaluation, but worth having the evaluation team know what will work in their context
It can raise expectations
Leslie’s perspective – the part that Leslie was happy with was about increased consumer perspective, but she was also unhappy with the overall result
It doesn’t satisfy everyone
Iffat’s perspective – even with the engagement, not everyone will buy in…Leslie’s disappointment; plus those people who are not that engaged

12. “Infrastructure of Trust” Create authentic opportunities for voice
The Sweet Spot
Authentic opportunities for voice
Realistic Expectations
“Infrastructure of Trust”
Create authentic opportunities for voice
Framing questions
Selecting evaluation teams and/or tools
Providing context and input so evaluation is relevant and feasible
Engaging in evaluation activities
Managing expectations – through building capacity about evaluation – which happens through engagement (logic model teaches trade-offs; RFP teaches need for focus; proposal review informs about budgets…)
Multiple audiences
Limited budgets
…Include relevant perspectives and interests of the full range of stakeholders
…maintain a balance between client needs and other needs.

13. Questions for you
What strategies have you used to engage grantees in evaluation?
What challenges have you faced?
Have you been able to find “the sweet spot”?

14. Thank you Information about the RPC: Paul Cecchettini Lyn Corbett
Iffat Hussain
Leslie Napper
Information about the RPC: