Methods Enabling Constituency Voice in Impact Evaluation for Improving Development Andre Proctor, Keystone March 2009

Rethinking IMPACT

IPAL innovations Public reporting for learning and improving Feedback Specific and comparative ToC System logic frame for dialogue ad learning

A new theory of relativity… …..e=mc2 IPALCV Improved Development

What is Constituency Voice Bringing the voices of constituents, especially those meant to benefit, meaningfully into decision-making. All constituents play a role in How we define success How we work towards it (strategy) How we learn (measure impacts and reflect on their meaning) How we communicate it (reporting and sharing)

CV addresses the 6 purposes of Impact Planning, Assessment and Learning (1) Improve projects ensures usefulness for constituents Demonstrate impact from different perspectives on key enablers of change, insight in causal mechanism, triangulation Inform strategy real time feedback

CV addresses the 6 purposes of Impact Planning, Assessment and Learning (2) Sustain credibility and legitimacy Confidence and trust among constituents Build capacity empowers constituents Educate society reporting re-imagined as a learning activity both for constituents and wider society

Comparative Constituency Feedback An alternative approach to measuring performance and driving improvements: Ask them! Ask recipients how useful they find an organisations work. Related to customer satisfaction. Empowers primary constituents.

An emerging field Macro level: Humanitarian Response Index All OECD government funding for humanitarian response Center for Effective Philanthropy Grantee perceptions of US foundations Micro level: Measuring Empowerment quality.bond.org.uk quality.bond.org.uk Social movement in Bangladesh Listen First Concern Worldwide pilot

Those most affected Citizens Communities Resource providers ODA Private foundations Corporate philanthropy INGOs Implementers National/local CSOs NGOs Networks Dialogue for improvement Feedback Response Feedback Dialogue for improvement

Comparative feedback

Comparative constituency feedback Structured feedback from primary constituents on their experience of receiving funding / services / assistance. Generates quantitative data, for a group of similar actors. Performance can be aggregated and benchmarked. Provides data from the bottom upwards for learning and improvement.

Case study

East-African grantmakers Cohort of 10 grantmakers. Keystone surveyed all of their grantees. 50% response rate, 336 completed surveys Six key areas: Nature of fundingApplication process Monitoring, reporting & evaluation RelationshipNon-financial support Grantmakers knowledge & influence

Ethical Framework AreaPurpose Primary constituents are involved in designing survey Constituency voice Relevance Independently administeredObjectivity Informed consent of respondents No one is forced to participate Anonymous responses & data kept confidential No discrimination Report back to constituentsShared learning and open dialogue

Summary findings (extract) XXXs grantees are very satisfied with XXXs application processes, rating it 15.3 out of 20. … XXX keeps grantees better informed of progress and provides them with clearer information than most other grantmakers. XXXs grantees rated it 6.4 out of 20 for non-financial support (e.g. capacity building or advice). All grantmakers struggle in this area. Around 40% of XXXs grantees do not receive assistance in each area; and in some areas over 20% of grantees receive assistance that they do not find helpful.

Follow up questions… Could XXX consider: more site visits, and think about what would be most valuable to do on a site visit? reviewing its communications – especially web site and its annual report as communication tools? improving the way that it listens and responds to feedback from Grantees (responding to the feedback from this survey could be a good start). developing its staff understanding of developmental issues and create opportunities for learning from constituents? others …

Conclusion Robust data on grantmakers performance – not just focusing on grantees. Comparison drives insight and learning; approach fits with other monitoring activities. Drives real change (e.g. CEP). Adaptable to many different situations; the field is emerging. Puts the rhetoric of empowerment and accountability into practice. The future … ??

Beyond Self-Reporting Re-imagine reporting as an opportunity for CV and learning Report to all constituents and use as feedback opportunity Reflect constituent voices and perspectives Reports are socially validated in mutually accountable learning dialogues

Discussion questions CV How does Constituency Voice contribute to Impact Evaluation for Improving Development? How useful could Comparative Constituency Feedback be in what you do?