1. Outcome Harvesting A tool for identifying and understanding results in complex circumstances Ricardo Wilson-Grau

2. Agenda Outcome Harvesting’s origins When is Outcome Harvesting a useful tool How does the tool work — exercise Six steps Methodological challenges

4. Inspired by the Outcome Mapping methodology. A tool for practitioners operating in dynamic, uncertain situations to monitor and evaluate the social change results they are achieving. e Harvesting Outcome Harvesting

5. Developed since 2003 by me and my colleagues Barbara Klugman, Claudia Fontes, Fe Briones Garcia, Gabriela Sánchez, Goele Scheers, Heather Britt, Jennifer Vincent, Julie Lafreniere, Juliette Majot, Marcie Mersky, Martha Nuñez, Mary Jane Real, Natalia Ortiz and Wolfgang Richert. Ricardo.Wilson-Grau@inter.nl.net

6. 1. International social change networks Ricardo.Wilson-Grau@inter.nl.net

7. 2. International development donors Ricardo.Wilson-Grau@inter.nl.net

8. What did all of these organisations have in common?

9. Planning Monitoring “I can honestly say that not a day goes by when we don’t use those evaluations in one way or another.” Evaluation Ricardo.Wilson-Grau@inter.nl.net

10. Planning Monitoring “I can honestly say that not a day goes by when we don’t use those evaluations in one way or another.” Evaluation Ricardo.Wilson-Grau@inter.nl.net

11. Why?

12. Relationships of cause and effect are KNOWN Simple 12

13. Annual polio vaccination campaign ACTIVITIES OUTCOMES IMPACT INPUTS Inspired by Jeff Conklin, cognexus.org Time OUTPUTS M&E of a simple intervention

14. ACTIVITIES OUTCOMES IMPACT INPUTS Time M&E of a simple intervention OUTPUTS ¿Are we effective? Are we efficient? Annual polio vaccination campaign

15. Relationships of cause and effect are UNKNOWN Complex

16. OUTPUT OUTCOME INPUTS ACTIVITY INPUTS ACTIVITY INPUTS ACTIVITY INPUTS OUTPUT ACTIVITY OUTPUT OUTCOME M&E of a complex intervention Time Strengthening nonviolent responses to communal conflict in the Horn of Africa

17. OUTPUT OUTCOME INPUTS ACTIVITY INPUTS ACTIVITY INPUTS ACTIVITY INPUTS OUTPUT ACTIVITY OUTPUT OUTCOME Time 17 Dimensions in which the relationships of cause and effect are known Not either or Ricardo.Wilson-Grau@inter.nl.net SIMPLE

18. OUTPUT OUTCOME INPUTS ACTIVITY INPUTS ACTIVITY INPUTS ACTIVITY INPUTS OUTPUT ACTIVITY OUTPUT OUTCOME Time COMPLEX Results are substantially unforeseeable. Ricardo.Wilson-Grau@inter.nl.net Not either or Unknown relations of cause and effect dominate.

19. Outcome Harvesting is NOT useful Ricardo.Wilson-Grau@inter.nl.net

20. Outcome Harvesting IS useful Ricardo.Wilson-Grau@inter.nl.net

21. When do you harvest outcomes? Time SummativeDevelopmentalFormative Progress of intervention

22. Track record Over 300 networks and associations, NGOs, community-based organisations, research institutes, and government agencies on all seven continents. Identifying and reporting almost 3,000 changes they have influenced. Ricardo.Wilson-Grau@inter.nl.net

23. Most recently In August, the UNDP identified Outcome Harvesting as one of eleven “innovative approaches to monitor and evaluate the performance of public policies, programmes and service delivery.” Ricardo.Wilson-Grau@inter.nl.net

24. SOCIAL ACTOR Individual Organisation Institution Group or community What is an outcome? 1. Social actors changing the way they do things Behaviour Relationships CHANGES Policies Practices Ricardo.Wilson-Grau@inter.nl.net

25. Example Change: In 2009, the Palestinian Authority revitalizes an employment fund for qualified people living in Palestine.

26. Cause Effect When is it your outcome? 2. A change that you influenced

27. Example of influence Change: In 2009, the Palestinian Authority revitalizes an employment fund for qualified people living in Palestine. Contribution: The Global Call to Action against Poverty (GCAP) coalition in Palestine launched the “Stand Up and Be Counted” campaign, which mobilized 1.2 million people in 2008. Working with the Ministry of Labor, the coalition also helped secure multilateral funding and technical assistance on the management of the fund. Ricardo.Wilson-Grau@inter.nl.net

28. First… Identify outcomes Paradigm shift

29. Second… Discover the activities and outputs that contributed Paradigm shift

30. Other information? Significance of the outcome Collaboration with other social actors Contribution of other actors and factors History Context Evidence of impact on people’s lives And so forth. Ricardo.Wilson-Grau@inter.nl.net

31. Example Change: In 2009, the Palestinian Authority revitalizes an employment fund for qualified people living in Palestine. Significance: Demonstration of how mass citizen action can be combined with the engagement of political decision makers to lead to transformative changes in government policy and practice. Ricardo.Wilson-Grau@inter.nl.net

32. Six steps of Outcome Harvesting Ricardo.Wilson-Grau@inter.nl.net Six steps: 1.Design the harvest 2.Review documentation and draft outcomes 3.Engage with informants 4.Substantiate 5.Analyse, interpret 6.Support use of findings

33. 1. Design the harvest, focusing on usefulness Ricardo.Wilson-Grau@inter.nl.net Primary intended users Principal intended uses Useful Outcome Harvesting questions Credible data

34. Credible data = credible sources People who personally know who changed and what changed. They give initial authenticity, confidence and credibility to the information you harvest. Thus, Outcome Harvesting is highly participatory.

35. 35 When? Who? What? Where? Less is more! 2. Review documentation and draft outcomes 3. Engage with informants to craft the outcome descriptions Ricardo.Wilson-Grau@inter.nl.net

36. Exercise Please take five minutes to read this 2012 annual report from FundNow. Then, with the person seated next to you: A. Identify one outcome: – Who changed behaviour, relationships, activities, actions, policies or practice? – What changed? – When did it change? – And where?

37. FundNow 2012 Annual Report Feedback from our allies and co-workers about FundNow’s activities during January-December 2012 has been very positive, emphasizing the usefulness of our research publications to inform about the funding challenges and opportunities facing women's organizations in the country. Our research has shown that women's rights organizations have been living in a form of "resist and survive" mode but doing amazing work for the advancement of women's rights. People always mention our online newsletter FundNow as a tool that has helped them to think more strategically about raising funds for their organizations. The preparatory research we did for each one of the13 meetings we convened was greatly appreciated. This year, our innovative research on trends in financing for gender equality and women's organization has been a strongly supportive tool for women’s groups obtaining more and better resources across the country. Based on the collection of "gems" of impact of projects implemented by the grant recipients of the government’s Fund for Leadership and Opportunities for Women (FLOW), we compiled a report that has been used by an ad hoc group of gender activists within the ministry responsible for FLOW. They used our compilation in their internal lobbying for the replenishment of FLOW, which resulted in the addition of US$1 million to the Fund in late 2012. In addition, this year we have successfully organized 2 strategic meetings for women's organizations on provincial resource mobilization. (In addition, we organised 7 more meetings in the capital for a total of 9.) There was intense interest in these meetings, the first held in those provinces. In one province, 5 funders and 18 women's organizations attended and in the other 4 and 12, respectively. The background research we presented was based on regional surveys carried out in preparation for the meetings; it was very well received. Participants said we were filling a clear gap of information with our findings. Many participants expressed that these meetings are a strategic opportunity to establish new contacts, engage in networking, develop strategies, and have a space for potential donors and grantees to meet and share experiences. Everyone said the meetings enabled them to better understand the context in which both funders and women's organizations operate. The meeting participants also had time to reflect on their work and said these events served as an inspiration. All the participants expressed their commitment to work together in the future to guarantee funding for women's organizations in the two provinces.

38. Exercise, continued B. How did FundNow contribute to the outcome? What activities and outputs plausibly contributed to the change in the social actor, however partial, indirect and even unintentional the contribution may have been?

39. Engage with Primary Intended User 1.To clarify and confirm the details of use 2.To establish the Outcome Harvesting questions to be answered 3.And identify the information (and its sources) required to answer the harvest questions 39

40. 4. Substantiate: Verify accuracy + deepen understanding A. Description: In late 2012, the ministry responsible for FLOW took the decision to add US$1 million to the Fund. B. Contribution: FundNow compiled research that was used by the ad hoc group within the ministry to advocate internally for replenishment of the Fund. 1. To what degree are you in agreement with the description of the ministry’s action as described? [ ] Fully agree [ ] Partially agree [ ] Disagree Comments if you like: 2. How much do you agree with the description of how FundNow influenced the ministry’s decision? [ ] Fully agree [ ] Partially agree [ ] Disagree Comments if you like:

41. Triangulation of sources Ricardo.Wilson-Grau@inter.nl.net Historical documentation Person(s) who influenced others over time Independent third parties SOURCES

42. METHODS Triangulation of methods Ricardo.Wilson-Grau@inter.nl.net

43. 5. Analyse and interpret the outcomes to answer the useful evaluation questions Ricardo.Wilson-Grau@inter.nl.net

44. Triangulation of analysis Ricardo.Wilson-Grau@inter.nl.net Methodologist Person(s) who influenced others over time Content experts ANALYSTS

45. Excel Database Ricardo.Wilson-Grau@inter.nl.net

46. Access Database Ricardo.Wilson-Grau@inter.nl.net

47. IMPACT Theory of change RESULTS SOCIAL ACTORS ASSUMPTIONS MISSION STRATEGIES

48. Systems approach

49. 6. Support for use of findings Report And Support

50. In sum Outcome Harvesting is “an evaluation approach that does not measure progress towards predetermined outcomes, but rather collects evidence of what has been achieved, and works backward to determine whether and how the project or intervention contributed to the change.” - UNDP Ricardo.Wilson-Grau@inter.nl.net

51. Further information: Outcome Mapping Learning Community, www.outcomemapping.ca Ricardo Wilson-Grau Consultoria em Gestão Empresarial Ltda Rua Marechal Marques Porto 2/402, Tijuca, Rio de Janeiro, CEP 20270-260, Brasil Telephone: 55 21 2284 6889; Skype: ricardowilsongrau Direct via VOIP, dialing locally from: The Netherlands (or Europe): 31 71 302 0429 USA or Canada: 1 347 404 5379 Questions? Ricardo.Wilson-Grau@inter.nl.net http://ow.ly/d65Y1