1. Outcome Mapping Bringing learning into development programmes 15-18 September 2009 Capetown South Africa Robert Chipimbi Simon Hearn

2. Acknowledgements  This presentation makes use of various materials that were shared by various OM community members on the OM learning community website. Without being exhaustive special thanks goes to Terry Smutylo, Enrique Mendizabal, Steff Deprez, Jan Van Ongevalle, Daniel Roduner, Kaia Ambrose and many others.

3. ´´The only real voyage of discovery exists, not in seeing new landscapes, but in having new eyes´´ Marcel Proust

5. Workshop objectives 1. Understand basic OM concepts & principles 2. Decide if & how OM applies to your work 3. Apply some OM tools

6. Before we start... Outcome Mapping is different from conventional PM&E models  It requires time to absorb the concepts and implications for your work  It might seem very complex at first (new concepts, new language, …)  Requires a change in thinking (new paradigm)  Generates different reactions Interest Confusion / doubt Disagreement Defense Let’s go for it! How can we ever do this? … DON’T WORRY - ASK QUESTIONS!

7. Developed between 1998 and 2001 by the International Development Research Centre (IDRC) as an answer to the difficulties faced in evaluation processes of development programs: Demonstrating results Proving cause/effect relationships Measuring & attributing impact What about unintended results? … origins of outcome mapping

8. Some core principles of outcome mapping 1. Embracing complexity 2. Sphere of influence 3. Boundary partners 4. Outcomes as behavioural changes

9. 1. Embracing Complexity Looking at the bigger picture See yourself as a part of an interconnected web of relationships and systems

10. Constant change “It’s not possible to see the same river twice”

11. Recognizing that change is…  Continuous  Complex  Non-linear  Not controllable  Multidirectional

12. Keep your eyes wide open… Being attentive along the journey is as important as the destination

13. opportunities for P,M,E and learning beginning Life cycle of the program endduring objectives, inputs, Activities outputs results, impact Often Dealing with social systems Non-linear process Complex relationships & interactions Unpredictable nature of human behaviour Unknown factors & actors

15. Impact (changes in environmental or social conditions) Objectives (activities, products) changes in behaviour Life cycle of the program different approaches Logical Frame Analysis Results based Mgt

16. 2. Sphere of influence Inputs Activities Outputs Outcomes Impact Sphere of control Sphere of interest Sphere of influence

17. Influence of local actors increases project influence decreases Relative Influence Along the Results Chain Inputs Activities Outputs Outcomes Impact Sphere of controlSphere of interest Sphere of influence Behavioural Changes

18. 3. Focus on boundary partners  Key concept is « boundary partners »  The individuals, groups, and organizations you work with directly and anticipate opportunities for influence

19. An actor-centered approach to development planning and M&E  “Development is essentially about people (groups and organisations) relating to each other and their environment.”  Although a programme can influence the achievement of outcomes, it cannot control them because ultimate responsibility rests with the people affected.  “Outcome Mapping centred around programme actors and their relationship in development programmes.”

20. A programme can not control change, it can only influence and contribute to changes at the level of its boundary partners Implementing team Boundary partner 1 Boundary partner 2 Boundary partner 3 Beneficiary 1 Beneficiary 2 Beneficiary 3 Sphere of Control Sphere of influence Sphere of interest

21. 4. Outcomes as behavioural changes Outcome mapping focuses on outcomes as changes in behaviour of those with whom the program or project is working directly (boundary partners)

22. Why behavioural change?  For each change in state (e.g. security of land for marginalized groups) there are always correlating changes in behaviour of certain people and groups.  Assessing changes in state – as often in logframes – not necessarily provide the kind of information that programmes need to improve their performance and relevance.  Development is done by and for people.

23. Principles of use  Flexibility OM needs to be adapted to use in your specific context. Not a fixed route but a guide for the journey we take.  Participatory OM implies dialogue and collaboration with partners. We co-create the ´map´ with our partners.  Evaluative thinking Fosters a reflective practice, organisational & social learning.

24. principles of use Flexible: Can be adapted to use & context (Not a fixed route but a guide for the journey we take) Complementary: combine with other methods

25. principles of use Participatory: seeks dialogue and collaboration with partners (We co- create the ´map´ with our partners) Evaluative: promotes culture of reflection, results oriented thinking, and social & organizational learning

26. where is the map? ✓ OM is a guide to the journey we take with our partners. We co-create the map. ✓ It focuses on the intention and what happens along the way ✓ The map is not the territory, it shows the route taken

27. The outcome mapping framework

29. Step 1: Vision stage 1 Intentional Design

30. I have a dream! Martin Luther King, Jr. August 28, 1963

31. The vision  Description of the universe in terms of human, social, economic or environmental conditions  Large-scale development-related changes  Describes the ideal world  Beyond program's capability  Functions as a lighthouse – something to guide movement  …and a star – something big to aim for  Written in present tense

32. Example vision statement  Education in Zimbabwe is reoriented towards critical thinking, action competence and responsible behaviour by individuals and groups to achieve sustainable living in a healthy environment. In this way, the Zimbabwean community is empowered to make informed individual and collaborative decisions, which ensures continual effective environmental management.

33. vision in graphic form, Nagaland (India)

34. ? ? vision facilitation questions Imagine that, 5-10 years from now, the program has been extremely successful. Things have improved beyond your most ambitious dreams. What changes have occurred? What (& how) are your intended beneficiaries doing? What are your partners doing? Describe the better world you are seeking.

35. Exercise Vision & Mission  Worksheet to be supplied

36. Step 2: Mission stage 1 Intentional Design

37. The mission is that “bite” of the vision statement on which the project is going to focus. The Mission

38. Mission statement Describes how the program will contribute to the Vision, including its main working areas and the partners with whom it will implement them. Written in future tense - as something the program will do

39. Example Mission Statement In support of this vision, St 2 eep will create an enabling environment for sustained EE implementation in Secondary Teachers’ Colleges through the encouragement of active learning processes which promote participation, critical thinking, informed decision-making, action competence and responsible citizenry. St 2 eep will conduct on-going pre- and in-service capacity enhancement, curriculum review and implementation and strive for the reorientation of assessment of teaching and learning in line with principles of Environmental Education. St 2 eep will lobby for motivational strategies and policies to ensure that EE is institutionalized in the education system and encourage outreach programs through networking with schools, communities and environmental interest groups. St 2 eep will ensure that graduates of the Secondary Teachers’ Colleges are able to implement EE in their teaching.

40. What areas do you need to work in? What do you need to do in these areas? Who can you work with? How will you stay effective, efficient, and relevant? ? ? Mission facilitation questions

41. Summary ✓ about the future ✓ observable ✓ idealistic ✓ not about the program ✓ feasible ✓ identifies activities and relationships ✓ about the program VisionMission

42. Exercise: Vision and Mission

43. step 3: Boundary Partners stage 1 Intentional Design

44. boundary partners Those individuals, groups, and organizations with whom the program ✓ interacts directly to effect change ✓ anticipates opportunities for influence ✓ engages in mutual learning

45. Program = boundary partners

46. has direct control over its inputs, activities,... in working with the boundary partners, but can not control change at level of its boundary partners and beneficiairies. The ultimate responsibility rest with the people affected. hopes to have direct influence / contribute to changes at the level of its boundary partners can only indirectly influence change at the level of ultimate beneficiaries (impact) a programme…..

47. boundary partners have boundary partners program program’s bp bp’s bp

48. strategic partners selected on the basis of their contribution to the mission a person or group with whom the program works directly to achieve the mission, without necessarily wanting to change the partner’s behavior Examples Other donors Other organisations doing similar work Media

49. St2eep impl team EE coordinators and VVOB facilitators Vision Mission Beneficiaries: Colleges, Lecturers & staff, Student Teachers, Pilot schools Communities, …. BP3 EE Steering teams ( EE Coordinator) BP1 Ministry of Higher & Tertiary Education BP2 College administrations Strategic partners Min. Of Education Min. Of Environment SADC REEP ZWEECF Universities NGO’s... Funding Organisations VVOB SADC REEP... Outcome Challenge 3 BP4 Dept of Teacher Education (UZ) Outcome Challenge 4 Outcome Challenge 2 Outcome Challenge 1 St2eep project 2005-2008 BP5 Ministry of Education, S., C. Outcome Challenge 5

50. ✓ In which individuals, groups, or organizations is your program trying to encourage change as a contribution to the vision? ✓ With whom will you work directly? ✓ Are you choosing boundary partners because you want to influence the ways they help or influence others? ? ? boundary partners facilitation questions

51. Boundary partner exercise  Worksheet to be supplied

52. Step 4: Outcome Challenges stage 1 Intentional Design

53. what indicators do you use for: Greater awareness… Empowered women… Reduced conflict… Increased collaboration… Gender sensitivity… Equal access… Active participation… Poverty alleviation… Strengthened capacity… In OM you describe what the BP is DOING DIFFERENTLY and those are your ‘results’. ?

54. ✓ describes the ideal behavioural changes, relationships, actions (professional practices) of a boundary partner for it to contribute to the vision outcome challenge

55. ✓ one for each boundary partner ✓ describes the boundary partners’ ideal (but possible) contribution to the vision ✓ written like this: “The program intends to see [boundary partner] who [description of behaviors in the active present tense]” outcome challenge characteristics

56. ✓ Ideally, in order to contribute to the vision, how would the boundary partner be behaving? ✓ With whom would they be interacting? ✓ What would they be doing to contribute maximally to the vision? ? ? outcome challenge facilitation questions

57. Example outcome challenge statement St2eep intends to see college administrators who continually support EE activities within the colleges by appointing full–time EE coordinators and providing office space, transport, equipment, finances, communication and other resources for successful EE implementation. These college administrators are cooperating with St2eep by incorporating EE in the college strategic plan and encouraging lecturers for EE orientation, workshops and programs. They are attending EE organised functions. They are actively supporting policy development and implementation through the recognition that EE is part and parcel of the curriculum and college functions.

58. Outcome challenge exercise  Worksheet to be supplied

59. Program Framework Vision: Mission: Boundary Partner: Outcome Challenge:

60. Step 5: Progress Markers stage 1 Intentional Design

61. progress markers ✓ Describe changes in actions, activities and relationships leading to the ideal outcome ✓ Articulate the complexity of the change process ✓ Can be monitored & observed ✓ Enable on-going assessment of partner’s progress (including unintended results)

62. progress markers (ladder of change) Expanding influence, helping others, sharing expertise Actively engaged, learning, commitment Early encouraging response to program, initial engagement Love to see Like to see Expect to see

63. progress markers are graduated ✓ move from initial to more profound changes in behaviour ✓ show transformation in a single boundary partner ✓ more informative than a single indicator

64. taken together, progress markers facilitate mid-course corrections and improvements articulate the complexity of change stimulate the program to consider how it can contribute to the most profound transformation possible suggest the logic model of change are NOT a checklist of accomplishments!

65. a network can be a boundary partner ResearchersService providers NGOsPolicy makers Project intends to get them working together to solve a common problem Expect: meet, listen to from each other Like: organize an event together Love: implement a joint initiative

66. Example Progress Markers The Project Expects to See College Administrators:  Authorising lecturers to participate in EE activities.  Providing office space and equipment for the EE coordinators. The program would Like to See College Administrators :  Providing transport, finances and other resources for EE activities.  Including and positioning EE, high on the agenda of staff meetings. The program would Love to See College Administrators :  Incorporating EE in the college strategic plan.  Appointing full time EE coordinators.

67.  Do not require consultant to write quantifiable indicators  Show directionality of change  Support learning function  Can be monitored & observed from the start of the program  Permit on-going assessment of partner’s progress (including unintended results)  Provide basis for dialogue with partners some advantages of progress markers

68. How many progress markers? It depends. Suggestion: total of 15, with most occurring in the “like to see” range. Life of Program Expect 4 Like 8 Love 3

69. Exercise Progress Markers  Identify 3 strips that are not progress markers  Divide remaining strips between Expect to see Like to see Love to see

70. Outcome challenge and progress markers Boundary Partners... Programme Team Interventions/strategies??

71. Step 6: Strategy Maps stage 1 Intentional Design

72. Strategy Maps - Which intervention strategies are we developing in order to support/influence the boundary partners towards the desired changes? - Outcome Mapping provide a framework to develop a mixed set of strategies to assure greater potential for success: 1. Strategies directly aimed at individuals, teams, organisation of the boundary partner 2. Strategies aimed at the environment in which the boundary partner is working - Programme interventions most likely change during the course of an initiative

73. causalpersuasivesupportive I aimed at individual boundary partner E aimed at boundary partner’s environment

74. 6 types of support strategies causalpersuasivesupportive I aimed at individual boundary partner direct Influence (e.g. funding, prepare a report, …) arouse new thinking; build skills, capacity (e.g. skill enhancement, methodological workshops, training) on-going support supporter/mentor who guides change over time,involvement is more frequent and sustained, nurturing for self- sufficiency (e.g. program member who provides regular guidance and input, coaching, expert, …) E aimed at boundary partner’s environment alter the physical or regulatory environment (e.g. technical transfer, policy change, internet access, terms of reference, rules, guidelines) broad information dissemination/ access to new info (e.g. Radio, TV, website, publications, conferences, workshops) create / strengthen peer networks partners working together and collectively supporting each other) (e.g. research network, …)

75. causalpersuasivesupportive I aimed at individual boundary partner How to produce an “immediate effect in the BP”? How to build capacity, to influence new ways of thinking or doing things? How will sustained support, guidance or mentoring be provided? E aimed at boundary partner’s environment How to alter the physical or policy environment? How to influence the quality, availability, accessibility of information? How to influence the media? What networks or relationships will be established, strengthen or utilized? strategies: facilitation questions ? ? I-2 I-3 I-1 E-1E-2E-3

76. causal persuasivesupportive I aimed at individual boundary partner Fund research projects Run workshops on quantitative and qualitative methods Offer internet reesarch courses Coordinate training on participatory methods Offer gender sensitivity training to those working with HIV-infected women Hire a professional writer to work on dissemination strategies with researchers Hire a fundraiser to help identify donors and develop a fundraising strategy Provide ongoing technical assistance E aimed at boundary partner’s environment Provide computers and internet access Include work with women and youth as a condition for the grant Organise regional conferences for HIV/Aids research community Develop internet site with tools and methods Publish ‘special paper’ series Establish fromal mentorship program that partners experienced and junior researchers. Facilitate the development of an electronic HIV/AIDS policy research network Example strategy map

77. value of strategy maps ✓ Clarify what the project is doing, with whom and why ✓ Articulate the range, mix and logic of the strategies ✓ Detect strategy gaps and over extension of resources ✓ Encourage multiple interventions to increase effectiveness

78. Outcome challenge and progress markers Boundary Partners Programme Team Organisational practices? Strategies & activities

79. Step 7: Organisational Practices stage 1 Intentional Design

80. Organisational practices  Supporting change in boundary partners requires that the program/organisation be able to change and adapt as well  Move beyond being efficient & effective (operational capacities) towards being relevant (adaptive capacities)  Therefore OM incorporates an ongoing assessment and reflection of the own practices & capacities into the design of the programme

81. what you do as an organization to ✓ keep learning ✓ foster creativity & innovation ✓ seek better ways to assist your partners ✓ maintain your niche ✓ maintain high level support ✓ build relationships

82. organizational practices 1.Prospecting for new ideas, opportunities, and resources 2.Seeking feedback from key informants 3.Obtaining the support of your next highest power 4.Assessing and (re)designing products, services, systems, and procedures

83. …organizational practices 5. Checking up on those already served to add value 6. Sharing your best wisdom with the world 7. Experimenting to remain innovative 8. Engaging in organizational reflection

84. Exercise – organisational practices  Worksheet to be supplied

85. 85 mission vision Girls & Women Community Leaders Women’s Self Help Groups Families Police State NGOs Banks Public Health Clinics Strategic Partners Strategies Project’s Outcome s Boundary Partners BP’s outcomes BAIF

86. Overview of Stage 1 Organizational Practices project team Interest Influence Control

88. stage 2 Outcome and Performance Monitoring

89. 4. Monitoring & Evaluation in OM « Not everything that counts can be counted. and not everything that can be counted, counts. » Albert Einstein Monitoring and evaluation

91. An outcome mapping based M&E system looks both at the processes that are going on within the programme and how the programme (through its processes) is contributing to certain outcomes (no causal relationship)

93. M&E – what’s the difference? Monitoring  Ongoing, continuous, more regular  Internal activity  Responsibility of project staff and management  Continuous feedback to improve programme & report on performance Evaluation  Periodic and time bound  Can be internal, often external  Responsibility of evaluator with staff and management  Periodic feedback  A more in depth study and assessment of an issue

95. Outcome challenge and progress markers (outcome journal) Strategies & activities (Strategy Journal) Organisational practices (Performance Journal) Boundary Partner Program Planning and monitoring opportunities in OM Change in behaviour Internal performance contextual information

96. Accountability & Learning: A Balancing Act

97. Changing the M&E perspective Old M&E:  M&E is mainly for external funding body  Concentrates on activities and financial reporting  Lots of data and little analysis (big reports!)  Little learning takes place  Little stakeholder participation  M&E is boring and not very useful for project staff and beneficiaries New M&E  M&E is for everybody involved in project  Also look at effects and outcomes and why there is success and failure  Learning is a key function of M&E  Good analysis on how to improve project  Lots of stakeholder participation  M&E is active, interesting and useful -> Evaluative thinking throughout the working cycle

98. Accountability & Learning: A Balancing Act

100. Exercise Monitoring tools  Monitoring Journals

101. Monitoring Plan Three additional questions: 1.How will we learn from the monitoring information, how do we make sure lessons learned feed into future planning? 2.How is the reporting going to be done? 3.Who gets the report?

102. Exercise – Monitoring Planning  See worksheet

103. Learning & Reporting (Info to users/events/audiences) Clarifying Intent (Both interventions & outcomes) M&E Planning (Choosing what to track) M&E Implementation (Data collection & interpretation) Useful M&E Depends On

104. 104 reviewing the intentional design 1. Read the vision statement Does this still reflect the program's dream? 2. Read the mission statement Is this the greatest contribution our program can make? Have we been doing this? Why? Why not? Should we add anything or take anything away? 3. Review boundary partners Is this who we are working with directly? Do we need to add or drop any boundary partners? 4. Review outcomesDo these accurately reflect transformations in our boundary partners as they strengthen their contributions to the vision? 5. Review progress markers Was the change process we set out accurate and useful? What now needs to be added or taken out? 6. Review strategiesWhat did we plan to do? Have we implemented these activities? Do we need to add, remove any? 7. Review organizational practices Are we doing everything we can to maintain & enhance our capacity to support our partners?

105. stage 3 Evaluation Planning

106. « Evaluative thinking is a learned behaviour that you have to practice and improve at on an on-going basis! » Michael Quinn Patton

107. Step 12: Evaluation Plan  A short description of the main planning elements of an evaluation study  For Whom? What? Where? How? By Whom? When? How Much?

108. Elements of an Evaluation Plan

109. Setting Evaluation Priorities Criteria to select evaluation issue include:  Learning needs  Accountability or Reporting requirements  Partners’ information needs

110. Guiding Principles for Evaluation  Balancing learning & accountability  Action oriented  Each study is unique  Process not only product  Build capacity at all levels through participation

111. Points to Remember: 1. Stay in touch with reality. 2. Celebrate the contributions of your team and your partners. 3. Be “idealistic realists.”

112. Social change has multiple, interacting causes. Trying to determine the “impact” achieved by a single intervention may be a waste of time and resources. Seek, instead, to deepen understanding of how the actors and factors influence each other in relation to the program’s vision. OM’s underlying message:

113. ongoing OM applications http://www.outcomemapping.ca GhanaKoreaSwitzerland KenyaEcuadorUnited Kingdom MadagascarMexicoBrussels MaliEgyptNetherlands NamibiaBhutanAustralia UgandaHondurasSri Lanka ZimbabweGuatemalaIndia

114. http://web.idrc.ca/en/ev-27705-201-1-DO_TOPIC.html examples of use:

115. for more information and to share your experiences www.outcomemapping.ca www.idrc.ca/evaluation www.mapeodealcances.net

116. Thank You Very Much