An introduction to Monitoring and Evaluation in Outcome Mapping Applied and adapted use Bern, 15 April 2010

Inspired by Jeff Conklin, cognexus.org Applied Monitoring Vision and Mission Intentional design IMPACT OUTCOMES OUTPUTS ACTIVITIES INPUTS Time Inspired by Jeff Conklin, cognexus.org

Inspired by Jeff Conklin, cognexus.org Applied Monitoring Vision and Mission Intentional design IMPACT Boundary Partners OUTCOMES OUTPUTS ACTIVITIES INPUTS CONTROL INFLUENCE Time Inspired by Jeff Conklin, cognexus.org

Applied Monitoring Time Vision and Mission IMPACT OUTCOMES OUTPUTS Intentional design IMPACT OUTCOMES OUTPUTS Performance Your programme’s functioning as an organizational unit ACTIVITIES INPUTS Time Inspired by Jeff Conklin, cognexus.org

Inspired by Jeff Conklin, cognexus.org Applied Monitoring Vision and Mission Intentional design IMPACT OUTCOMES Strategy The effectiveness of how you are attempting to influence boundary partners OUTPUTS ACTIVITIES INPUTS Time Inspired by Jeff Conklin, cognexus.org

Applied Monitoring Time Vision and Mission IMPACT OUTCOMES Outcomes Intentional design IMPACT OUTCOMES OUTPUTS Outcomes Change you influence in the boundary partners ACTIVITIES INPUTS Time Inspired by Jeff Conklin, cognexus.org

Adapted Monitoring For example… You have no baseline Are in complex situations where relations of cause and effect are unknown Are a non-hierarchal organisation You may fruitfully use Outcome Mapping to monitor your results if you... Have not done an intentional design and therefore do not have a baseline Are primarily interested in understanding the process through which you are influencing change Operate in complex situations in which cause and effect are unknown and therefore unexpected effects are common Are a non-hierarchal organisation in which success depends on units and individuals operating with autonomy and little control

Situations in which this… Vision and Mission Intentional design IMPACT OUTCOMES OUTPUTS ACTIVITIES INPUTS Time Inspired by Jeff Conklin, cognexus.org

Looks like this

Vision and Mission OUTCOME OUTPUT OUTCOME OUTPUT OUTCOME OUTCOME ACTIVITY OUTCOME OUTCOME OUTPUT OUTCOME ACTIVITY OUTPUT OUTPUT INPUTS OUTCOME ACTIVITY OUTCOME INPUTS ACTIVITY OUTPUT INPUTS INPUTS

You monitor by Watching social actors for changes in their behaviour, relationships, actions, policies or practices Focusing on the outcomes you are achieving Working backwards from outcomes to determine how you contributed to that change, rather than attempt to attribute the change to your activities And then deciding how to improve strategy and organisational performance in order to achieve outcomes Focus on what you achieve rather than what you do You ascertain how you contributed rather than attempt to attribute the change to your activities Monitor boundary actors. You monitor social actors for changes in their behaviour, relationships, actions, policies or practices. The focus of monitoring should be on the social actors you are attempting to influence and not on the activities you have carried out. This is another distinction compared to conventional monitoring. Monitor from outcomes to activities. When you have identified a change, you then work backwards to determine how you contributed to that change. Conventional evaluation (and by implication conventional monitoring) attempts to understand if what was planned is what resulted, or if this logic bears fruit: inputs -> activities -> outputs -> outcomes -> impact. In complex and dynamic situations where what results, and even what was done, was often not pre-planned, this logic is not applicable. Instead, using the approach of the coroner examining the cause of death, the criminal investigator the author of a crime, or the epidemiologist the source of a disease, the outcome mapper first identifies the outcome and then works back to figure out what activities contributed to it. You map the process of changes to which you are contributing And then make decisions on how to improve your strategy and organisational performance

Evaluation is the least developed of the three stages of Outcome Mapping, formally at least. Limited to designing the Evaluation Plan.

Elements of an Outcome Mapping Evaluation Plan EXAMPLE EVALUATION PLAN Evaluation Issue: Results Achieved by Recipient Research Institutions Who Will Use the Evaluation? How? When? Questions Information Sources Evaluation Methods Who Will Conduct and Manage the Evaluation? Date (Start & Finish) Cost Executive director to use findings to inform program proposal What influence has the institution had on research users and other researchers? Progress Markers & Outcome Journal Trip reports Strategy Journal Key informant interviews with program staff Senior Consultant with regional and sectoral expertise Six months 35K www.idrc.ca/evaluation

But as with monitoring, the outcome concept is wonderfully adaptable. It can potentially serve the three principal evaluation modes.

Michigan Association for Evaluation Annual Conference May 3, 2000 Formative evaluation “When the cook tastes the soup, that’s formative evaluation.” (Bob Stake) Provides information for program: improvement modification documentation management Provides feedback to improve program effectiveness before the end of the project. - Michigan Association for Evaluation, provided by John Seeley Michigan Association for Evaluation Annual Conference

Michigan Association for Evaluation Annual Conference May 3, 2000 Summative evaluation When the guest tastes it, that’s summative evaluation.” (Bob Stake) Conducted after the project is completed. Provide evidence to support judgment of the project’s worth. Is used in decisions about the merits of continuing the project or program. - Michigan Association for Evaluation Michigan Association for Evaluation Annual Conference

Developmental evaluation “When a guest and a cook concoct a soup together, that co-creation is developmental evaluation.” (Michael Quinn Patton) Supports innovation to develop a programme that is adapting to emergent and dynamic realities in complex environments. Infuse the development team discussions and decision-making with evaluative questions, thinking and data.

Examples of formal evaluations Treatment response for problematic use of ecstasy, ketamine and gamma-hydroxybutyrate in Australia Six civil society projects in Bosnia and Herzegovina Community development program with Roma and non-Roma communities in Bosnia and Herzegovina OM applied as a planning, monitoring and evaluation tool to the Ceja Andina project in Carchi, Ecuador A working group on 'values’ Use of OM to recommend a treatment response for problematic use of ecstasy, ketamine and gamma-hydroxybutyrate in Australia, by Josephine Norman Outcome Mapping evaluation of six civil society projects in Bosnia and Herzegovina, by Steve Powell Evaluation of community development program with Roma and non-Roma communities 2006-8 in Bosnia and Herzegovina, by Steve Powell OM applied as a planning, monitoring and evaluation tool to the Ceja Andina project in Carchi, Ecuador, by Kaia Ambrose Evaluation of a working group on 'values’, by Daniel Roduner, Switzerland All available through the The OM learning community website: www.outcomemapping.ca.

Examples of adaption of Outcome Mapping in evaluation 7 international networks – 2 based in Latin America, 2 in Europe, 2 in Asia, and I in the USA. 8 programmes of five grant makers – Oxfam Novib, IDRC, Ford Foundation, Hivos, and the Open Society Institute Their 130+ social change and development grantees Outcomes not predefined Since 2003 I have used the Outcome Mapping concept of outcomes in evaluations of: 7 international networks – 2 based in Latin America, 2 in Europe, 2 in Asia, and I in the USA. 8 programme s of five grant makers – Oxfam Novib, IDRC, Ford Foundation, Hivos, and the Open Society Institute assessing the outcomes of 131 social change and development grantees (NGOs, community based organisations, networks, research centres working on a wide range of issues – environment, human rights, agriculture, health, ICT, democracy) No one had predefined the vast majority of outcomes that we harvested.

Evaluation ■ Internal outcomes ■ External outcomes ■ Their significance ■ The organisation’s contribution ■ Internal Outcomes – The changes you are influencing in the key internal actors of your non-hierarchal, autonomously managed organisation, alliance, partnership or network. ■ External outcomes – The changes you are influencing – from inspiring to pressuring, passing through supporting, assisting, funding or in any way aiding – social actors in the world outside. In both cases you can identify as well: ■ Significance – Why is each change important? How does it complement counterbalance or undermine other outcomes?

In these evaluations we “harvested” descriptions of who changed what, when and where, its significance and the contribution of the organisation. In addition, we substantiate the outcomes with independent third parties who have a working knowledge of the change and how it came about. We mapped processes of change to understand how outcomes reinforce or undermine each other. In evaluation we “harvest” the same information as when monitoring (description of who changed what, when and where, its significance and how the organisation had contributed) In addition, we have substantiated the outcomes with independent third parties who have a working knowledge of the change and how it came about And in some cases of mapped processes of change to understand how outcomes reinforce or undermine each other.

Thank you! For more information, examples of use and to share your experiences www.outcomemapping.ca www.idrc.ca/evaluation Ricardo.Wilson-Grau@inter.nl.net