Complexity and Outcome Harvesting

Most recently In June the World Bank published 10 case studies and a toolkit for using Outcome Harvesting http://wbi.worldbank.org/wbi/document/cases-outcome-harvesting http://www.outcomemapping.ca/resource/resource.php?id=452

e Harvesting Origins

Inspired and informed by the Outcome Mapping methodology. e Harvesting Inspired and informed by the Outcome Mapping methodology. A tool for practitioners operating in dynamic, uncertain situations to monitor and evaluate the social change results they are achieving.

Outcome Harvesting Six steps: Design the harvest — users => uses => evaluation questions => data to be collected Review documentation and draft outcomes => intervention’s contribution Engage with informants — generally within the intervention Substantiate with independent but knowledgeable third parties Analyse, interpret — What? + So What? but not Now What? Support use of findings TRIANGULATION

International social change networks GLOBAL PARTNERSHIP FOR THE PREVENTION OF ARMED CONFLICT

International development funders

What did all of these organisations have in common? COMPLEXITY! What did all of these organisations have in common?

Relationships of cause and effect are KNOWN Simple Relationships of cause and effect are KNOWN Now I will link the cartoons with the fish soup story with complexity. To explain what I mean by complexity, I find it helpful to consider it’s opposite. A SIMPLE situation is when you want to plan to reach the island and you can calculate with considerable certainty the distance, the difficulty and therefore energy and time it will take you to swim or row or sail or tool over in a motor boat. It is simple because the relationships of cause and effect between what you will do and what you will achieve are KNOWN 9

M&E of a simple intervention Vision Annual polio vaccination campaign IMPACT OUTCOMES Plan OUTPUTS ACTIVITIES For example, the monitoring and evaluation of a simple development situation such as the annual polio vaccination campaign…EXPLAIN INPUTS Time Inspired by Jeff Conklin, cognexus.org

Relationships of cause and effect are UNKNOWN Complex Relationships of cause and effect are UNKNOWN A COMPLEX situation, in contrast, is his scenario. You want to get to the island but you cannot calculate when you will arrive or even if swimming or rowing or sailing or motor boating will be most effective. Relationships of cause and effect are UNKNOWN 11

M&E of a complex intervention Strengthening nonviolent responses to communal conflict in the Horn of Africa OUTCOME Vision OUTPUT ACTIVITY OUTCOME Time OUTCOME OUTPUT OUTCOME ACTIVITY OUTPUT OUTPUT INPUTS OUTCOME You may have a plan for which you have obtained resources but as soon as you begin to implement it, the activities change and their outputs too. Some lead to outcomes and other do not. And outcomes emerge in the most unexpected places. So in these situations, one or more of the five types of Developmental Evaluation will be appropriate. ACTIVITY OUTCOME INPUTS ACTIVITY OUTPUT INPUTS INPUTS Plan

Dimensions in which the relationships of cause and effect are known NOT EITHER OR OUTPUT OUTCOME INPUTS ACTIVITY Time SIMPLE Dimensions in which the relationships of cause and effect are known Importantly, not everything is complex. There are dimensions in which the relationships of cause and effect are known. 13 13

NOT EITHER OR COMPLEX Unknown relations of cause and effect dominate. OUTPUT OUTCOME INPUTS ACTIVITY Time COMPLEX Unknown relations of cause and effect dominate. Results are substantially unforeseeable. But in complex situations the most important relationships of cause and effect are unknown and therefore the results substantially unforeseeable. And if you do not distinguish correctly the difference between simple and complex dimensions…. 14 14

Outcome Harvesting and complexity High The greater the: 1. Disagreement about what is the development challenge 2. Disagreement about what is its solution 3. Uncertainty about what will be the results of your actions to solve the development challenge The more Outcome Harvesting may be useful COMPLEXIMETER What I find helpful for monitoring and evaluation from Snowden’s and Zimmerman’s complexity models is they identify three factors of complexity: 1. The higher the disagreement about a) what is the development challenge or b) its solution, the greater the complexity. 2. And c) the higher the uncertainty about what will be the results of your actions to solve the development challenge, also the more complexity you face. Degrees of disagreement about what to do and why, and what will be the results, underline that there are both subjective and objective dimensions to complexity because because it depends on what you know at a given moment of time and how true is that knowledge. Low

Michigan Association for Evaluation Annual Conference Formative evaluation Michigan Association for Evaluation Annual Conference May 3, 2000 Summative evaluation Developmental evaluation Developmental Evaluation is not, however, the same as evaluation of development. - “When the cook tastes the soup, that’s formative evaluation.” (Bob Stake) -When the guest tastes it, that’s summative evaluation.” (Bob Stake) “When a guest and a cook concoct a soup together, that co-creation is Developmental Evaluation.” (Michael Quinn Patton) - Developmental Evaluation is easily confused with international development evaluation. They are not the same, though Developmental Evaluation can be used in the evaluation of development. Developmental Evaluation supports the development of innovation, which can include an economic, political, social, cultural, technological, environmental or other type of development intervention. Development evaluation is about evaluating programmes, usually in developing countries, sponsored or funded through the different mechanisms of international development co-operation. Michigan Association for Evaluation Annual Conference

When do you evaluate? Developmental Formative Summative Time Progress of intervention Or instead would face this situation of uncertainty that they were going to have the intended results. Time

Long-term Goal (Impact) Objectives or Outcomes RESULTS BASED M&E INTENTIONAL DESIGN Long-term Goal (Impact) STEP 1: Vision Objectives or Outcomes STEP 2: Mission Indicators STEP 3: Boundary Partners Outputs Indicators STEP 4: Outcome Challenges Activities STEP 5: Progress Markers Inputs STEP 6: Strategy Maps

Long-term Goal (Impact) STEP 1: Vision GLOBAL PARTNERSHIP FOR THE PREVENTION OF ARMED CONFLICT Long-term Goal (Impact) STEP 1: Vision Objectives or Outcomes STEP 2: Mission Indicators STEP 3: Boundary Partners Outputs STEP 4: Outcome Challenges Indicators They tried but found up to 15 progress markers per International networks in which time is at a premium Activities STEP 5: Progress Markers Inputs STEP 6: Strategy Maps

Long-term Goal (Impact) Objectives or Outcomes ICT4D in Asia PAN and Africa ACACIA Nigeria Evidence‐based Health System Initiative EcoHealth Fieldbuilding Leadership Initiative Consorcio por la Salud, Ambiente y Desarrollo Long-term Goal (Impact) STEP 1: Vision Objectives or Outcomes STEP 2: Mission Indicators STEP 3: Boundary Partners Outputs STEP 4: Outcome Challenges Indicators They tried but found up to 15 progress markers per International networks in which time is at a premium Activities STEP 5: Progress Markers Inputs Regional Peacebuilding Programme in the Horn of Africa STEP 6: Strategy Maps

Long-term Goal (Impact) Objectives or Outcomes Art and Culture Programme in Central America Long-term Goal (Impact) STEP 1: Vision Objectives or Outcomes STEP 2: Mission Indicators STEP 3: Boundary Partners Outputs STEP 4: Outcome Challenges Indicators They tried but found up to 15 progress markers per International networks in which time is at a premium Activities STEP 5: Progress Markers Inputs STEP 6: Strategy Maps

https://undp.unteamworks.org/node/370238 In sum https://undp.unteamworks.org/node/370238 Outcome Harvesting is “an evaluation approach that does not measure progress towards predetermined outcomes, but rather collects evidence of what has been achieved, and works backward to determine whether and how the project or intervention contributed to the change.” - UNDP

Discussion Have any of you found other ways to apply the principles of Outcome Mapping to evaluate projects, programmes or organisations when the elements of intentional design have been missing?