Outcome Mapping Planning, Reflecting & Learning Shalini Kala July 2011

Outline OM: basic principles Overview of the steps An example of OM application My experience of using OM Q&A and short exercises in between

Acknowledgements This presentation makes use of materials from Simon Hearn and Terry Smutylo available at www.outcomemapping.ca

OM history 1990s: post-Rio need to demonstrate ‘sustainable’ results 1998: Barry Kibel and Outcome Engineering 1999: Methodological collaboration with projects 2000: Publication of manual in English 2002: Training, facilitation & usage globally 2006: OM Learning Community www.outcomemapping.ca 2008: Latin America Centre for Outcome Mapping (CLAMA) 2010: East Africa and beyond In

Evaluation challenges that led to the development of OM How do you measure what difference research is making How do you establishing cause & effect in open & complex system we work in Timing: when do you evaluate Iterative learning IDRC and its partners developed OM in response to some major evaluation challenges in 1990’s Timing : early, but impact hasn’t happened as yet; later, many actors and factors have contributes Learning: We wanted to encourage learning not just proving “this is what happened”, how do we improve as we go

The research to impact connection is complex Recognizing that connecting research to impact is complex, IDRC decided to focus on outcomes and not impact in the results chain.

Focus of Outcome Mapping Program influence decreases Community ownership increases OM defines outcomes very particularly as change in behavior – in a very broad sense - relationships, actions, activities - anything someone is doing. That’s the niche/focus of OM Important to focus on outcomes… Taking again our simplified results chain, we can map the ownership of our local partners or beneficiaries. We see that this increases. At the level of inputs, our partners don’t have a lot of say – decisions about money, who to work with and where to work are largely made by the program or even the donor. But at the impact level, it all depends on the partners and the benficiaries. Outcome mapping is focussed at the level of outcomes. This is because the partner ownership is high enough to be able to see an effect, some observe changes and the programme influence is high enough to be able to say with some confidence that those changes originated from the intervention. Inputs Activities Outputs Outcomes Impacts Outcome Mapping

Key ideas characterizing OM Looking at the bigger picture Recognizing that change is: continuous, multi- dimensional, complex, non-linear Learning as you go Contribution not attribution Outcome mapping is built on systems thinking: you are part of the big picture Embraces change: you don’t ignore or control because that is your context but try to manage the change as you go, be adaptable and flexible You have to be learning as you go, paying attention to what is happening not only to what you want to accomplish but how things are progressing as you go We were not going to look at causal attribution but influence and contribution that we make

Brief definition of OM A participatory method for planning, monitoring and evaluation Focused on changes in behaviour of those with whom the project or program works Oriented towards social & organizational learning A way to clarify intent: what are you trying to do? Why? With whom? How would you know that you are doing what you intended In a results’ chain OM helps you think about the « outcomes » section Definition of Program: A group of related projects and activities with a specific set of resources (human, financial, capital) directed to the achievement of a set of goals within a specified period of time Can be used by projects, organizations, and communities too

Three key concepts in OM Sphere of influence Boundary Partners Outcomes understood as changes in behaviour OM does three innovative things that add value to existing PM&E methods: Defines the system borders, roles and responsibilities where the program operates; Identifies the prominent actors who are the ongoing drivers of the changes; and Sets milestones that mark the path of change;

There is a limit to our influence Project Partners Beneficiaries The way a focus on measuring ‘impact’ plays out is not suitable in the context of many projects and programmes -> we need to recognise the limits of a project’s influence, and shape our planning, learning, and accountability functions around “outcomes”, which are further ‘upstream’ from impacts. Limits depend on time, geography, resources, contacts, politics Looking from the point of view of a project, we see Sphere of control = operational environment Sphere of Influence = Relationships & Interactions Sphere of Interest = social, economical, environmental states & trends DIRECT CONTROL DIRECT INFLUENCE INDIRECT INFLUENCE This relates to concepts you may be familiar with from the log frame, along the results chain through to intended impacts. The premise is -> we can’t control everything we’d like to see change -> this is not something unscientific: complexity theory (and common sense!) tells us that real, sustainable change involves the combination of a number of different factors, and is a product of the interaction of many different actors and stakeholders -> Outcome Mapping is concerned with the level where a programme has direct influence Complexity cross-reference: Systems with multiple actors, inter-related and connected with each other and with their environment Various forces interacting with each other, interdependent (e.g. political and social dimensions) In these situations, change occurs because of the interaction of multiple actors and factors; can’t be controlled by one programme Very difficult to predict what ‘impacts’ might be achieved in advance; SDOIC means inherent unpredictability, that isn’t unscientific but based on careful investigation Common mistakes include trying to deliver clear, specific, measurable outcomes; better to work with inevitable uncertainty than to plan based on flimsy predictions Russell Ackoff : 3 kinds of problems: Mess, problem and puzzle. MESS has no defined form or structure, not a clear understanding of what’s wrong, often involves economic, technological, ethical and political issues. Common mistake is to carve off part of a mess, deal with it as a problem and solve it as if it was a puzzle (as the simple causal chain from inputs to impact tries to do) -> need to recognise messy realities Sphere of interest Sphere of control Sphere of influence

Changes in behavior as ‘outcomes’ Inputs, activities, outputs Outcomes: Changes in behavior Impact: Changes in state The way a focus on measuring ‘impact’ plays out is not suitable in the context of many projects and programmes -> we need to recognise the limits of a project’s influence, and shape our planning, learning, and accountability functions around “outcomes”, which are further ‘upstream’ from impacts. Limits depend on time, geography, resources, contacts, politics Looking from the point of view of a project, we see Sphere of control = operational environment Sphere of Influence = Relationships & Interactions Sphere of Interest = social, economical, environmental states & trends DIRECT CONTROL DIRECT INFLUENCE INDIRECT INFLUENCE This relates to concepts you may be familiar with from the log frame, along the results chain through to intended impacts. The premise is -> we can’t control everything we’d like to see change -> this is not something unscientific: complexity theory (and common sense!) tells us that real, sustainable change involves the combination of a number of different factors, and is a product of the interaction of many different actors and stakeholders -> Outcome Mapping is concerned with the level where a programme has direct influence Complexity cross-reference: Systems with multiple actors, inter-related and connected with each other and with their environment Various forces interacting with each other, interdependent (e.g. political and social dimensions) In these situations, change occurs because of the interaction of multiple actors and factors; can’t be controlled by one programme Very difficult to predict what ‘impacts’ might be achieved in advance; SDOIC means inherent unpredictability, that isn’t unscientific but based on careful investigation Common mistakes include trying to deliver clear, specific, measurable outcomes; better to work with inevitable uncertainty than to plan based on flimsy predictions Russell Ackoff : 3 kinds of problems: Mess, problem and puzzle. MESS has no defined form or structure, not a clear understanding of what’s wrong, often involves economic, technological, ethical and political issues. Common mistake is to carve off part of a mess, deal with it as a problem and solve it as if it was a puzzle (as the simple causal chain from inputs to impact tries to do) -> need to recognise messy realities Sphere of interest Sphere of control Sphere of influence

Increased knowledge of techniques Farmers adopting drip irrigation Participating farmers learn how to use drip irrigation equipment Reduced numbers of new wells Increased knowledge of techniques Farmers participate in field trials Extension workers visit demonstration farms Farmers adopting drip irrigation methods Participatory research on demonstration farms to develop approaches to drip irrigation Training of extension workers Greater quantities of groundwater available Example... Extension workers promoting drip irrigation Publication of performance of different set-ups Source: Terry Smutylo

Who are your boundary partners? Programme Beneficiaries Stakeholders Boundary Partners

To what extent do these ideas apply to the programs and context you work in?

4 Key Planning Questions Why? Who? What? How? Vision Boundary Partners Outcomes Challenges, Progress Markers How does OM respond to these? Not go through these now – three day workshop Mission, Strategy Map, Organizational Practices 16

Step 1: Vision improved human, social, & environmental wellbeing This is the forest

A vision statement…. guides motivates is an accountability-free zone It is a description of the your wild dreams about program success and speaks to the world that you are seeking to create.

Step 2: Mission One tree in the forest (vision) that the project will nurture The mission is that “bite” of the vision statement on which the program is going to focus.

A mission statement describes… How the program intends to apply its resources in support of the vision The areas in which the program intends to work How the program will support the achievements of outcomes by its direct partners What do you do? Who are your principle collaborators? How do you work with them?

Swayamsiddha Project Context: Women in India are disempowered Project: Started in 2000, closed in 2005 Funded by CIDA and IDRC Managed by national NGO and local NGOs Aims: Develop network of government, non-governmental and community based organisations Increase gender responsiveness in local health care, families and community institutions Decreasing drudgery in women’s and girl’s work Increase access to and control of financial services Context: maternal mortality, literacy, underage marriage, malnutrition, legally, wages

Swayamsiddha Vision Across rural India, women and girls utilize and benefit from appropriate health care, education, food and water security and freedom from violence. Women have access to the markets, credit, banking and municipal services they need to pursue their livelihood goals. They use drudgery-reducing technologies and agricultural inputs that contribute to personal well-being and to ecological sustainability. Villages are fully served by public transport, are well lit at night and police enforce all laws fully and equitably. Girls attend school full time and families have the information and resources to make informed decisions regarding their health, safety and social needs. Gender equity governs household labor and decision-making; and men in the community understand and support gender-responsive laws. BAIF project worked in 6 states in India, where there is a wide range of gender inequality: Dreaming was not an option. 40 million fewer women than men Sex ratio is 927 women per 1000 men. literacy rates among women 3-50% up to 35% of girls under 18 are married Maternal mortality rate is highest in the world: 5 die per 1000 live births; 12% of deaths of women of reproductive age 40% lower wages than men Little control over health and fertility; Legally discriminated against in land & property rights

Swayamsiddha Mission The Swayamsiddha Project works with governments, NGOs & CBOs to improve women’s health and empowerment. It facilitates the development of women’s self help groups. It provides them with funding and training to help them influence community and government services to be more responsive to their health and livelihood needs. It fosters mutual respect and joint action between these self-help groups and: banks; police; health and social service providers; and government agencies. It researches and promotes the application of ecosystem approaches to human health in agriculture and in the provision of health and sanitation services. Swayamsiddha addresses equity issues in all its activities. It uses participatory methods to monitor progress, to learn how to become more effective in supporting its partners and to report on its results.

Step 3: Boundary Partners Those individuals, groups, & organizations with whom a program interacts directly to effect change & with whom the program can anticipate some opportunities for influence.

Swayamsiddha BPs BAIF IDRC CIDA Families Banks PHCs SHG Police Community Leaders State NGO Why boundary? – program doesn’t control them, can only influence; therefore the program is on the boundary of their world Program provides new tools, tech, opps or resources to its BPs but it doesn’t control them (what they do wit these resources); respnsiblity for change rests with the BPs Shows that a program’s influence on development (in the real world) happens through its partnerships Whose influence do you want to understand? OM (and nested BPs) gives a way to identify the level which you want to evaluate behavioural change Conduct stakeholder analysis to identify key actors, select boundary partners and identify their BPs, identify strategic partners (funders, training providers, experts who have knowledge to help BPs) BAIF IDRC CIDA

Questions

Group exercise 1: 15 minutes Get together in small groups with your boundary partners present here Come up with a vision & mission statement for your project Identify two levels of boundary partners of your project You might want to start by thinking of key words or ideas that should feature in these statements

Step 4: Outcome Challenge Describes behaviour of a single boundary partner Sets out the ideal actions, relationships, activities Describes the boundary partner’s contribution to the vision What should the BP be doing to contribute to vision? With whom would they be interacting How could this BP contribute maximally to the vision?

Outcome Challenge for Swayamsiddha SHG Women’s self help groups are taking action to make community and government services more responsive to the health and livelihood needs of women and girls. They influence banks, police, health and social service providers, local officials and state and national government agencies in relationships of mutual respect and joint action to improve women’s well being. Women’s self help groups arrange bank loans for members and for life skills training for girls to be included in the school curriculum. They influence local, state and national government policies and expenditures on community improvement and transportation and support women candidates to run for election to local government office. Found in Exercise 14B - Design Worksheet 1

Step 5: Progress Markers Love to see (Deep transformation) Like to see (Active engagement) Not a lock step progression Indicate DEPTH of change; signals that change is happening Milestones Spectrum of transformation OM challenge defines the deep transformation Expect to see (Early positive responses)

Progress Marker Checklist Each Progress Marker: Describes a changed behaviour by the boundary partner Can be monitored & observed As a set, Progress Markers: Are graduated from preliminary to more profound changes in behaviour Describe the change process of a single boundary partner Progress markers will be different for each BP but inter-woven.

Why Graduated Progress Markers? Articulate the complexity of the change process Allow negotiation of expectations between the program and its partners Permit on-going assessment of progress Encourage the program to think about how it can intentionally contribute to the most profound transformation possible Help identify mid-course corrections and improvements OM challenge can be really ambitious. Through progress markers you establish directionality and the steps envisaged to indicate if the project is moving in the direction of the OM challenge.

X

Sample progress markers: Swayamsiddha SHG Expect to See Women’s SHG’s: Holding meetings regularly Discussing a list of shared concerns contributing to a group bank account Like to See Women’s SHG’s: Soliciting training in maternal & child health for its members Lending money to Acquiring skills in managing credit programs Taking action responding to incidents of domestic violence Love to See Women’s SHG’s: Lobbying local government for expenditures on community improvements Approaching the State Transport Dept for bus service to their villages Putting forth candidates for election to local government council

Group exercise 2: 20 minutes Get together with your boundary partners present here Come up with an outcome challenge statement Come up with a list of associated progress markers

Step 6: Strategy Map I E Causal Persuasive Supportive I = Things that you do directly with your partners E = something that you can do in the environment of the boundary partner I-1: $, land titles, vaccinations, mosquito netting E-1: trash cans, no smoking policy, gender element in proposal to receive funds, I-3 - can support all the other strategies E-3 partnerships and networks that sustain change when program ends (in Swayamsiddha case, police chiefs were networked)

6 kinds of strategies I E causal persuasive supportive aimed at individual boundary partner strong influence arouse new thinking; build skills, capacity on-going support E aimed at the boundary partner`s environment Alter the physical, regulatory, or information environment Broad information dissemination; access to new info Create or strengthen peer networks A diversified set of strategies would be needed to lead to sustainable change

Step 7: Organisational Practices Prospecting for new ideas, opportunities, and resources Seeking feedback from key informants Obtaining the support of your next highest power Assessing and (re)designing products, services, systems, and procedures Checking up on those already served to add value Sharing your best wisdom with the world Experimenting to remain innovative Engaging in organizational reflection How does your team or organisation stay relevant, viable and effective? Taken together, the Ops describe a well-functioning program OPs relate to the program’s effectiveness (not just its efficiency) 1 – not being satisfied with the status quo 2 – how are we doing? What could we do differently or better? 3 - Keeping your manager, BoG, donor, community leader informed of your program and supportive of your work 4 – keeping up with your and your partners’ needs 5 – not just surveillance; monitoring in order to find out what is happening and provide more support 6 – bringing together and disseminating your knowledge and learnings 7 – creating space to try things differently / in a new way 8 – being a learning organization; reflecting on performance in order to improve

Outcome Mapping: Main Elements Vision Mission BOUNDARY PARTNERS Outcome Challenges Progress Markers Strategies Organizational Practices

OM principles of use Flexible: modular to be adapted to use and context Complementary: can be combined with other methods Evaluative: promotes culture of reflection, results oriented thinking, & social & organizational learning Participatory: seeks dialogue and collaboration with partners

Quick Questions

OM in real life Experience of networking actors to reduce rural poverty in Asia

ENRAP Knowledge Networking for Rural Development in Asia-Pacific Region Started in 1998, covering 5 countries-15 projects, expanded to 8 countries (40 projects), ended in March 2011 covering all of Asia-Pacific (over 65 projects) IFAD-IDRC collaboration Goal: Increased sharing of knowledge & information for rural poverty reduction in Asia. Objective: IFAD partners are using networks to actively share knowledge and information. Improved knowledge sharing was directed at supporting project performance

Worked through IFAD projects & later country offices ENRAP promoted knowledge sharing through network building across projects & among rural dev. players at the regional and national level. IFAD poverty reduction projects & grant projects National government agencies & NGOs People’s organizations (farmer federations) Research institutions Worked through IFAD projects & later country offices Depended on supportive environment at IFAD HQ for effectiveness Lack of clarity and common understanding on KM, its usefulness to IFAD operations, and if and how ENRAP was supporting the KM agenda. Division director, KM facilitator, CPMs….

Why did we choose OM? Project aimed at behavior change, in this case: networking for knowledge An evolving KM environment at IFAD A relatively small player (attribution vs. contribution) 1-2m for portfolio of over 450m 3-year phases External As the learning framework Change behaviour

What did we do? Adapted OM pragmatically Used along with LFA Worked with small number of boundary partners over email to draft the initial outcome map Revised the outcome map based on M&E survey result after one year of programming Contracted an M&E agency for annual monitoring

How did we use OM? To monitor progress Tracking change in behavior, in gen. Tracking comfort of network members with new technology (social media tools, internet, etc.), in particular To learn along the way to inform project mgt. To validate our perceptions Quantities (websites, mailing lists) not good indicators of assessing change. Needed qualitative assessment. Who are boundary partners and who are strategic ones? CPOs playing a lead role in national network facilitation; networks are growing and KS is happening through them

Benefits It works for better project management Able to demonstrate progressive change As a concept, easier to understand & apply Greater effectiveness in learning &monitoring because of the participatory nature Adaptable – we used what we could Growing community of users, very ready to help Can capture behaviour change in a relatively short period of time

Challenges Partner’s lack of understanding of OM Time & resource intensive Relatively less known Training is expensive Trainers are few Not a whole lot of organizations have experience with it Convincing partners requires time & effort Defining boundary partners; updating outcome map based on M&E survey Adapted to reduce time and resource intensity to a very workable level IFAD, IPGRI – lot of questions around comparing with LFA A young and at the cutting edge methodology. Not many people/organizations have experience with it. Not many trainers. Training is still quite costly.

OM resources The outcome mapping site www.outcomemapping.ca The Outcome Mapping guide Some links to other resources: New OM community book: Making Outcome Mapping Work: Innovations in Participatory Planning, Monitoring and Evaluation Integrating OM and LFA: VVOB Zimbabwe, VECO Indonesia, LFA and OM Synthesis: A discussion paper Monitoring and Evaluation: Considerations for learning-oriented Monitoring and Evaluation with Outcome Mapping, Guiding notes for facilitators: helping to sketch out an M and E system Step Zero Various toolkits produced by RAPID, particularly, Tools for Knowledge and Learning and Mapping political context (also see this recent Briefing Paper that describes the RAPID Outcome Mapping Approach) Barefoot Guide to working with organisations and social change

Questions