Assessing Contribution and Enhancing Social Learning: An Introduction to Outcome Mapping (OM) 16 December 2004 Kaia Ambrose

Presentation Outline OM in Carchi, Ecuador. Underlying logic of OM. Brief outline of stages of OM. Some key concepts - boundary partners, their changes and how we measure that change. OM and learning organizations. The OM toolbook. Conclusions and ideas for the afternoon session.

First, a little context… Ecopar Ecuadorian NGO focused on research, training and capacity building in tropical ecosystems/life zones. The Ceja Andina Project IDRC-funded. Sustainable use of agriculture and forest biodiversity in the ceja andina (cloud forest) region of the northern Ecuadorian Andes. Research-oriented (traditional and participatory), social learning processes, policy development and strengthening of local government. Developing OM since March 2002 – “walking it through and making it make sense in our context”.

M&E concerns in the Ceja Andina Project As a team and as an NGO, how can we learn from the project in order to improve the way we “do” development (in a research context)? How can we become a learning organization while still being accountable? How can we learn (iterative processes) together with our local partners? How can we bring creativity and real interest into the M&E process? (create culture of M&E) What is the balance between rigor and utility in our M&E objectives? Assess need for OM (complementary, bits of). Learning / accountability, process / product, rigor / utility, efforts / results. Encouraging iterative learning, timing, measuring development results of research. Evaluation in support of learning; learning as primary outcome of development program evaluation. Being attentive along the journey. Document, learn, report, share.

What is Outcome Mapping? an integrated and participatory M&E approach; also for planning. an approach that views outcomes as changes in the behaviour, relationships, or actions of partners. Development is accomplished through changes in the behaviour of people. a methodology that characterizes and assesses the program or project’s contributions and influence to the achievement of outcomes. Focus on behavioural change (logically linked to program activities, although not necessarily caused by them) – development is done by, and for people. Project can influence the achievement of outcomes, not control them because the ultimate responsibility lies with the people affected. Direct partners control their own development and changes; as development organizations, our role to facilitate that process. WE must delegate power and responsibility to endogenous actors. an approach for designing M&E in relation to the broader development context but assessment is within your sphere of influence.

Program / project’s Sphere of Influence The Real World PROGRAM (performance and strategies) Contributions to outcomes – outcomes in turn enhance possibilities for development impacts. Complexity of development process. For each change in state, there are correlating changes in behaviour. = Program’s partners (behavioural change)

Outcome Mapping does not: Focus on impact (we’re not in the impact business, we’re in the business of change). Look for attribution (we look for contribution). Isolate contributions of program / project in achievement of results. Promote linear, cause-effect thinking in a “sterile” context. Focus solely on actions promoted by program / project. Concepts of attribution and impact limits the possibilities of extrapolating lessons learned of evaluation and development initiatives (because they isolate key factors). Impact occurs downstream; confluence of events and actors. We must recognize contributions by other actors. Attribution and measuring downstream results are dealt with through a more direct focus on transformations in the actions / behaviour of actors. Bureaucratization not compatible for social learning, sharing visions, values, responsibility, innovation, risks. Attribution to donors does not take into account multiple endogenous contributions and conditions for sustainable development. Lineal forms of thought such as cause-and-effect, oppose the concept of development understood as a complex process that is produced in open systems. Drive to claim credit interferes with creation of knowledge (how & why impact occurs). Attribution of impact does not allow for multiple endogenous contributions and conditions necessary for sustainable development. Ignores the need for endogenous organization and communities to take ownership of program components. Cannot pre-plan development.

Influencing Outcomes Partners Program Assessing Changes in Behaviour Assessing Program Influence Assessing Internal Performance

Outcome Mapping                 Design stage, mid-term or ex-post.

OM can help us: Besides designing or clarifying programme logic, helps record and assess monitoring data: How far have our boundary partners progressed towards achieving outcomes? What are we doing to support the achievement of outcomes? Indicate cases of positive performance and areas of improvement; front stage and back stage. Evaluate intended and unexpected results; positive deviance. Gather data on the contribution that a program made to bringing about changes in its partners (strategies - new tools, aptitudes, knowledge). Establish evaluation priorities and evaluation plan. Evaluate capacities and how the project is supporting the development of these capacities. Reinforce endogenous capacity in order to continually respond to change.

Boundary Partners – Core ideas Boundary partners – individuals, groups, organizations with whom the program interacts directly and with whom the program anticipates opportunities for influence. Outcome challenges – description of the ideal changes in the behaviour, relationships, activities and/or actions of a BP that contribute to vision. Progress markers – a set of graduated indicators of changed behaviours for a BP that focus on the depth and quality of change. Program works with them to effect change; does not control them. Boundary partners – incorporates concept of endogenous knowledge. Recognizes that BP control change; development programs only facilitate process by providing access to new resources, ideas or opportunities. Devolve power and responsibility to endogenous actors. Humans central concept of OM; program’s contributions to development are planned and assessed based on its influence ont eh partners with whom it is working to effect change. For each change in state there are correlating changes in behaviour. Focusing M&E on the program’s BP makes it possible to obtain useful feedback about the program’s performance and results within its sphere of influence = improve performance.

Boundary Partners Social change is about relationships. We’re all interconnected. Social learning – create consensus, among multiple cognitive beings, regarding direction of program in terms of influencing outcomes, in order to arrive at concerted action. Boundary partners participate in process of OM (in M&E). People are cognitive beings, with different ways of viewing and acting in the world; as evaluators, we need to understand those different ways. Acknowledge and explain different perspectives. Identify and develop our interconnectedness. OM as a tool to engage with other actors in society. Present real-life pictures of human beings – their perceptions, their aspirations, their environment and challenges to realize their aspirations.

Boundary Partners (have their own BPs) Ceja Andina Project Municipality Rural Agriculture Association Town population Pesticide salesmen Programme’s Boundary Partners Boundary Partners’ Boundary Partners Programme

Ask yourselves … « In which individuals, groups, or organisations is our programme trying to encourage change so that they can contribute to the vision?  With whom will we work directly? » « Are we choosing X boundary partner because we want to influence their behaviour and actions, or because they will influence others? Or both?» « What behavioural changes do we (collectively, between project and boundary partners) want to see in BP that will contribute to the vision (Outcome Challenges)?.»

Progress Markers Describes progression of changed behaviours in the boundary partner (is a set, as opposed to a single indicator in order to demonstrate the depth and complexity of change process). Changes in actions, activities, & relationships leading up to the ideal outcome challenge statement. Expect to see, like to see, love to see. Encourages program to think about how it can intentionally contribute to the most profound transformation possible. Permit on-going assessment of partners’ progress (including unintended results).

Progress Markers = Change Ladder Love to see PMs Truly transformative. Set quite high. Like to see PMs More active learning, engagement. Expect to see PMs Early response to program’s basic activities.

Example Progress Markers The project Expects to See local communities: 1. Participating in field day demonstrations. 2. Establishing a structure for cooperation in the partnership. The project would Like to see local communities: 3. Articulating a vision for the ceja andina forest that is locally relevant. 4. Requesting new opportunities for training. The project would Love to see local communities: 5. Influencing national policy and debates on CBNRM.

Progress Marker Checklist Each Progress Marker: Describes a changed behaviour by the boundary partner Can be monitored & observed Limit the number of PMs. As a set, Progress Markers: Are graduated from easier to more difficult to achieve changes in behaviour Describe the change process of a single boundary partner Remember: progress markers are changeable!!

6 Types of Strategies Strategy Causal Persuasive Supportive I-1 The program facilitates new tools, techniques, resources in order to contribute to outcomes and development process. Strategy Causal Persuasive Supportive I-1 Direct Output I-2 Arouse New Skills/ Thinking I-3 Supporter who guides change over time Aimed at the Boundary Partner E-1 Alter physical or regulatory environment E-2 Modify the information system E-3 Create / Strengthen a Peer Network Aimed at the Boundary Partner`s Environment Strategy map for each Outcome Challenge. I1 Give $ I-2 All capacity building activities I-3 Mentor/expert involved over time (frequent, sustained) E-1 E.g. computer Make certain things part of agreements with BPs E.g. multidisciplinary teams, gender analysis E-2 Alter the message system around the BP Usually access tomore/different ifnroamtion E.g. radio, internet, conference, publications E-3 For when the program isn`t around Collectively supporting each other

Creating your strategy map

Organizational Practices Help us to define our role as an NGO according to changing contexts and shifting paradigms. Important to how the program is going to function to effectively fulfill its mission. Supporting change in its boundary partners requires that the program be able to change and adapt as well. The things that we do as an organization to: foster creativity & innovation seek the best ways to assist partners maintain our niche Risk taking. Invitation to review its values, methods, relationships and objectives. How can we get better at what we are doing in order to increase our contributions to outcomes? Program needs to know not only about development results, but also about processes by which they were attained, as well as about its own internal effectiveness.

Eight Organizational Practices Prospecting for new ideas, opportunities, & resources. Seeking feedback from key informants. Obtaining the support of your next highest power Assessing & (re)designing products, services, systems, and procedures. Checking up on those already served to add value. Sharing your best wisdom with the world. Experimenting to remain innovative. Engaging in organizational reflection.

Journals Record internal and external data: Set overall intentions and strategies; design and articulate program’s logic (how it will contribute to change within a complex system).                 Monitor contributions to outcomes. Monitor: changes in behaviour of partners, program strategies and organizational performance. Promote self-assessment for BPs and program Target priority areas for detailed evaluation studies. Strategy journal = systematic way to monitor actions in support of boundary partners (so that it can think strategically about its contributions and modify its actions as required). How well have we done? How can we improve? How can we build our capacity to be effective and relevant? Logical connection between strategies and performance – but relational not causal; reflect on environment. Performance journal = in order to provide the program with a systematic way to monitor its actions (so that it can think strategically about its contributions and modify its actions as required). Performance journal = how much and what kind of information do you need to collect about your strategies and organizational practices to meet your learning needs and reporting requirements? Evaluation = base management and programming decisions on systematically collected data. Choose a strategy, issue or relation to study / assess in depth. Establish priorities. Utilization is the ultimate purpose of evaluation. Record internal and external data: How we are progressing towards outcomes. How we are contributing to change. Indicate cases of positive performance /improvement. Assess intended and unexpected results.

Outcome and Performance Monitoring Boundary Partners Program Outcome Challenges and Progress Markers Strategies and Activities Organizational Practices Outcome Journal Strategy Journal Performance Journal

Conclusions OM can help a program to be more strategic about the actors it targets, the changes it expects to see, and the means it employs to support and facilitate those changes. OM is not based on a cause-effect framework; it recognizes that multiple, nonlinear events lead to change. It sees interrelationships, not a snapshot. Attribution becomes contribution.

Conclusions Focus is on behavioural change – monitors and evaluates whether a program has contributed to changes in behaviours in a way that would be logically consistent with supporting development changes in the future. Program must also learn and change, reconsider and adjust its goals, methods, interventions. Self-assessment, systematically collected. Remember – OM decisions still depend on other factors = resources, time,

Conclusions Improving rather than proving. Understanding rather than simply reporting. Creativity and knowledge, rather than just taking credit.

For more information, case studies, materials, articles and presentations on Outcome Mapping: www.idrc.ca/evaluation

Afternoon session (what I propose) Fun dynamic! Identify your boundary partner! Continued, in-depth presentation on: Boundary partners. Outcome challenges. Progress markers. Practical exercise on progress markers. Challenges / successes of the methodology. Open discussion. Postcard exercise (key messages).

Not everything that counts can be counted Not everything that counts can be counted. And not everything that can be counted, counts. Albert Einstein

OM challenges Workload / paperwork– we made too much work for ourselves and didn’t prioritize (too many progress markers, tried to monitor everything). Trial-and-error reporting system. Scaling-up within NGO. New boundary partners. Different boundary partners have different experiences with the projects (example: municipalities). Project team had no formal training on the methodology.

OM challenges Weak monitoring plan (no one to lead OM from the beginning). How to combine with more quantitative data collection. Encouraging on-going boundary partner participation in OM M&E activities.

OM successes Build cohesion with team and with partners. Identify and develop interconnectedness among boundary partners through dialogue, consensus and collaborative action; social learning space. How to connect with people becomes part of the methodology. Highly participatory. Donor is flexible – encourages learning, creativity, adjustments. Makes space for dialogue and interaction (and really listens!) Self-assessment “all around” (of team, of boundary partners).

OM successes Innovation of tools and application of tools. Build culture of M&E within NGO / team and boundary partners. Concrete data for developing evaluation plan. Meetings for discussion / self-assessment = learning organization (analyze our learning). Permanent system of reflection. Interest to learn more about OM by other NGOs in Ecuador (request for information, presentations, etc.) = good promotion of methodology (and project).