Choosing and using Outcome Mapping well OM Lab 2012 Keynote Address Patricia Rogers Patricia.Rogers@rmit.edu.au Good morning. It is an honour to be here with you this week to explore Outcome Mapping together. How appropriate it is to be meeting in Beirut – where East meets West, and where change and rebuilding is underway. I’m excited to be part of this historic occasion where we will hear from each other about our experiences of Outcome Mapping, learn from the past and present, and together shape the future. Before we move to individual cases, I’d like to share some thoughts about what we are seeking to learn from each other. So this morning I’m going to discuss two big questions, some examples and a big idea. Let’s start with the questions. I’m sure we have brought many questions with us to Beirut. I think there are two big ones.

When should we choose Outcome Mapping? The first one is about choosing Outcome Mapping. There are lots of ways of doing evaluation. What is it about the situation that makes it appropriate to choose Outcome Mapping? Does Outcome Mapping suit particular purposes? Particular options in terms of who will have control over the evaluation and who will conduct it? Is Outcome Mapping more or less suitable in combination with other particular methods or research designs? I am going to be listening to the cases to understand why people chose Outcome Mapping for their evaluation, and what we might learn about making this choice.

When should we choose Outcome Mapping? there is a lot of guidance about what impact evaluation looks like Simon Kneebone ‘Show Me The Change” http://www.flickr.com/photos/smtc/sets/72157624112171834/

When should we choose Outcome Mapping? there is a lot of guidance about what impact evaluation looks like Simon Kneebone ‘Show Me The Change” http://www.flickr.com/photos/smtc/sets/72157624112171834/

How do we use Outcome Mapping well? The second question is about using Outcome Mapping well. What are the skills needed to do it well? How do we know if we have those skills in our team? How do we develop them? Outcome Mapping, like other evaluation methods, is often done in ways that are adapted from its original form. When we’re planning what we will actually do, there are many different ways of proceeding. But at what point do these adaptations stop being Outcome Mapping? What are the essential elements for it to still be Outcome Mapping? Jessica Dart, who with Rick Davies developed and popularised another approach to evaluation - Most Significant Change – has talked about how she has seen all sorts of things being called Most Significant Change, even when essential elements of the approach are not present. When we are adapting Outcome Mapping, what are reasonable adapations? What are necessary adaptations to our situation? How do we know if we have lost the essence? To answer this, I think we need to learn from the process of knowledge translation in evidence-based policy. We need to be able to distinguish between the theories of change we are trying to activate and the theories of action about what we will do. Let me illustrate this.

Does it have to be pizza? A few years ago I was working on an evaluation of a middle school project that aimed to support kids who were in danger of getting into trouble and dropping out. The project involved mentoring, homework help and referral to specialist services. One of the elements of the project was a pizza night at the start of the school year, involving kids, parents and teachers where they could meet and start establishing relationships between home and school, and have some shared, fun experiences. Another school had heard about this program and was planning to implement it in the new school year. I was in the office when a phone call came in from them. They were all set to go for the year. They just had one question. Did it have to be pizza? This is a common issue when we seek to replicate success – confusing the details of the action with the details of the underpinning theory of how change comes about. It is helpful to have a very clear picture of how we think change will come about, and then adapt the specific activities to be locally suitable. So, changing from pizza to tacos or to falafel might be a useful local adaptation. But changing the feel of the event from a fun, shared event to a serious lecture on student behaviour would not be – even if they then served pizza. Before moving to

Example 1: Stepping away from Outcome Mapping

Example 2: Seeking out Outcome Mapping

Evaluation

Better projects, programs, policies Evaluation

Better projects, programs, policies ?? Changed behaviour – decision, actions Better projects, programs, policies Evaluation

Outcome Mapping can provide.. Which can contribute to… Accurate, timely and credible information that identifies and explains poor performance Decisions and actions to solve problems

Outcome Mapping can provide.. Which can contribute to… Accurate, timely and credible information that identifies and explains good performance Decisions and actions to reinforce, repeat, replicate expand good practice

Outcome Mapping can provide.. Which can contribute to… Clear signals about what is important More focus (attention and resources) on priority issues

Outcome Mapping can provide.. Which can contribute to… Clear accountability and consequences Increasedmotivation to find ways to improve performance

Outcome Mapping can provide.. Which can contribute to… Increased ability to generate and use information Ongoing capacity to learn

Outcome Mapping can provide.. Which can contribute to… Compelling narratives of achievements Improved support and resourcing

Outcome Mapping can provide.. Which can contribute to… Accurate, timely and credible information that identifies and explains poor performance Decisions and actions to solve problems Accurate, timely and credible information that identifies and explains good performance Decisions and actions to reinforce, repeat, replicate expand good practice Clear signals about what is important More focus (attention and resources) on priority issues Clear accountability and consequences Increasedmotivation to find ways to improve performance Increased ability to generate and use information Ongoing capacity to learn Compelling narratives of achievements Improved support and resourcing