Www.outcomemapping.ca Ten years of Outcome Mapping adaptations & support An analysis of how and where outcome mapping has been applied, how users have.

Slides:



Advertisements
Similar presentations
Health impact assessment explained
Advertisements

Tools for Policy Influence. RAPID Programme SMEPOL, Cairo, February, Practical Tools.
The Cost of Authoring with a Knowledge Layer Judy Kay and Lichao Li School of Information Technologies The University of Sydney, Australia.
Introducing the Researcher Development Framework (RDF) Gill Johnston, University of Sussex.
Intelligence Step 5 - Capacity Analysis Capacity Analysis Without capacity, the most innovative and brilliant interventions will not be implemented, wont.
How can Outcome Mapping help strengthen knowledge and learning strategies? Ben Ramalingam
Foundation of Nursing Studies in partnership with the Burdett Trust for Nursing Patients First: Supporting nurse led innovation in practice Workshop 1.
Working with the Teachers’ Standards in the context of ITE. Some key issues for ITE Partnerships to explore.
Good Evaluation Planning – and why this matters Presentation by Elliot Stern to Evaluation Network Meeting January 16 th 2015.
An introduction to Monitoring and Evaluation in Outcome Mapping Applied and adapted use Bern, 15 April 2010.
An Introduction to an Integrated P,M&E System developed by IDRC Kaia Ambrose October, 2005.
Huib Fred C Jan v O Beatrice B ? Petra K Nele B Heidi S Ben R Lorra T Daniel R Gordon M I want to learn what are the “10 most common arguments against.
The Road to CAPACITY BUILDING for IMPACT EVALUATION: A SHOWCASE from the MENA Region Cairo, Thursday April 2 nd 2009 Ziad Moussa Officer, Capacity Development.
INTEGRATING BENEFICIARY FEEDBACK INTO EVALUATION- A STRUCTURED APPROACH Presentation to UKES Conference May 2015 Theme: Theory and practice of inclusion.
Benchmarking as a management tool for continuous improvement in public services u Presentation to Ministry of Culture of the Russian Federation u Peter.
Workshop 1. Patients First Introductions Poster viewing Confidence line Time capsule Hopes, fears and expectations.
1 Public Entities Risk Management Forum Office of the Accountant General National Treasury Project Risk Management Jeets Hargovan Government Technical.
Submission Writing Fundamentals – Part Webinar Series Leonie Bryen.
Bond.org.uk The Bond Effectiveness Programme: developing a sector wide framework for assessing and demonstrating effectiveness July 2011.
Capable leadership is vital for meeting the challenges faced by aged care provider organisations and for the continued sustainability of the industry.
HR Business Impact Survey Introductory Slides can be adapted.
From Evidence to Action: Addressing Challenges to Knowledge Translation in RHAs The Need to Know Team Meeting May 30, 2005.
Stuart Hollis Where are we now? An exploration of the provision of teacher training programmes for the Learning and Skills Sector following the 2007 Workforce.
NSW Curriculum and Learning Innovation Centre Draft Senior Secondary Curriculum ENGLISH May, 2012.
Using motivational interviewing to improve social workers’ engagement of fathers in child protection Jonathan Scourfield, Cardiff University Nina Maxwell,
Needs Assessment: Young People’s Drug and Alcohol Services in Edinburgh City EADP Children, Young People and Families Network Event 7 th March 2012 Joanne.
Michalis Adamantiadis Transport Policy Adviser, SSATP SSATP Capacity Development Strategy Annual Meeting, December 2012.
JOINT STRATEGIC NEEDS ASSESSMENT Rebecca Cohen Policy Specialist, Chief Executive’s.
Aims of Workshop Introduce more effective school/University partnerships for the initial training of teachers through developing mentorship training Encourage.
RAPID Outcome Mapping Approach Simon Hearn, ODI 16 April 2010 Bern, Switzerland.
Monitoring and Evaluation of GeSCI’s Activities GeSCI Team Meeting 5-6 Dec 2007.
EDPQS in 10 minutes: Overview of European Drug Prevention Quality Standards (EDPQS) With financial support from the Drug Prevention and Information Programme.
Third Sector Evaluation: Challenges and Opportunities Presentation to the Public Legal Education in Canada National Conference on “Making an Impact” 26.
An introduction to Outcome Mapping AEA, Friday 26 October 2012 Simon Hearn, Overseas Development Institute
School Improvement Partnership Programme: Summary of interim findings March 2014.
Better Community Engagement Training for Trainers Course Day 1 This was developed as part of the Scottish Government’s Better Community Engagement Programme.
OPTIONS AND REQUIREMENTS FOR ENGAGEMENT OF CIVIL SOCIETY IN GEF PROJECTS presented by Ermath Harrington GEF Regional Focal Point.
SC. Asia Project Training on Developing National Action Plans on Sustainable Consumption Adriana Zacarias & Fu Lu UNEP SC. Asia Project 17 March 2005 Manila.
Using action learning to improve personal and organisational effectiveness.
Objectives Targets & Indicators for Adaptation to Climate Change A presentation to the expert meeting on climate change vulnerability and adaptation indicators,
Queen’s Management & Leadership Framework
Consultant Advance Research Team. Outline UNDERSTANDING M&E DATA NEEDS PEOPLE, PARTNERSHIP AND PLANNING 1.Organizational structures with HIV M&E functions.
The implementation programme for the 2008 SNA and supporting statistics UNECE special session on National Accounts for economies in transition Geneva,
Introduction to Outcome Mapping
UNDP Guidance for National Communication Project Proposals UNFCCC Workshop on the Preparation of National Communications from non-Annex I Parties Manila,
Monitoring Afghanistan, 2015 Food Security and Agriculture Working Group – 9 December 2015.
Developing a Monitoring and Evaluation Framework: Insight into internal stakeholder learnings Beth Ferguson AES Conference Sydney 2 September 2011.
Assessing Contribution and Enhancing Social Learning: An Introduction to Outcome Mapping (OM) 16 December 2004 Kaia Ambrose.
Secondary Curriculum Review Implications for teacher trainers.
Presentation By L. M. Baird And Scottish Health Council Research & Public Involvement Knowledge Exchange Event 12 th March 2015.
Initial Project Aims To increase the capacity of primary schools in partnership with parents to implement a sustainable health and sexuality education.
Tools for Mainstreaming Disaster Risk Reduction: Guidance Notes for Development Organisations Charlotte Benson and John Twigg Presented by Margaret Arnold.
WHY IS THIS HAPPENING IN THE PROGRAM? Session 5 Options for Further Investigation & Information Flow.
AGRO PARKS “The Policy Cycle” Alex Page Baku November 2014.
Session 1. The Central Principles of HiAP WORKSHOP: PREPARING FOR TRAINING IN HEALTH IN ALL POLICIES (HiAP) USING THE NEWLY LAUNCHED WHO HiAP TRAINING.
Session 1. The Central Principles of HiAP
Workshop to develop theories of change
Project Cycle Management
Technical Cooperation Section SEDI- Executive Office
Intro slide to all of these:
KEY PRINCIPLES OF THINKING SYSTEMICALLY
Gender Equality Ex post evaluation of the ESF ( )
Draft OECD Best Practices for Performance Budgeting
An Introduction to an Integrated P,M&E System developed by IDRC
Technology Enabled Care and Support in Devon
Building Knowledge about ESD Indicators
A Focus on Outcomes and Impact
I want to share “tips for overcoming resistance”
Evaluation: Framing Outcomes For Impact
Civil Society Facility and Media Programme Call for proposals: EuropeAid/162473/DH/ACT/Multi Webinar no. 3: Preparing effective Concept Note.
Presentation transcript:

Ten years of Outcome Mapping adaptations & support An analysis of how and where outcome mapping has been applied, how users have experienced OM and the support options available and required for its use. Richard Smith, John Mauremootoo & Kornelia Rassmann Outcome Mapping Learning Community Webinar 28 September 2012

Richard Smith, John Mauremootoo, Kornelia Rassmann, 28 September 2012 Purpose and objectives of the study Purpose Provide the OMLC, its stewards, funders and others interested in OM with an overview of where, how and why OM has been used, situations when OM is most / least useful, support needed, and experiences of those who have used / adapted OM, in order to further the use and development of OM and the OMLC. Objectives 1.Provide a stimulus for the OMLC to update tools to assist with information requests by extending the data available on OM applications and trainers/consultants. 2.Contribute to a fuller understanding of how OM can be used, further developed and promoted through analysis of OM applications and user experiences. 3.Inform the development of training and other support for OM users by identifying gaps in the current support available. Timeline: September 2011 – March 2012, in consultation with the OMLC stewards (preliminary results were presented at the OM Lab 2012 in February).

Richard Smith, John Mauremootoo, Kornelia Rassmann, 28 September 2012 Methods Two Excel databases: case studies and practitioners (including trainers) Types of data sources: 1. OMLC and IDRC databases / websites, online research, correspondence; in-depth interviews with OM users and consultants / those providing training or other support Definition of ‘OM’ agreed with OMLC Stewards: explicit reference to use of one or more of the 12 OM steps | ‘OM inspired‘: approaches that cite OM concepts but do not explicitly use OM steps Cautionary notes: The data is indicative of OM applications, user experiences and support available and required – this is not a comprehensive assessment Interviewees may be more likely to have a positive view of OM than others; however, we had a near 100% response rate to interview requests and many shared problems as well as solutions

Richard Smith, John Mauremootoo, Kornelia Rassmann, 28 September 2012 Data sources: 123 case studies

Richard Smith, John Mauremootoo, Kornelia Rassmann, 28 September 2012 Findings: scope Where and in what contexts has OM been used?

Richard Smith, John Mauremootoo, Kornelia Rassmann, 28 September 2012 OM use by region

Richard Smith, John Mauremootoo, Kornelia Rassmann, 28 September 2012 OM use by sector

Richard Smith, John Mauremootoo, Kornelia Rassmann, 28 September 2012 Findings: user experiences Motivation, benefits-challenges-solutions, training

Richard Smith, John Mauremootoo, Kornelia Rassmann, 28 September 2012 Motivations for using OM Enthusiasm for OM Dissatisfaction with LFA Recommendation from colleagues Piloting an alternative approach It was being used by the organisation

Richard Smith, John Mauremootoo, Kornelia Rassmann, 28 September 2012 Focusing on change in social actors that you only influence is a breakthrough I have heard described as a revelation, an epiphany and revolutionary. I came to appreciate OM as an attempt to be intellectually honest about RBM. LFA expects a direct link to results. In OM, it is explicitly recognised that results will not be solely attributable to the project / intervention. Rather the project will contribute to results. Image: Laurie Chipps, CC BY-ND 2.0

Richard Smith, John Mauremootoo, Kornelia Rassmann, 28 September 2012 Appreciation: OM Concepts Outcome orientation: OM forces people to use an outcomes-oriented planning approach. Most projects / programmes are not planned from back to front (i.e. results-oriented) but from the front (activity-oriented). Outcomes as behavioural change: OM introduces a paradigm shift which helped people realise that to achieve results we need to change attitudes. Boundary Partners and spheres of control, influence and concern: People realise they need to be modest and honest about what is in their scope / influence. Attribution/contribution: I came to appreciate OM as an attempt to be intellectually honest about RBM. In OM, it is explicitly recognised that results will not be solely attributable to the project.

Richard Smith, John Mauremootoo, Kornelia Rassmann, 28 September 2012 Appreciation: planning/design Planning - there is nothing more useful than OM. OM focuses people. Stop trying to change the world; focus on your sphere of influence. Tell me about 3-4 Boundary Partners, not 90 stakeholders. We use OM concepts mostly for project / programme planning, where we find the biggest deficits in our partners' capacity. Well planned is half monitored!

Richard Smith, John Mauremootoo, Kornelia Rassmann, 28 September 2012 Issues and solutions: planning/design letting go of their LFA history. People familiar with traditional PME approaches may be struggle with an approach that appears to be imprecise and fluid. OM mentoring as part of a continuous learning approach Plans can be very heavy and impractical where there are numerous BPs, long lists of Progress Markers and overlapping support strategies. Adapt plans so that they are situationally responsive Involving BPs in planning can be a challenge because “partners” often lack the time or inclination for engagement. Pragmatism regarding the involvement of BPs and use complementary methods to develop the intentional design when engagement is not possible.

Richard Smith, John Mauremootoo, Kornelia Rassmann, 28 September 2012 Appreciation: Monitoring [OM provides a] link to learning. With OM you are always monitoring your strategy; if it is not working, you change something. It provides evidence to base decision making on : quarterly planning meetings are based on evidence, not the views of the most assertive participant OM is great for organisational learning. It is the only PME method that tries to bridge across programmes and organisations, offering approaches for both in a combination

Richard Smith, John Mauremootoo, Kornelia Rassmann, 28 September 2012 Issues and solutions: monitoring Trying to 'do it by the book', using all the journals described in the OM manual is impractical / too demanding / generates too much data to be useful. It is important to be careful where to use Progress Marker data: the detail can obscure the bigger picture. Using journals risks an overload of data and a fall back into a report mode. Most projects adapted / simplified OM monitoring by: Use of only some of the steps / journals. Instead of journals, use of face to face meetings of the project team and Boundary Partners. Use of complementary methods

Richard Smith, John Mauremootoo, Kornelia Rassmann, 28 September 2012 Appreciation: Evaluation The OM concept of outcomes helps people to think about evaluation differently. It does not handcuff participants... An OM-inspired evaluation methodology, ‘Outcome harvesting’ suggests capturing what others can already see. But the evaluation actually produced outcome statements that were unexpected as they had not been captured by the monitoring we had been doing. We were surprised and impressed by the contributions our programme had made.

Richard Smith, John Mauremootoo, Kornelia Rassmann, 28 September 2012 Issues and solutions: evaluation The evaluation planning step is very brief OM is currently more useful for planning and monitoring, less so for evaluation purposes OM has inspired evaluations that capture behavioural change outcomes to which an intervention contributes Outcome Harvesting and other OM-inspired approaches can be used whether or not OM was used for designing or monitoring the intervention

Richard Smith, John Mauremootoo, Kornelia Rassmann, 28 September 2012 Training – is there a mismatch? Image: Frank Wuestefeld, CC BY-NC-SA 2.0 More could be achieved more quickly if introductory training can be given for those in similar situations rather than to mixed groups Support is needed for implementing partners to get the reflection/learning cycle moving. The findings suggest a need for more trainers and consultants experienced in using OM in a range of sectors and of trainers and learning tools in the multiple languages and locations in the economic south Many felt that training should go beyond “The seagull approach “ - land, xxxx and leave !

Richard Smith, John Mauremootoo, Kornelia Rassmann, 28 September 2012 Conclusions and suggestions Adaptation, enabling factors, when OM works best, training / support / learning resources, donors and log frames

Richard Smith, John Mauremootoo, Kornelia Rassmann, 28 September 2012 Never cook by the book! With some adaptations, its various elements and tools can be used separately or in conjunction with other processes (for example, a SWOT, a situational analysis, or an LFA). This manual does not provide the reader with instructions on how to adapt Outcome Mapping, but instead assumes it is being used in its totality. (p11, OM manual, Earl, Carden, Smutylo, 2001) Image: Lee Stranahan, CC BY-NC-SA 2.0

Richard Smith, John Mauremootoo, Kornelia Rassmann, 28 September 2012 Adaptations of OM Adaptation has taken four non-exclusive forms: 1. Use of some but not all of the 3 stages / 12 steps; 2. Use of one or more of the key concepts – such as outcomes defined as behavioural change - with or without any of the 12 steps; 3. Starting not with stage 1 (intentional design) but with monitoring or evaluation; 4. Using OM with other approaches, including the Logical Framework Approach and Most Significant Change.

Richard Smith, John Mauremootoo, Kornelia Rassmann, 28 September 2012 Use of OM with other approaches

Richard Smith, John Mauremootoo, Kornelia Rassmann, 28 September 2012 Proposed Enabling Factors Essential 1. Complexity in the intervention environment. 2. Appreciation of the rationale for OM application. 3. Champions and appropriate technical support. Optional 4. Funder support for using OM 5. Support for and understanding of OM at the executive level 6. The promotion of an organisational learning culture 7. An appreciation of the value of a results and learning-oriented PME system at multiple levels in the organisation 8. Availability of sufficient resources

Richard Smith, John Mauremootoo, Kornelia Rassmann, 28 September 2012 When OM works best The ‘OM receptivity continuum’ Enabling factors for the use of OM

Richard Smith, John Mauremootoo, Kornelia Rassmann, 28 September 2012 Training, support & learning resources Many users lack the confidence needed to adapt OM to their situation - > three suggestions: Step 0 training: trainees can position their intervention on the OM receptivity continuum and prepare for OM use Mentoring: including remote mentoring - a cost-effective way of providing ongoing support for champions of OM E-learning tool: encourage OM adaptations through a structured access to existing / new OMLC resources that – Introduces OM concepts in the context of other PME concepts – Introduces case studies to explain how OM can be simplified / used with other PM&E approaches / non-linear use of OM tools and concepts – Encourages the use of OM thinking in evaluation / impact evaluation

Richard Smith, John Mauremootoo, Kornelia Rassmann, 28 September 2012 Donors and LFA [OM is] inconsistent with the way donors work. OM and LFA are incompatible Still, many persist with using OM often alongside LFA and are excited about the results AND we identified 36 funders of interventions that used OM However, most funders are probably often unaware when OM has been used because of the strategies used: – Using OM ‘by stealth’ i.e. without the terminology – Using OM internally and LFA for donor reporting

Richard Smith, John Mauremootoo, Kornelia Rassmann, 28 September 2012 Proposed outreach to donors After 10 years of OM, there is much potential for outreach to foundations and public funders: – Donor-specific training / ongoing support in participatory PME (OM and other approaches) – Promote OM as part of a toolbox to be drawn from, not alternative that has to used to replace existing approaches in entirety – Publicise examples of how OM adds value in LFA context e.g. how OM outcomes help understand progress / obstacles to achieving impacts – Seek champions within donors and facilitate the sharing of experiences among donors – Describe contexts where OM is most / least useful

Richard Smith, John Mauremootoo, Kornelia Rassmann, 28 September 2012 Thank you! For a summary, see OM Ideas 4: For the full report, Ten years of OM adaptations and support: The authors are independent consultants experienced in programme management, design and evaluation using OM and other approaches, with international development, environment and agriculture expertise and a passion for helping realise the potential of partnership networks. Richard Smith: John Mauremootoo: Kornelia Rassmann: