1. Monitoring and Evaluation of Influence John Young

2. 2 Outline Why measure impact? What to measure? What sort of policy change? Planning/measuring in complex environments Behaviour change Some tools we use Other tools A systematic approach Recommendations Resources 2

3. 3 Why should you do M&E? To learn To manage better To get more funds To keep funders/clients happy (“what is the guarantee that doing this actually helps us?”) 3

4. 4 What should you measure? “If you don't know where you are going, any road will get you there” 4

5. 5 Monitoring and Evaluation Agenda Setting Decision Making Policy Implementation Policy Formulation Policy processes are... Civil Society Donors Cabinet Parliament Ministries Private Sector

6. 6 RAPID Outcome Mapping Approach

7. 7 Policy objectives Discursive: Client- focused services Attitudinal: Farmers have good ideas Procedural: Participatory approaches to service development Content: UU20, UU25. New guidelines Behavioural: Approach being applied in practice 7

8. 8 Visibility or substance? VisibilitySubstance Short term ‘relevant’ researchLong term research Focus on solutions for ‘agreed’ problems Engage with the definition of the problem Media exposureLobby, network, horse trading Briefing papers, Opinion piecesEstimates, costed proposals, policy options Website, Blogs, Facebook, etc.Academic publications, long reports Online communities with millions of hits Communities with the right people Delegations at high level global conferences Private meetings at Party conferences and private meetings while planning for the high level conferences Event focused influenceProblem focused influence Global Go-To-SurveyProspect magazine Think Tank of the year 8

9. 9 The Cynefn Framework 9

10. 10 Focus on behaviour change InputsActivitiesOutputs Other Actors Project Team OutcomeImpact Outcomes Impact Outcomes Impact Behaviour Change 10

11. 11 Professionalisation of Public Services. Structural Adjustment → collapse of services. Paravet projects emerge. ITDG projects. Privatisation. ITDG Paravet network. Rapid spread in North. KVB letter (January 1998). Multistakeholder WSs → new policies. Still not approved / passed! Episode Studies 1970s 1980s 1990s 2000s Professionalisation of Public Services. Structural Adjustment Privatisation ITDG Paravet network and change of DVS. KVB letter (January 1998). Multistakeholder WSs → new policies. ITDG projects – collaborative research. The Hubl Study Dr Kajume International Research 11

12. 12 Outcome Mapping OUTCOME MAPPING: Building Learning and Reflection into Development Programs Sarah Earl, Fred Carden, and Terry Smutylo http://www.idrc.ca/en/ev-9330-201-1- DO_TOPIC.html 12

13. 13 Social Network Analysis www.odi.org.uk/RAPID/Tools/Toolkits/KM/Social_network_analysis.html 13

14. 14 RAPID Outcome Assessment www.odi.org.uk/RAPID/Publications/RAPID_WP_266.html 14

15. 15 15 minute team debrief, conducted in a “rank-free” environment. After Action Reviews What was supposed to happen? What actually happened? Why was there a difference? What can we learn from this? 15

16. 16 Most Significant Change 1.Collect “stories of change” from different stakeholders 2.Systematic analysis of “significance”. http://www.mande.co.uk/docs/MSCGuide.pdf 16

17. 17 Impact Logs 17

18. 18 ODI Impact Logs 18 Contact: n.scott@odi.org.uk

19. 19 Others.... Classical case studies (IDRC, IFPRI) Stories of Change (Denning) Innovation Histories (CIAT) HERG Payback Framework (Hanney) Micro-Narratives (Snowden) Impact matrices (Davies) Peer evaluations (CHSRF) Systematic reviews? RCTs? 19

20. 20 A systematic approach 1.Evidence and advice: Peer-review of outputs Uptake logs Outcome mapping 2.Public campaigns and advocacy Surveys and focus groups Media tracking logs Media/public frame analysis 3.Lobbying Records of meetings Tracking people Key informants 20 http://www.odi.org.uk/resources/details.asp?id=5252&title=monitoring- evaluation-me-policy-influence

21. 21 1.Strategy and direction –are you doing the right thing? 2.Management –are you doing what you planned to do? 3.Outputs – are the outputs appropriate for the audience? 4.Uptake – are people aware of your work? 5.Outcomes and impacts –are you having any impact? A systematic approach 1.Strategy and direction 2.Management 3.Outputs 4.Uptake 5.Outcomes and impacts 21

22. 22 3ie PIM Project – Initial ideas 22

23. 23 Conclusions Research to influence: Clear objectives Understand the context Theory of change Iterative / learning approach Measuring impact Clear objectives Theory of change 5-levels Multiple methods Triangulation Expect the unexpected 23

24. 24 Recommendations Strategy: theory of change, impact pathway, peer review, log frame Management: appreciative inquiry, logs, AARs, PRINCE2 Output: logs, peer-review, Uptake: logs, webstats, surveys Impact: outcome mapping, stories of change, episode studies, peer review 24

25. 25 Resources Helping researchers become policy entrepreneurs. How to develop engagement strategies for evidence-based policy-making. John Young and Enrique Mendizabal. ODI Briefing Paper 53. 2009 - http://www.odi.org.uk/resources/details.asp?id=1127&title=becom e-policy-entrepreneur-roma Outcome Mapping: Building Learning and Reflection into Development Programs. Sarah Earl, Fred Carden, and Terry Smutylo IDRC - http://www.idrc.ca/EN/Resources/Publications /Pages/IDRCBookDetails.aspx?PublicationID=121 Making a difference: M&E of policy research - ODI Working Paper 281. Ingie Hovland. July 2007. http://www.odi.org.ukwww.odi.org. uk/resources/docs/2426.pdf A guide to monitoring and evaluating policy influence. Harry JOnes. ODI Background Notes, February 2011 - http://www.odi.org.uk/ resources/details.asp?id=5252&title=monitoring-evaluation-me- policy-influence 25