1. Measuring policy influence: like measuring thin air? John Young: j.young@odi.org.uk

2. Monitoring and Evaluation Agenda Setting Decision Making Policy Implementation Policy Formulation The policy cycle Civil Society Donors Cabinet Parliament Ministries Private Sector 2

3. Research plays a minor role Kate Bird et al, Fracture Points in Social Policies for Chronic Poverty Reduction, ODI WP242, 2004 (http://www.odi.org.uk/publications/working_papers/wp242.pdf)

4. There are many other factors Scientific Evidence Experience & Expertise Judgement Resources Values and Policy Context Habits & Tradition Lobbyists & Pressure Groups Pragmatics & Contingencies Source: Phil Davies Impact to Insight Meeting, ODI, 2005

5. Example: health care in Tanzania “The results of household disease surveys informed processes of health service reform which contributed to a 43 and 46 per cent reduction in infant mortality between 2000 and 2003 in two districts in rural Tanzania.” TEHIP Project, Tanzania: www.idrc.ca/tehip

6. What is policy change? 6 Discursive: Client-focused services Attitudinal: Farmers have good ideas Procedural: Participatory approaches to service development Content: UU20, UU25. New guidelines and programmes Behavioural: Approach being applied in practice

7. It’s all about behaviour change 7 InputsActivitiesOutputs By the Project OutcomeImpact Outcomes Impact Outcomes Impact Behaviour Change By other Actors Level of Activity

8. Professionalisation of Public Services. Structural Adjustment → collapse of services. Paravet projects emerge. ITDG projects. Privatisation. ITDG Paravet network. Rapid spread in North. KVB letter (January 1998). Multistakeholder WSs → new policies. Still not approved / passed! 1970s 1980s 1990s 2000s Professionalisation of Public Services. Structural Adjustment Privatisation ITDG Paravet network change of DVS. KVB letter (January 1998). Multistakeholder WSs → new policies. ITDG projects – Action-research. The Hubl Study Dr Kajume Animal Healthcare in Kenya International Research

9. Lots of methods Classical case studies (IDRC, IFPRI) Episode studies (ODI/RAPID) Stories of Change (Denning) Most Significant Change (Davies) Micro-Narratives / Sensemaker (Snowden) Outcome Mapping (IDRC) Impact matrices (Davies) Peer evaluations (CHSRF) HERG Payback Framework (Brunel) Systematic reviews (DFID) RCTs (IDS) 9

10. 1.Strategy and direction –are you doing the right thing? 2.Management –are you doing what you planned to do? 3.Outputs – are the outputs appropriate for the audience? 4.Uptake – are people aware of your work? 5.Outcomes and impacts –are you having any impact? A systematic approach 1.Strategy and direction 2.Management 3.Outputs 4.Uptake 5.Outcomes and impacts 10

11. 1.Strategy and direction - Logframes; Social Network Analysis; Impact Pathways etc 2.Management –‘Fit for Purpose’ Reviews; Quality Audits; Horizontal Evaluation; 3.Outputs – Peer review; Evaluating websites; Evaluating networks; After Action Reviews 4.Uptake – Impact Logs; New Areas for Citation Analysis; User Surveys 5.Outcomes and impacts –Outcome Mapping; RAPID Outcome Assessment; Most Significant Change; Innovation Histories; Episode Studies A systematic approach 1.Strategy and direction 2.Management 3.Outputs 4.Uptake 5.Outcomes and impacts 11

12. Logical frameworks GoalIndicatorMOV PurposeIndicatorMOVAssumptions/Risks Output 1IndicatorMOVAssumptions/Risks Output 2IndicatorMOV Output 3IndicatorMOV Output 4IndicatorMOV √ √

13. Theories of change Causal Chain - succession of elements with logical links (eg log- frame approach) Dimensions of influence - overlapping domains which interact, where it is possible to influence (eg RAPID CEL Framework) Actor-centred theories - where the behaviour of actors can be influenced (eg Outcome Mapping) 13

14. 14 Theories of change

15. After Action Review What was supposed to happen? What actually happened? Why was there a difference? What can we learn from it? 15 minute team debrief, conducted in a “rank-free” environment. 15

16. ODI CommStats 16

17. Stories of change 1.Essential elements: Situation before Context Situation after What changed and why 2.Most Significant Change (Davies) Stories of change from different stakeholders Systematic analysis of significance. 3.Micro-narratives (Snowden) http://www.mande.co.uk/docs/MSCGuide.pdf 17

18. RAPID Outcome Assessment 18 www.odi.org.uk/RAPID/Publications/RAPID_WP_266.html

19. M&E in ODI

20. Think Tank Initiative evaluation 20 Theory of Change and Assumptions Theory of Change and Ass Theory of Theory of Change and Assumptions ImpactUptakeManagement

21. Review of ToC: testing the assumptions – Literature review – (Desk-based) political economy analysis – Interviews Assessment of capacity change – Quantitative analysis of M&E data – Interviews and focus-groups – Stories of change Policy impact – (In country) political economy analysis – Case studies – Stories of change – Interviews and focus-groups 21 Think Tank Initiative evaluation