Measuring policy influence: like measuring thin air? John Young:

Monitoring and Evaluation Agenda Setting Decision Making Policy Implementation Policy Formulation The policy cycle Civil Society Donors Cabinet Parliament Ministries Private Sector 2

Research plays a minor role Kate Bird et al, Fracture Points in Social Policies for Chronic Poverty Reduction, ODI WP242, 2004 (

There are many other factors Scientific Evidence Experience & Expertise Judgement Resources Values and Policy Context Habits & Tradition Lobbyists & Pressure Groups Pragmatics & Contingencies Source: Phil Davies Impact to Insight Meeting, ODI, 2005

Example: health care in Tanzania “The results of household disease surveys informed processes of health service reform which contributed to a 43 and 46 per cent reduction in infant mortality between 2000 and 2003 in two districts in rural Tanzania.” TEHIP Project, Tanzania:

What is policy change? 6 Discursive: Client-focused services Attitudinal: Farmers have good ideas Procedural: Participatory approaches to service development Content: UU20, UU25. New guidelines and programmes Behavioural: Approach being applied in practice

It’s all about behaviour change 7 InputsActivitiesOutputs By the Project OutcomeImpact Outcomes Impact Outcomes Impact Behaviour Change By other Actors Level of Activity

Professionalisation of Public Services. Structural Adjustment → collapse of services. Paravet projects emerge. ITDG projects. Privatisation. ITDG Paravet network. Rapid spread in North. KVB letter (January 1998). Multistakeholder WSs → new policies. Still not approved / passed! 1970s 1980s 1990s 2000s Professionalisation of Public Services. Structural Adjustment Privatisation ITDG Paravet network change of DVS. KVB letter (January 1998). Multistakeholder WSs → new policies. ITDG projects – Action-research. The Hubl Study Dr Kajume Animal Healthcare in Kenya International Research

Lots of methods Classical case studies (IDRC, IFPRI) Episode studies (ODI/RAPID) Stories of Change (Denning) Most Significant Change (Davies) Micro-Narratives / Sensemaker (Snowden) Outcome Mapping (IDRC) Impact matrices (Davies) Peer evaluations (CHSRF) HERG Payback Framework (Brunel) Systematic reviews (DFID) RCTs (IDS) 9

1.Strategy and direction –are you doing the right thing? 2.Management –are you doing what you planned to do? 3.Outputs – are the outputs appropriate for the audience? 4.Uptake – are people aware of your work? 5.Outcomes and impacts –are you having any impact? A systematic approach 1.Strategy and direction 2.Management 3.Outputs 4.Uptake 5.Outcomes and impacts 10

1.Strategy and direction - Logframes; Social Network Analysis; Impact Pathways etc 2.Management –‘Fit for Purpose’ Reviews; Quality Audits; Horizontal Evaluation; 3.Outputs – Peer review; Evaluating websites; Evaluating networks; After Action Reviews 4.Uptake – Impact Logs; New Areas for Citation Analysis; User Surveys 5.Outcomes and impacts –Outcome Mapping; RAPID Outcome Assessment; Most Significant Change; Innovation Histories; Episode Studies A systematic approach 1.Strategy and direction 2.Management 3.Outputs 4.Uptake 5.Outcomes and impacts 11

Logical frameworks GoalIndicatorMOV PurposeIndicatorMOVAssumptions/Risks Output 1IndicatorMOVAssumptions/Risks Output 2IndicatorMOV Output 3IndicatorMOV Output 4IndicatorMOV √ √

Theories of change Causal Chain - succession of elements with logical links (eg log- frame approach) Dimensions of influence - overlapping domains which interact, where it is possible to influence (eg RAPID CEL Framework) Actor-centred theories - where the behaviour of actors can be influenced (eg Outcome Mapping) 13

14 Theories of change

After Action Review What was supposed to happen? What actually happened? Why was there a difference? What can we learn from it? 15 minute team debrief, conducted in a “rank-free” environment. 15

ODI CommStats 16

Stories of change 1.Essential elements: Situation before Context Situation after What changed and why 2.Most Significant Change (Davies) Stories of change from different stakeholders Systematic analysis of significance. 3.Micro-narratives (Snowden) 17

RAPID Outcome Assessment 18

M&E in ODI

Think Tank Initiative evaluation 20 Theory of Change and Assumptions Theory of Change and Ass Theory of Theory of Change and Assumptions ImpactUptakeManagement

Review of ToC: testing the assumptions – Literature review – (Desk-based) political economy analysis – Interviews Assessment of capacity change – Quantitative analysis of M&E data – Interviews and focus-groups – Stories of change Policy impact – (In country) political economy analysis – Case studies – Stories of change – Interviews and focus-groups 21 Think Tank Initiative evaluation