2. www.3ieimpact.org Philip Davies The Challenges Ahead for Impact Evaluation Studies in Poland Using Mixed Methods of Evaluation for Policy Making Purposes Philip Davies International Initiative for Impact Evaluation [3ie] 8 th Evaluation Conference, Evaluation in the System of Public Policies Warsaw, Poland, 12-13 November 2012

3. www.3ieimpact.org Philip Davies What Is To Be Evaluated? Intervention effectiveness - what works? Implementation effectiveness – how/why it works? Diversity of effectiveness across different groups – what works for whom and when? Experiential effectiveness - users’ views Resource effectiveness - at what cost/benefit?

4. www.3ieimpact.org Philip Davies Theory Based Evaluation How is a policy/intervention supposed to work? Impact (or summative) evaluation Does the policy (programme, intervention) work? Process (or formative) evaluation How, why, and under what conditions does the policy (programme, intervention) work? Developmental evaluation Supporting innovation and adaptation. Providing feedback, generating learning, supporting direction or affirming changes in direction in real time. Building the evidence base over time. Which translates into:

5. www.3ieimpact.org Philip Davies The key to the challenges ahead is: What’s the Question?

6. www.3ieimpact.org Philip Davies Evidence for Policy Theories of Change How is the policy supposed to work? Logic Model

7. www.3ieimpact.org Philip Davies Evaluation: Theory of Change/Logic Model How is a policy/programme supposed to work? What activities, mechanisms, people have to be in place? And in what sequence – what is the causal chain? What resources are required – and are available? What data are required – and are available? Is the policy/programme feasible/achievable?

8. www.3ieimpact.org Philip Davies Evidence for Policy Theories of Change How is the policy supposed to work? Logic Model What is already known about this issue? Harness Existing Evidence Research Synthesis

9. www.3ieimpact.org Philip Davies Types of Research Synthesis Statistical Meta-Analyses(6-18 months) Narrative Systematic Reviews (6-12 Months) Rapid Evidence Assessments (1-3 Months) Evidence Maps and Gap Maps (1 Month) Meta-Ethnography /Qualitative Synthesis (6-12 Months)

10. www.3ieimpact.org Philip Davies Evidence for Policy Theories of Change How is the policy supposed to work? Logic Model What is already known about this issue? Harness Existing Evidence Research Synthesis What is the nature, size and dynamics of the problem? Administrative Data Surveys Census Data Qualitative Research Descriptive and Experiential Evidence

11. www.3ieimpact.org Philip Davies Evidence for Policy Theories of Change How is the policy supposed to work? Logic Model What is already known about this issue? Harness Existing Evidence Research Synthesis What is the nature, size and dynamics of the problem? What has been shown to work elsewhere? Evidence of Proven Effectiveness Experimental and Quasi- Experimental Evidence Descriptive and Experiential Evidence Administrative Data Surveys Census Data Qualitative Research

12. www.3ieimpact.org Philip Davies Experimental and Quasi-Experimental Evidence Randomised Controlled Trials Matched Comparison Designs - Propensity Score Matching Difference-in-Differences Regression Discontinuity Designs Interrupted Time Series Designs Instrumental Variable Analysis

13. www.3ieimpact.org Philip Davies Evidence for Policy Theories of Change How is the policy supposed to work? Logic Model What is already known about the problem/policy? Harness Existing Evidence Research Synthesis What is the nature, size and dynamics of the problem? What has been shown to work elsewhere? Evidence of Proven Effectiveness Experimental and Quasi- Experimental Evidence How do we make the policy work? Implementation Evidence Case Studies Interviews Focus Groups Ethnography Operations Research Descriptive Analysis and Experiential Evidence Administrative Data Surveys Census Data Qualitative Research

14. www.3ieimpact.org Philip Davies Impact Evaluations Evaluations of Outcome Attainment (Have targets been met?) Data Point 1 Data Point 2 Data Point 3 Priority Review Theory of Change Case Studies Interviews Focus Groups Ethnography Admin data Surveys Operations Analysis

15. www.3ieimpact.org Philip Davies Evidence for Policy Theories of Change How is the policy supposed to work? Logic Model What is already known about the problem/policy? Harness Existing Evidence Research Synthesis What is the nature, size and dynamics of the problem? What has been shown to work elsewhere? Evidence of Proven Effectiveness Experimental and Quasi- Experimental Evidence How do we make The policy work? Implementation Evidence/ Case Studies Interviews Focus Groups Ethnography Operations Research What are the costs and benefits of the policy ] Economic and Econometric Evidence Cost-Benefit/ Effectiveness/ Utility Analysis Descriptive Analysis and Experiential Evidence Administrative Data Surveys Census Data Qualitative Research

16. www.3ieimpact.org Philip Davies Economic Appraisal No policy programme or project should be adopted without first having the answer to these questions: Are there better ways to achieve this objective [Cost-Effectiveness Analysis] Are there better uses for these resources? [Cost-Benefit Analysis]

17. www.3ieimpact.org Philip Davies Evidence for Policy Theories of Change How is the policy supposed to work? Logic Model What is already known about the problem/policy? Harness Existing Evidence Research Synthesis What is the nature, size and dynamics of the problem? Descriptive and Experiential Evidence What has been shown to work elsewhere? Evidence of Proven Effectiveness Experimental and Quasi- Experimental Evidence How do we make The policy work? Implementation Evidence/ Case Studies Interviews Focus Groups Ethnography Operations Research What are the costs and benefits of the policy ] Economic and Econometric Evidence Cost-Benefit/ Effectiveness/ Utility Analysis What are the ethical implications of the policy? Ethical Evidence Social Ethics Public Consultation Administrative Data Surveys Census Data Qualitative Research

18. www.3ieimpact.org Philip Davies Theory of Change Activities Mechanisms People Resources Impact and Process Evaluations Systematic Review of All Evaluations Theorise SpecifyTest Review Refine The Evaluation Process Re-TestAccumulate

19. www.3ieimpact.org Philip Davies An Example: Conditional Cash Transfers Theory of Change Activities Mechanisms People Resources Impact and Process Evaluations Systematic Review of All Evaluations Theorise SpecifyTest Review CCTS incentivise parents to send children to school Improve enrolment and attendance Improve learning outcomes By enforcing the conditionality of schooling Changing the behaviour of parents and children Altering the relative costs and benefits of schooling versus other uses of children’s time >100 impact evaluations Many process evaluations Testing CCTs in Latin America, Asia, Africa Petrosino et al, (2012) Systematic Review of 23 studies Quality Education For All Children CCTs increase enrolments and attendance No overall impact on learning outcomes.

20. www.3ieimpact.org Philip Davies Theory of Change Activities Mechanisms People Resources Impact and Process Evaluations Systematic Review of All Evaluations Theorise SpecifyTest Review Refine Re-TestAccumulate Deadweight issues Levels of payment Payment recipient Reward performance Measure learning outcomes Supply-side interventions needed Teacher quality a mediating factor Phased transfers? Need to re-test refinements/mec hanisms Focus on contextual specificity Build an evidence base over time Establish generalisability And context specificity Avoid Type I/II errors

21. www.3ieimpact.org Philip Davies Thank you Philip Davies Email: pdavies@3ieimpact.orgpdavies@3ieimpact.org +44 (0)207 958 8350 Visit www.3ieimpact.org