Philip Davies The Challenges Ahead for Impact Evaluation Studies in Poland Using Mixed Methods of Evaluation for Policy Making Purposes Philip Davies International Initiative for Impact Evaluation [3ie] 8 th Evaluation Conference, Evaluation in the System of Public Policies Warsaw, Poland, November 2012

Philip Davies What Is To Be Evaluated? Intervention effectiveness - what works? Implementation effectiveness – how/why it works? Diversity of effectiveness across different groups – what works for whom and when? Experiential effectiveness - users’ views Resource effectiveness - at what cost/benefit?

Philip Davies Theory Based Evaluation How is a policy/intervention supposed to work? Impact (or summative) evaluation Does the policy (programme, intervention) work? Process (or formative) evaluation How, why, and under what conditions does the policy (programme, intervention) work? Developmental evaluation Supporting innovation and adaptation. Providing feedback, generating learning, supporting direction or affirming changes in direction in real time. Building the evidence base over time. Which translates into:

Philip Davies The key to the challenges ahead is: What’s the Question?

Philip Davies Evidence for Policy Theories of Change How is the policy supposed to work? Logic Model

Philip Davies Evaluation: Theory of Change/Logic Model How is a policy/programme supposed to work? What activities, mechanisms, people have to be in place? And in what sequence – what is the causal chain? What resources are required – and are available? What data are required – and are available? Is the policy/programme feasible/achievable?

Philip Davies Evidence for Policy Theories of Change How is the policy supposed to work? Logic Model What is already known about this issue? Harness Existing Evidence Research Synthesis

Philip Davies Types of Research Synthesis Statistical Meta-Analyses(6-18 months) Narrative Systematic Reviews (6-12 Months) Rapid Evidence Assessments (1-3 Months) Evidence Maps and Gap Maps (1 Month) Meta-Ethnography /Qualitative Synthesis (6-12 Months)

Philip Davies Evidence for Policy Theories of Change How is the policy supposed to work? Logic Model What is already known about this issue? Harness Existing Evidence Research Synthesis What is the nature, size and dynamics of the problem? Administrative Data Surveys Census Data Qualitative Research Descriptive and Experiential Evidence

Philip Davies Evidence for Policy Theories of Change How is the policy supposed to work? Logic Model What is already known about this issue? Harness Existing Evidence Research Synthesis What is the nature, size and dynamics of the problem? What has been shown to work elsewhere? Evidence of Proven Effectiveness Experimental and Quasi- Experimental Evidence Descriptive and Experiential Evidence Administrative Data Surveys Census Data Qualitative Research

Philip Davies Experimental and Quasi-Experimental Evidence Randomised Controlled Trials Matched Comparison Designs - Propensity Score Matching Difference-in-Differences Regression Discontinuity Designs Interrupted Time Series Designs Instrumental Variable Analysis

Philip Davies Evidence for Policy Theories of Change How is the policy supposed to work? Logic Model What is already known about the problem/policy? Harness Existing Evidence Research Synthesis What is the nature, size and dynamics of the problem? What has been shown to work elsewhere? Evidence of Proven Effectiveness Experimental and Quasi- Experimental Evidence How do we make the policy work? Implementation Evidence Case Studies Interviews Focus Groups Ethnography Operations Research Descriptive Analysis and Experiential Evidence Administrative Data Surveys Census Data Qualitative Research

Philip Davies Impact Evaluations Evaluations of Outcome Attainment (Have targets been met?) Data Point 1 Data Point 2 Data Point 3 Priority Review Theory of Change Case Studies Interviews Focus Groups Ethnography Admin data Surveys Operations Analysis

Philip Davies Evidence for Policy Theories of Change How is the policy supposed to work? Logic Model What is already known about the problem/policy? Harness Existing Evidence Research Synthesis What is the nature, size and dynamics of the problem? What has been shown to work elsewhere? Evidence of Proven Effectiveness Experimental and Quasi- Experimental Evidence How do we make The policy work? Implementation Evidence/ Case Studies Interviews Focus Groups Ethnography Operations Research What are the costs and benefits of the policy ] Economic and Econometric Evidence Cost-Benefit/ Effectiveness/ Utility Analysis Descriptive Analysis and Experiential Evidence Administrative Data Surveys Census Data Qualitative Research

Philip Davies Economic Appraisal No policy programme or project should be adopted without first having the answer to these questions: Are there better ways to achieve this objective [Cost-Effectiveness Analysis] Are there better uses for these resources? [Cost-Benefit Analysis]

Philip Davies Evidence for Policy Theories of Change How is the policy supposed to work? Logic Model What is already known about the problem/policy? Harness Existing Evidence Research Synthesis What is the nature, size and dynamics of the problem? Descriptive and Experiential Evidence What has been shown to work elsewhere? Evidence of Proven Effectiveness Experimental and Quasi- Experimental Evidence How do we make The policy work? Implementation Evidence/ Case Studies Interviews Focus Groups Ethnography Operations Research What are the costs and benefits of the policy ] Economic and Econometric Evidence Cost-Benefit/ Effectiveness/ Utility Analysis What are the ethical implications of the policy? Ethical Evidence Social Ethics Public Consultation Administrative Data Surveys Census Data Qualitative Research

Philip Davies Theory of Change Activities Mechanisms People Resources Impact and Process Evaluations Systematic Review of All Evaluations Theorise SpecifyTest Review Refine The Evaluation Process Re-TestAccumulate

Philip Davies An Example: Conditional Cash Transfers Theory of Change Activities Mechanisms People Resources Impact and Process Evaluations Systematic Review of All Evaluations Theorise SpecifyTest Review CCTS incentivise parents to send children to school Improve enrolment and attendance Improve learning outcomes By enforcing the conditionality of schooling Changing the behaviour of parents and children Altering the relative costs and benefits of schooling versus other uses of children’s time >100 impact evaluations Many process evaluations Testing CCTs in Latin America, Asia, Africa Petrosino et al, (2012) Systematic Review of 23 studies Quality Education For All Children CCTs increase enrolments and attendance No overall impact on learning outcomes.

Philip Davies Theory of Change Activities Mechanisms People Resources Impact and Process Evaluations Systematic Review of All Evaluations Theorise SpecifyTest Review Refine Re-TestAccumulate Deadweight issues Levels of payment Payment recipient Reward performance Measure learning outcomes Supply-side interventions needed Teacher quality a mediating factor Phased transfers? Need to re-test refinements/mec hanisms Focus on contextual specificity Build an evidence base over time Establish generalisability And context specificity Avoid Type I/II errors

Philip Davies Thank you Philip Davies (0) Visit