www.3ieimpact.org Philip Davies The Challenges Ahead for Impact Evaluation Studies in Poland Using Mixed Methods of Evaluation for Policy Making Purposes.

Slides:



Advertisements
Similar presentations
Evidence-Based Policy at the Cabinet Office
Advertisements

Impact analysis and counterfactuals in practise: the case of Structural Funds support for enterprise Gerhard Untiedt GEFRA-Münster,Germany Conference:
Scoring Goals or Changing the Game: What Impacts Should We Measure? Jonathan Lomas Canadian Health Services Research Foundation Presentation to ESRC Symposium:
Mywish K. Maredia Michigan State University
A Guide to Education Research in the Era of NCLB Brian Jacob University of Michigan December 5, 2007.
How Do We Know if a Charter School is Really Succeeding? – Various Approaches to Investigating School Effectiveness October 2012 Missouri Charter Public.
Government Social Research Unit Philip Davies PhD Government Social Research Unit HM Treasury London SW1A 2HQ What Can Social.
Evidence for effective learning and teaching: ways and means Professor Marilyn Hammick July 2009.
VIII Evaluation Conference ‘Methodological Developments and Challenges in UK Policy Evaluation’ Daniel Fujiwara Senior Economist Cabinet Office & London.
V MEASURING IMPACT Kristy Muir Stephen Bennett ENACTUS November 2013.
Howard White Theory Based Evaluation Impact Evaluation Howard White International Initiative for Impact Evaluation.
Evaluation Research, aka Program Evaluation. Definitions Program Evaluation is not a “method” but an example of applied social research. From Rossi and.
Making Impact Evaluations Happen World Bank Operational Experience 6 th European Conference on Evaluation of Cohesion Policy 30 November 2009 Warsaw Joost.
Slide 5.1 Saunders, Lewis and Thornhill, Research Methods for Business Students, 5 th Edition, © Mark Saunders, Philip Lewis and Adrian Thornhill 2009.
Seminar on selected evaluation methodology issues 评估方法论研讨会 Independent Office of Evaluation of IFAD Beijing, 16 July 2014 国际农业发展基金独立评估办公室 2014 年 7 月 16.
Philip Davies Using Evidence for Policy and Practice Philip Davies International Initiative for Impact Evaluation [3ie] BCURE Evidence-Informed.
Types of Evaluation.
Methods and Approaches to investigate the UK Education System Sandra McNally, University of Surrey and Centre for Economic Performance, London School of.
Philip Davies Identifying the Problem Philip Davies International Initiative for Impact Evaluation [3ie] BCURE Evidence-Informed Decision-Making.
Product Evaluation the outcome phase. Do the magic bullets work? How do you know when an innovative educational program has “worked”? How do you know.
Access to health care, social protection, and household costs of illness proposal Cost of illness working group INDEPTH AGM 2009, Pune.
Research and Evaluation Center Jeffrey A. Butts John Jay College of Criminal Justice City University of New York August 7, 2012 How Researchers Generate.
Assessment of Higher Education Learning Outcomes (AHELO): Update Deborah Roseveare Head, Skills beyond School Division Directorate for Education OECD 31.
Striving for Quality Using continuous improvement strategies to increase program quality, implementation fidelity and durability Steve Goodman Director.
Philip Davies How is a Policy Supposed to Work? – Theory of Change Analysis Philip Davies International Initiative for Impact Evaluation.
Philip Davies Making Evidence Accessible and Relevant for Policy and Practice Philip Davies International Initiative for Impact Evaluation.
Copyright © 2014 Wolters Kluwer Health | Lippincott Williams & Wilkins Chapter 18 Mixed Methods and Other Special Types of Research.
Impact Evaluation for Evaluating Social Safety Nets at the World Bank Javier E. Baez Economist IEG World Bank AEA Conference San Antonio, TX November 12,
Copyright © 2008 Wolters Kluwer Health | Lippincott Williams & Wilkins Chapter 12 Undertaking Research for Specific Purposes.
South East Asia - Optimising Reproductive & Child Health Outcomes in Developing Countries SEA-ORCHID Project Centre for Perinatal Health Services Research,
Evaluation methods and tools (Focus on delivery mechanism) Jela Tvrdonova, 2014.
Semester 2: Lecture 9 Analyzing Qualitative Data: Evaluation Research Prepared by: Dr. Lloyd Waller ©
ALP CONFERENCE / June xx, COMMUNITY COLLEGE RESEARCH CENTER June 14, 2013 Nikki Edgecombe, Community College Research Center Hunter Boylan, National.
Overview of Evaluation Designs. Learning objectives By the end of this presentation, you will be able to: Explain evaluation design Describe the differences.
Evidencing Outcomes Ruth Mann / George Box Commissioning Strategies Group, NOMS February 2014 UNCLASSIFIED.
EVALUATION APPROACHES Heather Aquilina 24 March 2015.
Implementation and process evaluation: developing our approach Ann Lendrum University of Manchester Neil Humphrey University of Manchester Gemma Moss Institute.
Evaluation – Principles and methods Matt Barnard Head of Evaluation NSPCC.
Rapid Evidence Assessments and Evidence Gap Maps
© Nuffield Trust 22 June 2015 Matched Control Studies: Methods and case studies Cono Ariti
Howard White International Initiative for Impact Evaluation (3ie)
Impact Evaluation in Education Introduction to Monitoring and Evaluation Andrew Jenkins 23/03/14.
PPA 502 – Program Evaluation Lecture 2c – Process Evaluation.
Program Evaluation NCLB. Training Objectives No Child Left Behind Program Series: Program Evaluation To provide consistency across the State regarding.
Evaluation and Impact of Entrepreneurial Education and Training Malcolm Maguire Transnational High Level Seminar on National Policies and Impact of Entrepreneurship.
1 Government Social Research Unit Randomised Controlled Trials Conference University of York, September 2006 Why Governments Need.
Ch. 2 Tools of Positive Economics. Theoretical Tools of Public Finance theoretical tools The set of tools designed to understand the mechanics behind.
Evaluation Framework for New Income Management (NIM) Ilan Katz (SPRC) AES Conference Sydney September 2011.
© P. Vermeulen / Handicap International © W. Daniels pour Handicap International © B. Franck / Handicap International « Inclusive Employment Going Further.
Philip Davies What Is Evidence, and How Can It Improve Decision Making? Philip Davies International Initiative for Impact Evaluation.
1 External influences shaping the evaluation of a tertiary child protection program Andrew Anderson, The Benevolent Society.
Quasi Experimental and single case experimental designs
Evaluation Designs Adrienne DiTommaso, MPA, CNCS Office of Research and Evaluation.
Evaluation design and implementation Puja Myles
Agenda: Quasi Experimental Design: Basics WSIPP drug court evaluation Outcomes and Indicators for your projects Next time: bring qualitative instrument.
Program Evaluation Principles and Applications PAS 2010.
Philip Davies What Are Systematic Reviews, And Why Do We Need Them? Philip Davies International Initiative for Impact Evaluation [3ie]
What works in education: where and why? Howard White International Initiative for Impact Evaluation.
What is Impact Evaluation … and How Do We Use It? Deon Filmer Development Research Group, The World Bank Evidence-Based Decision-Making in Education Workshop.
World Bank Group Impact Evaluations: Relevance and Effectiveness Jeff Tanner Task Team Leader for Impact Evaluations Independent Evaluation Group Based.
Using Qualitative Methods to Identify System Dynamics and Inform System Evaluation Design Margaret Hargreaves Mathematica Policy Research American Evaluation.
IMPLEMENTATION AND PROCESS EVALUATION PBAF 526. Today: Recap last week Next week: Bring in picture with program theory and evaluation questions Partners?
Implementation Science: Finding Common Ground and Perspectives Laura Reichenbach, Evidence Project, Population Council International Conference on Family.
Writing for academia: Getting your medical education research into print: AMEE Guide No. 74 Medical Teacher 2012, e1–e9, Early Online Mitra Amini MD MPH.
Bringing Diversity into Impact Evaluation: Towards a Broadened View of Design and Methods for Impact Evaluation Sanjeev Sridharan.
Evaluation Nicola Bowtell. Some myths about evaluation 2Presentation title - edit in Header and Footer always involves extensive questionnaires means.
Systematic Reviews of Evidence Introduction & Applications AEA 2014 Claire Morgan Senior Research Associate, WestEd.
Chapter 6 Selecting a Design. Research Design The overall approach to the study that details all the major components describing how the research will.
Introduction to Evaluation
Assess Plan Do Review Resource 1: Types of Evaluation – which type of approach should I use? There are a number of different types of evaluation, the most.
Presentation transcript:

Philip Davies The Challenges Ahead for Impact Evaluation Studies in Poland Using Mixed Methods of Evaluation for Policy Making Purposes Philip Davies International Initiative for Impact Evaluation [3ie] 8 th Evaluation Conference, Evaluation in the System of Public Policies Warsaw, Poland, November 2012

Philip Davies What Is To Be Evaluated? Intervention effectiveness - what works? Implementation effectiveness – how/why it works? Diversity of effectiveness across different groups – what works for whom and when? Experiential effectiveness - users’ views Resource effectiveness - at what cost/benefit?

Philip Davies Theory Based Evaluation How is a policy/intervention supposed to work? Impact (or summative) evaluation Does the policy (programme, intervention) work? Process (or formative) evaluation How, why, and under what conditions does the policy (programme, intervention) work? Developmental evaluation Supporting innovation and adaptation. Providing feedback, generating learning, supporting direction or affirming changes in direction in real time. Building the evidence base over time. Which translates into:

Philip Davies The key to the challenges ahead is: What’s the Question?

Philip Davies Evidence for Policy Theories of Change How is the policy supposed to work? Logic Model

Philip Davies Evaluation: Theory of Change/Logic Model How is a policy/programme supposed to work? What activities, mechanisms, people have to be in place? And in what sequence – what is the causal chain? What resources are required – and are available? What data are required – and are available? Is the policy/programme feasible/achievable?

Philip Davies Evidence for Policy Theories of Change How is the policy supposed to work? Logic Model What is already known about this issue? Harness Existing Evidence Research Synthesis

Philip Davies Types of Research Synthesis Statistical Meta-Analyses(6-18 months) Narrative Systematic Reviews (6-12 Months) Rapid Evidence Assessments (1-3 Months) Evidence Maps and Gap Maps (1 Month) Meta-Ethnography /Qualitative Synthesis (6-12 Months)

Philip Davies Evidence for Policy Theories of Change How is the policy supposed to work? Logic Model What is already known about this issue? Harness Existing Evidence Research Synthesis What is the nature, size and dynamics of the problem? Administrative Data Surveys Census Data Qualitative Research Descriptive and Experiential Evidence

Philip Davies Evidence for Policy Theories of Change How is the policy supposed to work? Logic Model What is already known about this issue? Harness Existing Evidence Research Synthesis What is the nature, size and dynamics of the problem? What has been shown to work elsewhere? Evidence of Proven Effectiveness Experimental and Quasi- Experimental Evidence Descriptive and Experiential Evidence Administrative Data Surveys Census Data Qualitative Research

Philip Davies Experimental and Quasi-Experimental Evidence Randomised Controlled Trials Matched Comparison Designs - Propensity Score Matching Difference-in-Differences Regression Discontinuity Designs Interrupted Time Series Designs Instrumental Variable Analysis

Philip Davies Evidence for Policy Theories of Change How is the policy supposed to work? Logic Model What is already known about the problem/policy? Harness Existing Evidence Research Synthesis What is the nature, size and dynamics of the problem? What has been shown to work elsewhere? Evidence of Proven Effectiveness Experimental and Quasi- Experimental Evidence How do we make the policy work? Implementation Evidence Case Studies Interviews Focus Groups Ethnography Operations Research Descriptive Analysis and Experiential Evidence Administrative Data Surveys Census Data Qualitative Research

Philip Davies Impact Evaluations Evaluations of Outcome Attainment (Have targets been met?) Data Point 1 Data Point 2 Data Point 3 Priority Review Theory of Change Case Studies Interviews Focus Groups Ethnography Admin data Surveys Operations Analysis

Philip Davies Evidence for Policy Theories of Change How is the policy supposed to work? Logic Model What is already known about the problem/policy? Harness Existing Evidence Research Synthesis What is the nature, size and dynamics of the problem? What has been shown to work elsewhere? Evidence of Proven Effectiveness Experimental and Quasi- Experimental Evidence How do we make The policy work? Implementation Evidence/ Case Studies Interviews Focus Groups Ethnography Operations Research What are the costs and benefits of the policy ] Economic and Econometric Evidence Cost-Benefit/ Effectiveness/ Utility Analysis Descriptive Analysis and Experiential Evidence Administrative Data Surveys Census Data Qualitative Research

Philip Davies Economic Appraisal No policy programme or project should be adopted without first having the answer to these questions: Are there better ways to achieve this objective [Cost-Effectiveness Analysis] Are there better uses for these resources? [Cost-Benefit Analysis]

Philip Davies Evidence for Policy Theories of Change How is the policy supposed to work? Logic Model What is already known about the problem/policy? Harness Existing Evidence Research Synthesis What is the nature, size and dynamics of the problem? Descriptive and Experiential Evidence What has been shown to work elsewhere? Evidence of Proven Effectiveness Experimental and Quasi- Experimental Evidence How do we make The policy work? Implementation Evidence/ Case Studies Interviews Focus Groups Ethnography Operations Research What are the costs and benefits of the policy ] Economic and Econometric Evidence Cost-Benefit/ Effectiveness/ Utility Analysis What are the ethical implications of the policy? Ethical Evidence Social Ethics Public Consultation Administrative Data Surveys Census Data Qualitative Research

Philip Davies Theory of Change Activities Mechanisms People Resources Impact and Process Evaluations Systematic Review of All Evaluations Theorise SpecifyTest Review Refine The Evaluation Process Re-TestAccumulate

Philip Davies An Example: Conditional Cash Transfers Theory of Change Activities Mechanisms People Resources Impact and Process Evaluations Systematic Review of All Evaluations Theorise SpecifyTest Review CCTS incentivise parents to send children to school Improve enrolment and attendance Improve learning outcomes By enforcing the conditionality of schooling Changing the behaviour of parents and children Altering the relative costs and benefits of schooling versus other uses of children’s time >100 impact evaluations Many process evaluations Testing CCTs in Latin America, Asia, Africa Petrosino et al, (2012) Systematic Review of 23 studies Quality Education For All Children CCTs increase enrolments and attendance No overall impact on learning outcomes.

Philip Davies Theory of Change Activities Mechanisms People Resources Impact and Process Evaluations Systematic Review of All Evaluations Theorise SpecifyTest Review Refine Re-TestAccumulate Deadweight issues Levels of payment Payment recipient Reward performance Measure learning outcomes Supply-side interventions needed Teacher quality a mediating factor Phased transfers? Need to re-test refinements/mec hanisms Focus on contextual specificity Build an evidence base over time Establish generalisability And context specificity Avoid Type I/II errors

Philip Davies Thank you Philip Davies (0) Visit