Why Use Randomized Evaluation? Presentation by Shawn Cole, Harvard Business School and J-PAL Presenter: Felipe Barrera-Osorio, World Bank 1 APEIE Workshop.

Slides:



Advertisements
Similar presentations
AFRICA IMPACT EVALUATION INITIATIVE, AFTRL Africa Program for Education Impact Evaluation Muna Meky Impact Evaluation Cluster, AFTRL Slides by Paul J.
Advertisements

The World Bank Human Development Network Spanish Impact Evaluation Fund.
The World Bank Human Development Network Spanish Impact Evaluation Fund.
The World Bank Human Development Network Spanish Impact Evaluation Fund.
Choosing the level of randomization
How to randomize? povertyactionlab.org. Quick Review: Why Randomize Choosing the Sample Frame Choosing the Unit of Randomization Options How to Choose.
Chapter 11 What Works and What Doesn’t. Are Hospitals Good for You? From Angrist and Pischke, Mostly Harmless Econometrics.
Povertyactionlab.org How to Randomize? Abhijit Banerjee Massachusetts Institute of Technology.
The counterfactual logic for public policy evaluation Alberto Martini hard at first, natural later 1.
What could go wrong? Deon Filmer Development Research Group, The World Bank Evidence-Based Decision-Making in Education Workshop Africa Program for Education.
Assessing Program Impact Chapter 8. Impact assessments answer… Does a program really work? Does a program produce desired effects over and above what.
1 Managing Threats to Randomization. Threat (1): Spillovers If people in the control group get treated, randomization is no more perfect Choose the appropriate.
Impact Evaluation: The case of Bogotá’s concession schools Felipe Barrera-Osorio World Bank 1 October 2010.
EXPERIMENTS AND OBSERVATIONAL STUDIES Chance Hofmann and Nick Quigley
Practical Sampling for Impact Evaluations Cyrus Samii, Columbia University.
SAMPLING AND STATISTICAL POWER Erich Battistin Kinnon Scott Erich Battistin Kinnon Scott University of Padua DECRG, World Bank University of Padua DECRG,
Experiments and Observational Studies.  A study at a high school in California compared academic performance of music students with that of non-music.
Cross-Country Workshop for Impact Evaluations in Agriculture and Community Driven Development Addis Ababa, April 13-16, 2009 AIM-CDD Using Randomized Evaluations.
1 Randomization in Practice. Unit of randomization Randomizing at the individual level Randomizing at the group level –School –Community / village –Health.
AADAPT Workshop Latin America Brasilia, November 16-20, 2009 Non-Experimental Methods Florence Kondylis.
Measuring Impact: Experiments
Global Workshop on Development Impact Evaluation in Finance and Private Sector Rio de Janeiro, June 6-10, 2011 Mattea Stein Quasi Experimental Methods.
AADAPT Workshop South Asia Goa, December 17-21, 2009 Nandini Krishnan 1.
Quasi Experimental Methods I Nethra Palaniswamy Development Strategy and Governance International Food Policy Research Institute.
CAUSAL INFERENCE Shwetlena Sabarwal Africa Program for Education Impact Evaluation Accra, Ghana, May 2010.
LT 4.2 Designing Experiments Thanks to James Jaszczak, American Nicaraguan School.
The days ahead Monday-Wednesday –Training workshop on how to measure the actual reduction in HIV incidence that is caused by implementation of MC programs.
Africa Impact Evaluation Program on AIDS (AIM-AIDS) Cape Town, South Africa March 8 – 13, Causal Inference Nandini Krishnan Africa Impact Evaluation.
Impact Evaluation Designs for Male Circumcision Sandi McCoy University of California, Berkeley Male Circumcision Evaluation Workshop and Operations Meeting.
Rigorous Quasi-Experimental Evaluations: Design Considerations Sung-Woo Cho, Ph.D. June 11, 2015 Success from the Start: Round 4 Convening US Department.
Chapter 3.1.  Observational Study: involves passive data collection (observe, record or measure but don’t interfere)  Experiment: ~Involves active data.
Nigeria Impact Evaluation Community of Practice Abuja, Nigeria, April 2, 2014 Measuring Program Impacts Through Randomization David Evans (World Bank)
Why Use Randomized Evaluation? Isabel Beltran, World Bank.
Applying impact evaluation tools A hypothetical fertilizer project.
Impact Evaluation for Evidence-Based Policy Making
Measuring Impact 1 Non-experimental methods 2 Experiments
Africa Impact Evaluation Program on AIDS (AIM-AIDS) Cape Town, South Africa March 8 – 13, Steps in Implementing an Impact Evaluation Nandini Krishnan.
TRANSLATING RESEARCH INTO ACTION What and How to randomize? July 9, 2011 Dhaka Raymond Guiteras, Assistant Professor University of Maryland povertyactionlab.org.
CHOOSING THE LEVEL OF RANDOMIZATION. Unit of Randomization: Individual?
Africa Program for Education Impact Evaluation Dakar, Senegal December 15-19, 2008 Experimental Methods Muna Meky Economist Africa Impact Evaluation Initiative.
AADAPT Workshop Latin America Brasilia, November 16-20, 2009 Laura Chioda.
Implementing an impact evaluation under constraints Emanuela Galasso (DECRG) Prem Learning Week May 2 nd, 2006.
IMPACT EVALUATION WORKSHOP ISTANBUL, TURKEY MAY
Overview of evaluation of SME policy – Why and How.
Bilal Siddiqi Istanbul, May 12, 2015 Measuring Impact: Non-Experimental Methods.
Social Experimentation & Randomized Evaluations Hélène Giacobino Director J-PAL Europe DG EMPLOI, Brussells,Nov 2011 World Bank Bratislawa December 2011.
Global Workshop on Development Impact Evaluation in Finance and Private Sector Rio de Janeiro, June 6-10, 2011 Using Randomized Evaluations to Improve.
Public Finance and Public Policy Jonathan Gruber Third Edition Copyright © 2010 Worth Publishers 1 of 24 Copyright © 2010 Worth Publishers.
What is Impact Evaluation … and How Do We Use It? Deon Filmer Development Research Group, The World Bank Evidence-Based Decision-Making in Education Workshop.
Africa Impact Evaluation Program on AIDS (AIM-AIDS) Cape Town, South Africa March 8 – 13, Randomization.
Impact Evaluation for Evidence-Based Policy Making Arianna Legovini Lead Specialist Africa Impact Evaluation Initiative.
Copyright © 2015 Inter-American Development Bank. This work is licensed under a Creative Commons IGO 3.0 Attribution-Non Commercial-No Derivatives (CC-IGO.
Common Pitfalls in Randomized Evaluations Jenny C. Aker Tufts University.
Development Impact Evaluation Initiative Innovations in investment climate reforms Paris, Nov 13, 2012 In collaboration with the Investment Climate Global.
Cross-Country Workshop for Impact Evaluations in Agriculture and Community Driven Development Addis Ababa, April 13-16, Causal Inference Nandini.
Kenya Evidence Forum - June 14, 2016 Using Evidence to Improve Policy and Program Designs How do we interpret “evidence”? Aidan Coville, Economist, World.
Why Use Randomized Evaluation?
Using Randomized Evaluations to Improve Policy
Measuring Results and Impact Evaluation: From Promises into Evidence
Quasi Experimental Methods I
Quasi Experimental Methods I
Using Randomized Evaluations to Improve Policy
Development Impact Evaluation in Finance and Private Sector
Impact Evaluation Toolbox
Implementation Challenges
Randomization This presentation draws on previous presentations by Muna Meky, Arianna Legovini, Jed Friedman, David Evans and Sebastian Martinez.
Randomization This presentation draws on previous presentations by Muna Meky, Arianna Legovini, Jed Friedman, David Evans and Sebastian Martinez.
Impact Evaluation Designs for Male Circumcision
Applying Impact Evaluation Tools: Hypothetical Fertilizer Project
Presentation transcript:

Why Use Randomized Evaluation? Presentation by Shawn Cole, Harvard Business School and J-PAL Presenter: Felipe Barrera-Osorio, World Bank 1 APEIE Workshop Ghana, May 10-14

Fundamental Question What is the effect of a program or intervention? – Does microfinance reduce poverty? – Does streamlining business registration encourage entrepreneurship? – Does auditing reduce tax evasion?

Explaining to grandparents Nicholas Kristoff, New York Times Columnist (11/20/2009) – “ One of the challenges with the empirical approach is that aid organizations typically claim that every project succeeds. Failures are buried so as not to discourage donors, and evaluations are often done by the organizations themselves — ensuring that every intervention is above average. Yet recently there has been a revolution in evaluation, led by economists at Poverty Action Lab at MIT. – The idea is to introduce new aid initiatives randomly in some areas and not in others [or to some people and not to others], and to measure how much change occurred and at what cost. This approach is expensive but gives a much clearer sense of which interventions are most cost- effective.”

Objective To Identify the causal effect of an intervention – Identify the impact of the program Need to find out what would have happened without the program – Cannot observe the same person with and without the program at the same point of time 4

Correlation is not causation Higher profits Credit Use OR ? 2) ? 1) Higher profits Business Skills Credit Question: Does providing credit increase firm profits? Suppose we observe that firms with more credit also earn higher profits.

6 (+6) increase in gross operating margin Illustration: Credit Program (Before-After) A credit program was offered in Why did operating margin increase?

Motivation Hard to distinguish causation from correlation by analyzing existing (retrospective) data – However complex, statistics can only see that X moves with Y – Hard to correct for unobserved characteristics, like motivation/ability – May be very important- also affect outcomes of interest Selection bias a major issue for impact evaluation – Projects started at specific times and places for particular reasons – Participants may be selected or self-select into programs – People who have access to credit are likely to be very different from the average entrepreneur, looking at their profits will give you a misleading impression of the benefits of credit 7

8 (+4) Impact of the program (+2) Impact of other (external) factors Illustration: Credit Program (Valid Counterfactual) * Macroeconomic environment affects control group * Program impact easily identified

Experimental Design All those in the study have the same chance of being in the treatment or comparison group By design, treatment and comparison have the same characteristics (observed and unobserved), on average – Only difference is treatment Yields unbiased impact estimates 9

Medical Trials Analogy Medical trials: – Take 1,000 subjects – Assign 50% to treatment group, 50% to control – On average Age in treatment and control group the same Pre-existing health in both groups the same Expected evolution of health in both groups the same – Track outcomes for treatment and control groups – “Gold standard” of scientific research Development projects – Many projects amenable to similar design

Options for Randomization Lottery (0nly some receive) – Lottery to receive new loans Random phase-in (everyone gets it eventually) – Some groups or individuals get credit each year Variation in treatment – Some get matching grant, others get credit, others get business development services etc Encouragement design – Some farmers get home visit to explain loan product, others do not 11

Lottery among the qualified Must receive the program Not suitable for the program Randomize who gets the program

Opportunities Budget constraint prevents full coverage – Random assignment (lottery) is fair and transparent Limited implementation capacity – Phase-in gives all the same chance to go first No evidence on which alternative is best – Random assignment to alternatives with equal ex ante chance of success 13

Opportunities for Randomization Take up of existing program is not complete – Provide information or incentive for some to sign up- Randomize encouragement Pilot a new program – Good opportunity to test design before scaling up Operational changes to ongoing programs – Good opportunity to test changes before scaling them up 14

Different levels you can randomize at – Individual/owner/firm – Business Association – Village level 15 – Women’s association – Regulatory jurisdiction/ administrative district – School level

Group or individual randomization? If a program impacts a whole group-- usually randomize whole community to treatment or comparison Easier to get big enough sample if randomize individuals Individual randomizationGroup randomization

Unit of Randomization Randomizing at higher level sometimes necessary: – Political constraints on differential treatment within community – Practical constraints—confusing to implement different versions – Spillover effects may require higher level randomization Randomizing at group level requires many groups because of within community correlation – Micro-credit program to treat 100,000 people. Choose Senegal and Gambia, and randomly offer program in one country. – What do we learn? – Similar problem if choose only 4 or only 10 districts 17

Elements of an experimental design 18 Random assignment Treatment GroupControl Group Participants  Non-participants Evaluation sample Potential participants TailorsFurniture manufacturers Target population SMEs

External and Internal Validity (1) External validity – The evaluation sample is representative of the total population – The results in the sample represent the results in the population  We can apply the lessons to the whole population Internal validity – The intervention and comparison groups are truly comparable –  estimated effect of the intervention/program on the evaluated population reflects the real impact on that population 19

External and Internal Validity (2) An evaluation can have internal validity without external validity – Example: A randomized evaluation of encouraging informal firms to register in urban areas may not tell us much about impact of a similar program in rural areas An evaluation without internal validity, can’t have external validity – If you don’t know whether a program works in one place, then you have learnt nothing about whether it works elsewhere. 20

Internal & external validity 21 Random Sample- Randomization Randomization National Population Representative Sample of National Population

Internal validity 22 Stratification Randomization Population Population stratum Samples of Population Stratum Example: Evaluating a program that targets women

23 Representative but biased: useless National Population Non-random assignment USELESS! Randomization Example: Randomly select 1 in 100 firms in Senegal. Among this sample, compare those with bank loans to those without.

Efficacy & Effectiveness Efficacy – Proof of concept – Smaller scale – Pilot in ideal conditions Effectiveness – At scale – Prevailing implementation arrangements -- “real life” Higher or lower impact? Higher or lower costs? 24

Advantages of “experiments” Clear and precise causal impact Relative to other methods – Provide correct estimates – Much easier to analyze- Difference in averages – Easier to explain – More convincing to policymakers – Methodologically uncontroversial 25

Machines do NOT Raise ethical or practical concerns about randomization Fail to comply with Treatment Find a better Treatment Move away—so lost to measurement Refuse to answer questionnaires  Human beings can be a little more challenging!

What if there are constraints on randomization? Some interventions can’t be assigned randomly Partial take up or demand-driven interventions: Randomly promote the program to some – Participants make their own choices about adoption Perhaps there is contamination- for instance, if some in the control group take-up treatment 27

Those who get receive marketing treatment are more likely to enroll But who got marketing was determined randomly, so not correlated with other observables/non-observables – Compare average outcomes of two groups: promoted/not promoted – Effect of offering the encouragement (Intent-To-Treat) – Effect of the intervention on the complier population (Local Average Treatment Effect) LATE= ITT/proportion of those who took it up

Assigned to treatment Assigned to control DifferenceImpact: Average treatment effect on the treated Non-treated Treated Proportion treated 100%0%100% Impact of assignment 100% Mean outcome Intent-to-treat estimate 23/100%=23 Average treatment on the treated

Randomly Encouraged Not encouragedDifferenceImpact: Average treatment effect on compliers Non-treated (did not take up program) Treated (did take up program) Proportion treated 70%30%40% Impact of encouragement 100% Outcome Intent-to-treat estimate 8/40%=20 Local average treatment effect

Common pitfalls to avoid Calculating sample size incorrectly – Randomizing one district to treatment and one district to control and calculating sample size on number of people you interview Collecting data in treatment and control differently Counting those assigned to treatment who do not take up program as control—don’t undo your randomization!! 31

When is it really not possible? The treatment already assigned and announced and no possibility for expansion of treatment The program is over (retrospective) Universal take up already Program is national and non excludable – Freedom of the press, exchange rate policy (sometimes some components can be randomized) Sample size is too small to make it worth it 32

Further Resources Google “Impact Evaluation World Bank” DIME group: JPAL, IPA presenter:

Thank You 34