Benefits and limits of randomization 2.4

Tailoring the evaluation to the question Advantage: answer the specific question well – We design our evaluation to answer exactly the question we want to answer, – we choose the place and time Disadvantage: one evaluation only answers the specific question(s) it was designed to answer – One treatment and comparison group, creates one difference so usually answers only one question

General vs. specific research Indonesia family life survey – Survey on education, health, assets. Representative of 83% of Indonesia, 4 waves since every 1993 – Use to assess shocks or programs where happens to be some quasi randomness in implementation – Used for evaluating school building program, (Duflo, 2000) Community monitoring evaluation in Indonesia – Communities randomized to receive either more external or more local monitoring of road projects – Know from start we can compare these two approaches to monitoring – Collect exactly the data needed to monitor corruption by digging up random sample of road and weigh materials used

Few assumptions, transparent findings We know there is no selection bias, – only systematic difference between treatment and comparison groups is the program Fewer assumptions than for quasi randomized – that we have controlled for all differences – that the formula for administering the program cut off was strictly applied Direct comparison between two groups gives us our basic result – less complex analysis helps transparency of findings

Prospective evaluation What is a prospective evaluation? – Evaluation designed in advance Advantages – Collect specific data – Collaborative design and evaluation Disadvantages? – Long term results emerge in the long run – Q: What approaches could give us long run results in the short run?

When is a randomized evaluation not useful? When outcomes can only be measured at a very high level – Fixed vs. floating exchange rates – Freedom of the press When general equilibrium effects are important Outcome (like price) is determined by the aggregate of hundreds or thousands of interactions If we change some interactions but not others, we can only observe the aggregate effect Only possible to study when markets are somewhat separate

Example: job counseling in France Unemployed youth were given job counseling Those given counseling found jobs sooner than those not given counseling But what was the overall effect on youth employment: did the counseled youth just displace other youth? Randomized how many youth counseled by city Where more youth counseled, uncounseled did worse, i.e. displacement was a real problem Crepon et al 2013

Ethical considerations: background Belmont principles (and equivalent in other countries) establish ethical rules for research. Include 3 key principles: – Respect for persons: participants should be informed of risks and given a choice about participation – Benefice: the risks of research should be carefully weighed against the benefits. Risks should be minimized – Justice: the people (and the type of people) who take the risks should be those who benefit

Implications: approval for study Many universities have Institutional Review Boards who review all human subject research of students, faculty and staff May also require approvals from review board where study takes place Approval needed before study starts – Good to get general approval at conceptual stage to cover piloting – Provide detailed surveys and final numbers of subjects covered before study launch – Approvals usually need to be updated every year with updates of people reached and any adverse consequences

Respect for persons Informed consent must be gained from participants in the study – Waivers given if minimal risk and cost of collecting consent is high, e.g. knowing they are part of study will change behavior substantially Particular care needed for those who might not be able to judge risks well or find it hard to say no (e.g. children and prisoners) Do you need informed consent for all those in treatment communities when not everyone is surveyed? – Do people opt into the program or is entire community impacted? – Sometimes ask consent at community meeting Data that could identify an individual must be kept confidential – As early as possible take out information that could identify individuals – Use deidentified data for analysis and publication

Justice provision Test questions of relevance to those involved in the study Unethical to test a drug on prisoners and only sell drug to rich people Best if participants themselves gain from findings – E.g. the program found to work, scaled up in study location – Not always possible Ethical to test program designed to help poor Ethiopian farmers on poor Ethiopian farmers

Benefice principle Potential benefits must outweigh potential harm Researcher should seek to minimize risk Is researcher responsible for risk of program or only for the risk of harm associated with the evaluation? Program may have risks but that does not make evaluating it unethical – Are program participants informed of program risks? – Would the program have gone ahead anyway? – Researchers should seek to minimize risk in program they evaluate but important to evaluate risky programs that are going ahead anyway – Angrist, 1990 evaluated impact of draft lottery for Vietnam War

Risk of harm from evaluating Do fewer people receive the program because of the evaluation? – Ethical to reduce coverage if benefits of program unclear and evaluation can improve effectiveness of program Do different people receive the program? – Is the program less well targeted than it would have been without the evaluation? – Cost of less good targeting needs to be offset against gains from evaluation – Will evaluation help improve knowledge of who to target? Risk of confidential data becoming public – See respect for persons

What are the benefits? If we answer an important question well, this can have big benefits to society and those studied – If we find harm, program can be shut down – If we find benefit, program can be extended – Learn lessons potentially relevant to other situations The ability of randomized evaluations to learn about causality is relevant for ethics – The better we answer the question the higher the benefit, key component of ethics considerations