1. Ethics, Human Subjects Approval and Consent in Randomized Evaluations
The title of this presentation is “What is Evaluation”
Its primary goal is to create a common vocabulary so that we’re speaking the same language
Different disciplines have their own terms and possibly one of the main barriers to Public Health experts working with Economists working with Anthropologists working with practitioners working with USAID (for example) is that they’re using different vocabulary even though they’re saying the same thing
This lecture is not to “teach” you the correct vocabulary. Rather it’s to lay out the vocabulary we will be using. And rather than dictating that you use this vocabulary, I ask that you accept and at least for this week, adopt this vocabulary so that we don’t get stuck over semantics
But a large point of this lecture is to discuss “why evaluate”
And at the end, why randomize
William will get into the technical details of why randomize, and Bruno will cover some of the strategies of evaluation design that help us clear some political and ethical hurdles.
But I’d like to get some of our discomfort in the open, out of the way, so we can plow into the technical details, without slowing down because of that ethical, or political thorn in our side
Rachel Glennerster
Executive Director, J-PAL
J-pal

2. Overview Ethical principles in research
When ethical approval is needed
Respect for persons
Justice principle
The beneficence principle
Ethics of different forms of randomization
IRB regulations
An example
J-PAL 102x | Ethics

3. Ethical Considerations: Background
Belmont principles establish ethical rules for research that involves human subjects.
Include 3 key principles:
Respect for persons
Beneficence
Justice
Q: what ethical tradition do these principles derive from?
J-PAL 102x | Ethics

4. Implications: Approval for Research
Many universities have Institutional Review Boards
Research is considered to involve human subjects if
interacting with human subjects
collecting data on individuals
using data that includes personally identifying information
“Practice” is not covered but “research” is
J-PAL 102x | Ethics

5. Definition of Research
Precise definitions of research differ:
Most preparatory work is not research
If you are going to present results as evidence, its research
Q: do the ethical rules cover the evaluation or also the program being evaluated?
J-PAL 102x | Ethics

6. Practice vs Research: Two Extremes
In evaluation of Vietnam war: lottery draft was not subject to ethical approval
When researcher invents and test new vaccine on humans, the program (vaccine) is part of research
Potential criteria for judging if the program is covered by IRB rules
Is the researcher undertaking the program ?
Would the program have gone ahead without the evaluation?
Is program design of the program influenced by the evaluation?
Is the program novell?
In IRB documentation, explain the program being evaluated and role of evaluator
J-PAL 102x | Ethics

7. Respect for Persons: Implications
Informed consent needed for study participants
Particular care needed for children and prisoners
Data that could identify an individual must be kept confidential
J-PAL 102x | Ethics

8. Consent in Clustered RCTs: Who is Subject?
Q: if a study is randomized at the community level, who do we need to get consent from?
Depends on the line between practice and research:
One approach is to get “community approval”
In practice most IRBs take into account:
Cost of getting consent
Whether program is voluntary
The extent the evaluation changes the running of the program
Risks of the program
J-PAL 102x | Ethics

9. Beneficence principle: Implications
Potential benefits must outweigh potential harm
Researcher should seek to minimize risk
Is researcher responsible for risk of program or evaluation?
Program may have risks but that does not make evaluating it unethical
Q: what are the risks and benefits of evaluating a risky program?
J-PAL 102x | Ethics

10. Potential Risks of Harm from Evaluating
Do fewer people receive the program because of the evaluation?
Do different people receive the program?
Does the evaluation change targeting?
Will evaluation help improve knowledge of who to target?
Risk of confidential data becoming public
J-PAL 102x | Ethics

11. What are the Benefits?
Answer an important question is a benefit to society
The ability of randomized evaluations to learn about causality is relevant for ethics
J-PAL 102x | Ethics

12. Justice Provision: Implications
Test questions of relevance to those involved in the study
Unethical to test a drug on prisoners and only sell drug to rich people
Best if participants themselves gain from findings
Ethical if the types of people who gain are the types of people who are subjects
J-PAL 102x | Ethics

13. Ethics with Different Randomization Designs
J-PAL 102x | Ethics

14. Simple Treatment Lottery
Units randomized to receive access (or not) to the program
Potential disadvantage
Comparison group never gets program
J-PAL 102x | Ethics

15. Simple Lottery and Beneficence
Q: Is the beneficence principle compatible lottery?
Are fewer people being given access to the program?
Is there evidence program works
Would the program have gone ahead anyway?
Does the evaluation make targeting worse?
J-PAL 102x | Ethics

16. Example: Ethical Costs and Targeting
City 1
City 2
Least poor
Poorest
Program target
J-PAL 102x | Ethics

17. Lottery Around the Cutoff
Units scored on eligibility criteria
High scores admitted,
low scorers not admitted
Intermediate scores randomized
Most useful when:
Clear eligibility criteria
Program is over subscribed
J-PAL 102x | Ethics

18. Example: Credit Scoring in Philippines
Credit score
100
Accept all 60
85% treatment
15% comparison
Sample for study 45
60% treatment
40% comparison 30
Reject all
Karlan and Zinman, 2011
J-PAL 102x | Ethics

19. Ethics of lottery around the cutoff
Q: When is it ethical to admit some who have lower scores than others not admitted?
Q: When is it not ethical to use randomized cut off
Q: What are examples?
J-PAL 102x | Ethics

20. Randomized Phase-In
J-PAL 102x | Ethics

21. Ethics of randomized phase-in
Q: In a randomized phase in design, everyone gets the program eventually.
Does that mean never ethical costs?
J-PAL 102x | Ethics

22. Randomized Rotation Program rotates from one group to another randomly
Most useful when:
Resources limited and will not increase over time
Rotation is a natural part of a program
Advantage:
Everyone gets the program but always a control group
J-PAL 102x | Ethics

23. Ethics of Rotation Design
Q: to what extent does the rotation design reduce the risk of harm from evaluation?
Q: are there situations where rotation design might increase the risks?
J-PAL 102x | Ethics

24. Encouragement Design
People given an encouragement to take up the program
Most useful when:
Everyone is already eligible but take up is low
Advantage:
No one is denied access
J-PAL 102x | Ethics

25. Ethics of Encouragement Design
Q: As no one is denied access, are there no costs?
Program may be risky
If the program is likely beneficial and we can afford to encourage everyone, why deny some the encouragement
J-PAL 102x | Ethics

26. Achieving Compliance with IRB

27. Basic requirements for IRB
Research design including all surveys reviewed and approved before research starts
All personnel on the project are human-subjects certified
Precise wording of consent forms must be approved
Data management plan is approved and complied with
Annual updates of research status and subjects reached
Any breach in IRB compliance or any adverse events are reported to IRB committee
J-PAL 102x | Ethics

28. Personally Identified Information
Information that can be used to identify individual or households
Combination of information may identify someone
Note US health data must comply with HIPPA
J-PAL 102x | Ethics

29. Data Management Plans Paper surveys Data in digital form
Include method of recording consent on all surveys
Design questionnaire so that PII can be easily removed
Secure storage and disposal of surveys
Data in digital form
Encrypt all data with PII
Use encryption software for all devises
Separate PII from non-PII
J-PAL 102x | Ethics

30. Timing of Approval
Exploratory work, such as questionnaire design and piloting, may not count as “research”
Surveys have to be submitted for IRB approval, so creating survey is prior to IRB approval
If you want to publish your pilot it is research!
Be careful not to slip from pilot to research without getting approval
One strategy useful to get general approval at early conceptual stage as work starts to become more systematic over time, with larger scale piloting.
Before full scale study starts provide research design, detailed surveys and final number of subjects to IRB for approval.
Don’t start before it is approved!
Approvals usually need to be updated every year with updates of people reached and any adverse consequences
Exemptions: if study has very low risk of harm an exemption can be applied for
When risk is low, expedited review may be requested
When no PII is collected may ask for waiver or expedited review. Sometimes totally exempt (eg you do not collect the data)
J-PAL 102x | Ethics

31. Checklist for IRB compliance
All project staff have take IRB course and sent certifications*
Survey structured with PII-Consent detachable from questionnaire
Field staff sign a confidentiality agreement before working with data/surveys*
Using IRB approved consent form*
Unique ID code written on every page*
PII-Consent separated from questionnaire prior to data entry
Hard copies stored in a secure location*
* Required
J-PAL 102x | Ethics

32. IRB checklist continued
Use encryption for storage, transmission and access of PII data*
Make 2 backup copies (encrypted) of the original data*
Store backup copies on a secured server
Confirm data entry operators have removed data from their computers*
Separate PII from non-PII whenever possible
* Required
J-PAL 102x | Ethics

33. Check Local IRB rules
Rules and interpretation of rules for IRB vary widely by institution, country, and field
The information given here is illustrative
It is important to check the detailed rules in in the institution to which an individual researcher is based and the country in which a study is being carried out
J-PAL 102x | Ethics

34. Case Study of Challenges of Clustered Randomized Evaluation

35. Improving learning in India
Pratham is a large Indian education NGO
“every child in school….and learning well”
Successful urban program, new program for rural areas
Developed tool to test learning, community members generated village score cards
Facilitated village meeting where information on ways to improve education were shared, eg VEC
Trained volunteers to run after school reading camps
J-PAL 102x | Ethics

36. Research Design Randomized at village level Data collected on:
65 villages received information on how to improve education
65 received information and scorecards
65 received information, scorecards, and reading camps
85 in comparison group
Data collected on:
Children’s learning levels
Teacher absenteeism
Parents preferences and actions in promoting education
Village Education Committee members knowledge and actions
J-PAL 102x | Ethics

37. Who is the subject of the research?
All households? Children? Teachers? Village leaders? Those interviewed?
All impacted by the intervention, are they all impacted by research?
Indirect effects of any action can far reaching
Do we need to get permission of one shopkeeper before giving a loan to a neighboring shop keeper who might cut into their market?
Informed consent for what?
To be interviewed?
To have data collected on them? (eg teacher absenteeism)
To allow intervention to go ahead? (ie veto power)
J-PAL 102x | Ethics

38. What is research and what is practice?
Practice: Pratham regulated as an NGO in India
Right and ability to implement their program without informed consent of everyone in the village
Eg can provide information about villager rights without teacher or village leader giving their consent, though leader might be impacted
Pratham worked closely with researchers to design the program (drawing on their knowledge of what works), does that make it research?
Research: Systematic study leading to general lessons
Bad studies aren’t regulated, good ones are?
Changes made to implementation in order to evaluate (in this case none)
Data collection storage and analysis (informed consent needed)
J-PAL 102x | Ethics

39. Possible Criteria for Regulating CRT’s
Regulations for answering these questions are surprisingly underdeveloped. The following are some suggested criteria
Would the intervention have happened anyway? What change is due to the evaluation?
Subjects are those impacted by changes due to evaluation
Is participation in the intervention voluntary?
More careful assessment and consent rules for involuntary programs
Some deference to local standards and regulations
Eg right of community to collect information on absent teachers
Level of risk and benefit
Parent may punish their child if they find out they are performing badly
But benefit from study in improving education
J-PAL 102x | Ethics
J-pal

40. Conclusion
Ensuring compliance with ethical regulation and norms is hard work but very important
There is often no obvious right answer
Think carefully
Understand your own IRB as different institutions interpret the rules differently
Provide a general overview of the context in which the research fits
Eg what consent process makes sense
What is the implementation program being evaluated
Try and be clear about what is the marginal impact and role of the research as separate from the implementation
J-PAL 102x | Ethics