1 Implementation of Randomized Trials David Myers American Institutes for Research Washington, DC Prepared for IES/NCER Summer Research Training Institute, 2008

2 Context Conducting effectiveness trials and not efficacy trials Design and implementation of random assignment should not distort the program/intervention or the population served Programs should run at capacity

3 Illustrative Studies: Personal Involvement Upward Bound –Nationally representative study –67 sites and more than 3,000 students –Student RA within site – Workplace Literacy –3 sites and about 500 adult learners –Learner RA within site –"Addressing Literacy Needs at Work: Implementation and Impact of Workplace Literacy Programs. Final Report." Washington, DC: Mathematica Policy Research, Inc., October 1998

4 Illustrative Studies: Personal Involvement 21st Century Community Learning Centers –12 grantees (elementary school study) –Student RA within site – NYC Voucher Experiment –2000 students –Family RA –

5 Illustrative Studies: Personal Involvement Reading Comprehension –10 districts and 40 schools –School RA – Closing the Reading Gap –32 schools (4 interventions) –Student RA to reading groups within schools –

6 Illustrative Studies: Personal Involvement Quantum Opportunities –7 sites and about 1000 students –Student RA within site –

7 Practical Problems in Implementation Technical Political and Ethical Challenges Recruitment (not independent of P and E Challenges)

8 Further Exploration: Technical Cross-overs –Members of C get into T or something like T (use UB example and James Comer example) Into T -- straight forward adjustment (AIR, 1996) Something like T -- ?; affects interpretation Attrition –Post baseline -- “straight forward” adjustment (e.g., MI) –Before baseline -- problematic, no information –How to fix?????

9 Further Exploration: Technical Unbalanced designs –Don’t demand additional recruitment Large variance in selection probabilities (weights) –“over subscription” didn’t meet expectations (UB example) Dishonest assignment -- post randomization –Sites don’t tell all students they have been selected for the program (UB example) –Sites ignore RA and move controls into T

10 Further Exploration: Ethical and Political Challenges Random assignment isn’t fair –Programs afraid of denying services to students Is it fair to never give a student a chance? (UB example) Random assignment will force a program/teacher to serve a different population –Role of stratification to serve the desired mix Some students (units) must be served –Role of the “wild card” before randomization –Wild cards excluded from analysis

11 Further Exploration: Ethical and Political Challenges All “seats” must be filled –Role of the waiting list Random selection within strata, if needed and desired by program operators T and C groups analyzed as implemented at initial randomization When programs believe recruitment will distort the population –Identify “most likely” and “least likely” to serve –Prior to randomization and stratify in analysis

12 Further Exploration: Ethical and Political Challenges New treatments/interventions are hard to sell –Core programs vs. supplemental programs –Does the core program align with other curriculum and state assessments? Concerns about making AYP –Will a supplemental program reduce hours of instruction in a core area such as ELA?

13 Strategies for Recruitment: Schools Start with assistant superintendent for instruction (reading, math, science) or someone of similar stature Quickly develop a relationship with an office and not just an individual -- staff come and go Determine who needs to approve and to buy into participation –Superintendent -- they may need to go to the school board –Principals –Teachers –Parents and community groups

14 Strategies for Recruitment: Schools Be prepared to meet with them –Have a recruitment team Technical expertise in the design –Be prepared to tell audience why RA is valuable and not in a technical sense! Deep knowledge about the intervention -- most important Example of a meeting (next slide)

15 Irrefutable Evidence

16 Strategies for Recruitment: Schools Establish expectations for the researchers and the schools –Minimize legal talk Will bring in their general counsel and …. –Responsibilities and timelines Obtaining consent Data collection Administering instruments and tests Answering questions from parents and others