1. Conducting RCTs in Schools: Challenges and Solutions 2007 AEA Annual Conference Research for Better Schools Kelly Feighan, Senior Research Coordinator Jill Feldman, Director of Evaluation

2. The Memphis Striving Readers Project: A Brief Overview One of eight (5-year) US DOE Striving Readers projects Includes a whole-school and targeted intervention to improve middle school students’ achievement in reading and core content areas, esp. struggling adolescent readers Conducted in eight urban, high-poverty middle schools (6 th – 8 th grades)

3. A Description of the School-Wide Intervention and Design Memphis Content Literacy Academy (MCLA) –Developed by team at University of Memphis –Provides intensive PD for content area teachers –Designed to increase teachers’ knowledge/use of SBRR To improve students’ reading achievement To improve students’ achievement in core content areas Matched pairs of schools randomly assigned to treatment or control

4. A Description of the Targeted Intervention: READ 180 TM READ 180 TM - Enterprise Edition (Scholastic, Inc., 2005) Designed as a 90 minute instructional sequence –20 minutes of whole group instruction –Three rotations (20 minutes each = 60 minutes total) Small group instruction (facilitated by the teacher) Use of READ 180 TM computer software Independent or modeled reading –10 minutes of whole-group wrap-up Nineteen READ 180 TM teachers, 48 sections

5. A Description of the READ 180 TM Research Design Random assignment of “eligible” students enrolled at 8 SR schools, where eligibility means: –No prior participation in READ 180 TM –Two or more grade levels behind in reading –Scores in bottom quartile on state assessment (TCAP) READ 180 TM is the treatment Counterfactual (business as usual*) is the control

6. Research Design (continued) Outcome measures 1.Iowa Test of Basic Skills (ITBS) Vocabulary Fluency comprehension 2.TCAP Reading Social Studies Science Mathematics

7. Designed to be a Study of “Effectiveness” Answers questions about whether or not READ 180 TM is likely to work in other contexts (not whether or not it “can” work) Requires a one-way “firewall” that prevents RBS from sharing feedback collected about implementation with MCS

8. A Successful RCT Study Depends on Collaborative Relationships… …with –MCS Office of Research, Evaluation and Assessment –Principals in each SR school (treatment and control) –Coaches –Teachers (in treatment and control schools) –Parents (of all eligible students, regardless of condition) –READ 180 TM publisher

9. Real World Challenges 1.Balancing local knowledge of students’ needs within the identified “eligible pool” without creating selection bias 2.Addressing high rates of student mobility 3.Accurately describing the counterfactual 4.Obtaining parental consent (and students’ assent) to administer the ITBS 5.Preventing crossover 6.Dealing with (inevitable) startup delays

10. One Pre-emptive Approach (that has worked for us, so far…) Hosting regularly scheduled monthly “chats” among evaluation team members to review plans and procedures related to: Random assignment Data Collection Data handling and analyses Interpretation of analyses Identification of potential threats to the RCT design Next steps

11. Approaches for Addressing Specific Challenges Challenges 1 & 2: 1.Balancing local knowledge of eligible students’ needs without creating selection bias 2.High rates of student mobility Approaches: 1.Allow principals to “opt out” eligible students before RA 2.Implement 2 waves of RA within first 20 days

12. Approaches for Addressing Specific Challenges Challenge 3: Describing the counterfactual Approaches: Review schools’ improvement plans Interview principals Interview instructional facilitators Examine students’ electronic rosters Create school profiles Remain vigilant and alert

13. Approaches for Addressing Specific Challenges Challenge 4: Obtaining (passive) parental consent and student assent Approaches: Providing forms at on-site registration Mailing forms to students’ home addresses

14. Approaches for Addressing Specific Challenges Challenge 5: Preventing crossover (while minimizing disruptions to the rostering process) Approaches: Provide each school with students’ assignments in all SR schools Permit schools to fill vacancies with 7 th & 8 th grade students not previously assigned to the control group RA of control group students when/if necessary

15. Approaches for Addressing Specific Challenges Challenge 6: Dealing with (inevitable) startup delays Approaches: Document specific delays Use data about actual implementation to interpret results Identify other sources of pressure that threaten high fidelity implementation

16. Preliminary Findings Year 1

17. Emerging Plans for Additional Analyses 1.Disaggregating descriptive analyses by grade and school 2.Examining whether scores of R180 students approach school means 3.Analyzing differences in teacher characteristics b/w R180 and ELA teachers by school 4.Exploring whether patterns exist among missing data 5.Investigating correlations b/w SRI data and outcomes

18. What Have Others Experienced? These were challenges and approaches we used… What challenges have you faced? How have you dealt with them? How have various approaches worked (or not)? Contact Information: feighan@rbs.org