1. Elizabeth Spier, PhDJohannes Bos, PhD Principal ResearcherSenior Vice President FAST in Philadelphia SEPTEMBER 2015 Copyright © 2015 American Institutes for Research. All rights reserved. Keeping an RCT Relevant When Program Participation is Low

2. The Intervention 2

3. Families and Schools Together (FAST) is a school-based initiative intended to improve children’s academic and social outcomes. For this study, FAST is being implemented in 30 persistently-low-performing schools in Philadelphia. The School District of Philadelphia is the eighth largest in the United States, and serves many ethnically and linguistically diverse, high-poverty communities. Families and Schools Together 3

4. FAST invites students and their families into the school for a series of after-school sessions focused on improving connections within families, among families, and between families and schools Objectives include: Better communications and less stress at home Better social support for families Better behavior by the students (at home and in school) More positive school-home engagement and collaboration How FAST Works 4

5. Investment in Innovation (i3) grant from U.S. Department of Education, focused on school turnaround. School turnaround is achieved by getting and keeping children on track academically and socially. FAST was offered universally to all families of incoming kindergarten (grade 0) students. A relatively high level of FAST participation is necessary to improve outcomes across the school cohort (not just for individual students). The Current Project 5

6. FAST Logic Model 6

7. The Evaluation 7

8. Original Sampling Design 8 Random Selection Random Assignment 60 Eligible Schools 60 Schools 30 Schools FAST 2013 12 Students Cohort 1 12 Students Cohort 2 30 Schools FAST 2016 12 Students Cohort 1 12 Students Cohort 2

9. The sampling design was based on these assumptions: If FAST was offered to all incoming students and their families, most would choose to participate. Families who started to participate in FAST would continue to do so, so that most students’ families (at least 60%) would attend 4 or more sessions. If most students in the cohort participated in FAST, then impacts (if any) would be evident at the cohort level. Assumptions 9

10. Reality 10

11. Sampling Design Amplifies Consequences 11 Kindergarten Class

12. Sampling Design Amplifies Consequences 12 Kindergarten Classroom Expected Participants Kindergarten Class Reality

13. Sampling Design Amplifies Consequences 13 Kindergarten Classroom Expected Participants Study Sample Kindergarten Classroom Reality Study Sample

14. Sampling Design Amplifies Consequences 14 Kindergarten Classroom Study Sample

15. Alternative Supplemental Design 15 Kindergarten Classroom Enhanced study sample Study Sample

16. We are keeping the original RCT intact but redirected some study resources to a supplemental study to measure the effect of “treatment on the treated” For this study, we are using a two-level propensity score matching design that maximizes the contrast between participants and non-participants Note that this supplemental study answers somewhat different research questions! Alternative Supplemental Design 16

17. Multi-Level Mahalanobis Matching 17 Treatment Control

18. Multi-Level Mahalanobis Matching 18 12 3 4 5 6 78 84 7 5 1 62 3 Treatment Control STEP 1 Match 8 treatment schools with highest FAST participation rates to comparable control schools

19. Multi-Level Mahalanobis Matching 19 1234 5678 3 4 5 6 78 1234 5678 5 1 62 3 Treatment Control STEP 2 Collect baseline data on all kindergarten students and their families in these 16 schools

20. Multi-Level Mahalanobis Matching 20 TreatmentControl STEP 3 Identify participating children in treatment schools and match them to similar children in control schools

21. Multi-Level Mahalanobis Matching 21 TreatmentControl STEP 4 Use resulting sample to estimate impact of participating in FAST (treatment on the treated)

22. Closing Thoughts 22 When planning RCT with non-universal data collection, consider take-up and participation as a factor. Design studies to capture both individual-level “treatment on the treated” effects and school or classroom-level effects, ideally with comparable rigor for both. When possible, collect universal baseline data even in cases where you plan to use random subsamples for follow-up data collection (increases flexibility). Please stay tuned for impact results!

23. Elizabeth Spier, PhDJohannes Bos, PhD 650-843-8226650-843-8110 espier@air.orgjbos@air.org 2800 Campus Drive, Suite 200 San Mateo, CA 94403 General Information: 650-843-8100 TTY: 650-493-2209 www.air.org 23