1. Hopkins PSMG Oct 20051 An Incomplete Model for Trial Designs for Moving to Scale: The Whole Day Program C Hendricks Brown Sheppard Kellam Jeanne Poduska Amy Windham John Reid Carla Ford Natalie Keegan Supported by R01 DA15409-02 PI: Kellam Ro1 MH40859-17 PI: Brown Baltimore City Public School System

2. Hopkins PSMG Oct 20052 Premise The Whole Day Program is where the school would want to be in 5 years, systemwide. What specific steps should be examined along the way?

3. Hopkins PSMG Oct 20053 Primary Question Effective? Who benefits, for how long? Who doesn’t benefit, who is harmed? Efficacy Trial 12 Schools 24 Classrooms 600 Students 3 Time Periods

4. Hopkins PSMG Oct 20054 Differential impact on spring of first grade reading achievement by baseline reading (fall of first grade): control against two interventions Control Classroom / Parent Interventions Brown: Statistical Power

5. Hopkins PSMG Oct 20055 Presuming Effectiveness Sustainabililty Do WD teachers continue to sustain the same effects on children that they have in following years? 12 WD Teachers vs Themselves & 12 Wait- Listed Controls Scalability Can program be successfully implemented outside the original 12 schools throughout the school system?

6. Hopkins PSMG Oct 20056 Scale to What? From 2 To all 4 of the School Areas From 12 To 20 To 135 Schools From 24 To 40 To 400 First-Grade Classrooms

7. Hopkins PSMG Oct 20057 An Evidence-Based Decision Tree Efficacious especially w/ High-Risk Youth? Sustainable? Feasible to Scale? Y Y Go to Scale! Y N N N

8. Hopkins PSMG Oct 20058 The Real Decision Tree for BCPSS Something Different

9. Hopkins PSMG Oct 20059 Person, Place, and Time Child Classroom / Teacher (1 st Grade) School Area (Region of Baltimore) Time – Scale(s): 10 Secs, Min, Quarter, Year Child : Track from 1 st through 3 rd Grade Teacher: Track over 3 years (cohorts)

10. Hopkins PSMG Oct 200510 Randomization 20 Schools in 2 Areas randomly assigned to One of 12 schools where trial begins One of 8 schools that are “wait-listed” Children randomly assigned to classes within school Teachers/Classes randomly assigned to WD or wait-list

11. Hopkins PSMG Oct 200511 Baltimore Whole Day Trial Assignment of Classrooms/Teachers in First Year Half the Teachers are Wait-Listed for WD to 3 rd Year Replicate Same Design w/ SC and and WD BOTH in 12 Different Schools Brown: Statistical Power

12. Hopkins PSMG Oct 200512 Effectiveness Evaluation Compare 3 years of Outcomes for Children in the 12 WD classes to those in 12 Standard classes -- 1 st Cohort Examine Level of Program Implementation in the 12 WD and 12 Standard Classrooms in 1 st Cohort

13. Hopkins PSMG Oct 200513 Classroom Observations by Independent Observers 3 times per year WD and SC classrooms AM-Reading Instruction PM-Other subjects Teacher Practices Timed Observations Checklist Global Ratings of Quality Student Behavior On-Task/Off-task Ratings of Disruptive, Aggressive, and Shy Behavior

14. Hopkins PSMG Oct 200514 Classroom Observation Data Collection Form Teacher Practices and Student Behavior Actual Form

15. Hopkins PSMG Oct 200515 Domains of Reading Instruction Word Work – phonemic awareness, alphabetic instruction, letter recognition, work work Reading Comprehension – reading comprehension, previewing a book, reading books Language – oral language, vocabulary, reading own writing, writing, grammar Non-reading Instruction – interruptions, teacher out of classroom Giving instructionally related reading directions Feedback in response to behavior Feedback in response to a reading relevant comment or event

16. Hopkins PSMG Oct 200516 Classroom Observations Baseline Data Overview 12 schools 24 teachers (12 Whole Day, 12 Standard setting) Each teacher observed 3 times per year, at each time point: –1 reading instruction (am observation) –1 non-reading instruction (pm observation) Length of observations at each time point: Teacher: mean = 164 minutes Each student: mean = 9 min/student

17. Hopkins PSMG Oct 200517 Given Teacher Instructional Practices, Classroom Behavior (Standard Practice Classrooms during Reading Instruction at Baseline) Well behaved v Fairly well behaved: X 2 =36.83 p<10 -7 Word WorkReading Comprehension Language Well behaved216 67.3% 184 61.3% 84 39.6% Fairly well behaved 99 30.8% 109 36.3% 115 54.2% Poorly behaved 1.3% 7 2.3% 13 6.1% Chaotic5 1.6% 00

18. Hopkins PSMG Oct 200518 Student behavior: Reading v Other subjects (Standard Practice Classrooms at Baseline) Never off-task v Ever off-task: X 2 =29.80 p<10 -8 AM-ReadingPM-Other subjects Never off task1138 95% 679 88.3% Off task-1 interval13 1.1% 13 1.7% Off task-2 intervals5.4% 10 1.3% Off task-3 intervals42 3.5% 67 8.7%

19. Hopkins PSMG Oct 200519 Effectiveness Results for Cohort 1 (1 of 2) At Baseline Level of Off-Task Behavior is Well Balanced across Intervention Condition Reduction in Off-task Behavior Control 21% WD 9% 50% Reduction in Aggressive Behavior for Males in Less Structured Settings

20. Hopkins PSMG Oct 200520 Graphical Comparison for Off-Task Behavior

21. Hopkins PSMG Oct 200521 Comparison of Teachers

22. Hopkins PSMG Oct 200522 Unsolved Multilevel Modeling Issues Child Gender, Baseline Off-Task Classroom Intervention, Classroom Baseline Aggr School Time-Varying Classroom Context Binary Outcomes

23. Hopkins PSMG Oct 200523 Sustainability Evaluation (Funded) Multilevel Support Compare Teacher Implementation (and Child Outcomes) for 12 WD Teachers in 1 st through 3 rd Year for First Grade Classes

24. Hopkins PSMG Oct 200524 Sustainability Hypotheses and N’s & Years Teacher practices over time will be related to the quality of WD integration at the level of the School Building Team. 12  20  135 2 Teacher practices over time will be related to the quality of WD integration at the level of the School Area over time. 2  4

25. Hopkins PSMG Oct 200525 Scalability Design Change in fidelity (Teacher practices) and impact (Child Outcomes) in succeeding years after initial training 12  24  135

26. Hopkins PSMG Oct 200526 Combined RCT and Implementation Trial Trial After a Trial Design

27. Hopkins PSMG Oct 200527 How to Go From 24 to 400 Classes? A Dynamic Wait-Listed type of design Randomly Assign WHEN schools and classes obtain training, coaching, and full implementation Continue to assess intervention fidelity Paired-down measure of Teacher Practices Random Assign Children to Classes?

28. Hopkins PSMG Oct 200528 Classes Schools Waiting Active School Trained Wait Listed Sustained TrainedSustained

29. Hopkins PSMG Oct 200529 Plausible Time Scale # Classes (Schools) Time Period/Cohort New WDSustainabilityComparisonWait-Listed Effectiveness12 (12)012 (Same 12)376 (123) Sustain012 (12)12 (Same 12)376 (123) Scalability12 (12)12 (Same 12)20 (8)356 (115) Full Scalability Yr 1 90 (30)44 (20)40 (30)230 (85) Yr 290 (25)130 (50)40 (25)140 (60) Yr 390 (30)220 (70)40 (30)50 (35) Yr 490 (35)310 (100)0 (0)