1. Whole Day First Grade Program: Effectiveness, Sustaining, & Building a Base for Going to Scale Jeanne Poduska, ScD Center for Integrating Education and Prevention Research in Schools, American Institutes for Research Presented at: New Directions for Mental Health and Drug Abuse Effectiveness and Dissemination Trial Research and Methodology October 28, 2005

2. Contributors and Support Sheppard G. Kellam, M.D. 1 Jeanne Poduska, Sc.D. 1 C. Hendricks Brown, Ph.D. 2 Carla Ford, Ph.D. 1 Amy Windham, Ph.D. 1 Natalie L. Keegan 1 John Reid, Ph.D. 3 Bonnie Copeland, Ph.D. 4 Linda Chinnia, 4 National Institute on Drug Abuse, Division of Epidemiology, Services and Prevention Research, Prevention Research Branch 1R01 DA015409 and 1R01 DA019984 (and support from NICHD) 1 American Institutes for Research, Center for Integrating Education and Prevention Research in Schools 2 University of South Florida, Prevention Science Methodology Group 3 Oregon Social Learning Center 4 Baltimore City Public School System

3. Phases of Education and Prevention Trials

4. Developmental Epidemiology: A Strategy for Prevention Research Community Epidemiology Life Course Development Preventive Interventions

5. Research roots of WD First Generation of Baltimore trials tested separately Mastery Learning (ML) aimed at reading, and Good Behavior Game (GBG) aimed at classroom management of student behavior in First Grade. –Each had short-term impact on its target; GBG had long term impact on very high risk children. Second Generation tested combination of instructional improvement and GBG and showed impact on reading and behavior into middle school, still room for improvement. –A family/classroom program also seemed promising. Oregon Prevention Research Center tested family/ classroom partnership (LIFT program) and found impact among higher risk children.

6. Add Outcome Slide

7. How do you Establish Community & Institutional Partnerships? 1.Analyze social/political structure 2.Learn the vision of problems and priorities 3.Identify mutual self-interest within and across leadership 4.Fit your interest under the vision of leadership 5.Request ad-hoc oversight committee of leaders 6.Work through trust This process takes about a year

8. Functions of a Community & Institutional Board Establish vision and priorities Examine/critique/approve/support proposed programs Communicate constituents’ concerns, values, priorities, and acceptable language to program leaders Communicate program needs to constituent leaders Continually assess absent constituencies Work toward institutionalizing programs

9. Baltimore City Public School System Partners Patricia Welch, Ph.D., Chair, BCPSS Board of School Commissioners Bonnie Copeland, Ph.D., Chief Executive Officer Linda Chinnia, Chief Academic Officer Ben Feldman, Ph.D., Research, Evaluation & Accountability Officer Sue Cutter, Area 1 Interim Academic Officer James Smith, Area 2 Academic Officer Gwen Cleage, Curriculum and Instruction Officer La Verne Sykes, Director, Parent and Community Involvement Deborah Wortham, Ph.D., Director of Professional Development Delores Alexander, Coordinator of Reading Marietta English, President, Baltimore Teachers Union James Gittings, Public School Supervisors and Administrators Association 20 Schools, Principals, and Teachers Areas 1 & 2

10. Three Foci of Baltimore Whole Day First Grade Program (WD) Improve reading instruction in all academic subjects thru the day, based on prior Baltimore trials. Improve classroom behavior management, based on prior Baltimore trials. Improve parent/classroom partnership re reading and behavior, based on prior Baltimore and Eugene, Oregon trials.

11. Lessons Learned From First Two Generations of Trials in Baltimore Without a system to mentor, model, and monitor teacher practices over time, practices are not sustained. A multi-level institutional structure with mentoring and monitoring is required to implement interventions with fidelity. Teachers need support from principals; principals from area leaders; area leaders from chief academic and executive officers.

12. Goals of WD Trial Can we find synergy from the combination? Can the practices be sustained over time? Can practices be scaled up by school system with same effect?

13. Baltimore Whole Day First Grade Program Analytic Model Classroom Behavior Management Classrooms Decreased Aggressive, Disruptive Classrooms Teachers’ Effective Academic Instruction Poor Reading Skills Improved Reading Skills Decreased Later Conduct & Anti- social Personality Disorders Effective Family-Classroom Partnerships Other mediating or moderating variables: Family structure and poverty Deviant peers Whole Day Whole Day First Grade Education and Prevention Program Decreased Later Depressive Disorders Depressive Symptoms Decreased Depressive Symptoms Poor Achievement Increased Achievement Decreased Later Substance Abuse Individual Aggression Decreased Individual Aggression School Success & Decreased Drop-Out School building Community economic health, resources, drugs, and violence

14. Multi-Level Structure to Sustain Practices and Go to Scale BALTIMORE CITY PUBLIC SCHOOL SYSTEM Board of School Commissioners Chief Executive Officer Chief Academic Officer Professional Development Officer Curriculum & Instruction Officer Area Academic Officers Lead Coaches PrincipalsCoaches Teachers Research and Evaluation Office

15. Core Team BALTIMORE CITY PUBLIC SCHOOL SYSTEM Board of School Commissioner Chief Executive Officer Chief Academic Officer Professional Development Officer Curriculum & Instruction Officer Area Academic Officers Lead Coaches PrincipalsCoaches Teachers Research and Evaluation Office

16. Schools Committee BALTIMORE CITY PUBLIC SCHOOL SYSTEM Board of School Commissioner Chief Executive Officer Chief Academic Officer Professional Development Officer Curriculum & Instruction Officer Area Academic Officers Lead Coaches PrincipalsCoaches Teachers Research and Evaluation Office

17. Within-school Random Assignment* of Teachers/Classrooms and Children in the Whole Day First Grade Program IN EACH SCHOOL: WD SC (a)SC (b) Random Children randomly assigned to classrooms Teachers randomly assigned to classrooms *Brown & Liao. AJCP, 1999.

18. Effectiveness Design of the Whole Day First Grade Program: Follow 2 Cohorts of 1 st Graders to 3 rd Grade 2003-042004-052005-06 12 Teachers WD Cohort 1 Grade 1 Cohort 1 Grade 2 Cohort 2 Grade 1 Cohort 1 Grade 3 Cohort 2 Grade 2 12 Teachers SC Cohort 1 Grade 1Cohort 1 Grade 2 Cohort 2 Grade 1 Cohort 1 Grade 3 Cohort 2 Grade 2 Remaining Teachers

19. Sustainability Design of the Whole Day First Grade Program: Follow Teachers over Cohorts of 1 st Graders 2003-042004-052005-06 12 Teachers WD Cohort 1 Grade 1 Cohort 2 Grade 1 Cohort 3 Grade 1 12 Teachers SC Cohort 1 Grade 1 Cohort 2 Grade 1 Remaining Teachers

20. Scaling-Up Design of the Whole Day First Grade Program: 3 Cohorts of 1 st Graders 2003-042004-052005-06 12 Teachers WD Cohort 1 Grade 1 Cohort 2 Grade 1 Cohort 3 Grade 1 12 Teachers SC Trained in WD Cohort 1 Grade 1 Cohort 2 Grade 1 Cohort 3 Grade 1 Remaining Teachers Trained in WD Cohort 3 Grade 1

21. Design of the Whole Day First Grade Program 2003-042004-052005-06 12 Teachers WD Effectiveness Cohort 1 Grade 1 Effectiveness Sustainability Cohort 1 Grade 2 Cohort 2 Grade 1 Effectiveness   Sustainability Cohort 1 Grade 3 Cohort 2 Grade 2 Cohort 3 Grade 1 12 Teachers SC Effectiveness Cohort 1 Grade 1 Effectiveness Sustainability Cohort 1 Grade 2 Cohort 2 Grade 1 Effectiveness   Scaling Up (to WD) Cohort 1 Grade 3 Cohort 2 Grade 2 Cohort 3 Grade 1 Remaining Teachers Scaling Up (to WD) Cohort 3 Grade 1

22. Primary Measurement Objectives in the WD Trial Implementation Proximal effects of intervention on student behavior, achievement, and psychological well- being Longer term (distal) effects Teacher practices Hypothesized or known mediating and moderating variables Measures of the multi-level structure

23. Fidelity of Whole Day First Grade Program Implementation Checklist Excerpt CLASSROOM BEHAVIOR MANAGEMENT INDICATORS QUALITY RATING 20. Instructs students on class rules by:  Announcing the game and reviewing rules.  Setting and adhering to time limits for the game.  Providing appropriate independent assignments during the game. YYYYYYYY NNNNNNNN 1 2 3 4 5 6 Classroom Behavior Management Global Rating: Global Rating is not necessarily the sum of individual quality ratings. 1 2 3 4 5 6

24. Measures of Effectiveness Social Adaptational Status OUTCOMEMODEMEASURE ACADEMIC ACHIEVTStudent interviewWoodcock Johnson Houston Word Reading Scales Teacher InterviewTeacher Observation of Classroom Adaptation (TOCA) Record reviewSchool records, Standardized scores BEHAVIOR Time on-task/off-taskClassroom Obs.Teacher Observation/Student Engagement (TOSE) - Foorman Aggressive behaviorClassroom Obs. Teacher Interview TOSE TOCA Disruptive behaviorClassroom Obs. Teacher Interview TOSE TOCA Concentration prob’sClassroom Obs. Teacher Interview TOSE TOCA Social IsolationClassroom Obs. Teacher Interview TOSE TOCA

25. Measures of Effectiveness Psychological Well-being OUTCOMEMODEMEASURE DepressionStudent interviewDirect Clinical Observation Baltimore How I Feel AnxietyStudent interviewDirect Clinical Observation Baltimore How I Feel Flat affectStudent interviewDirect Clinical Observation HyperkinesisStudent interviewDirect Clinical Observation Bizarre behaviorStudent interviewDirect Clinical Observation

26. Teacher Practices Observations by Independent Observers (3x per year) Teacher Practices Timed Observation of Instruction (Foorman) Global ratings

27. Measures of Multi-Level System to Support Sustaining Practices and Scaling-Up Defined expectations for each level of the multi-level structure along four organizational variables: power/structure, tasks, resources, human factor Attendance logs and observers’ ratings of occurrence and quality for expectations 1 Core Team; 2 AAOs; 12 school teams

28. Chief Academic Officer Core Team Area Academic Officers & Lead Coaches School Coaches / Principals Monitoring and Mentoring Feedback Loop Mentoring Monitoring Data from Monitoring Effective Mentoring Teacher Practices

29. CLASSROOM BEHAVIOR MANAGEMENT INDICATORS QUALITY RATING 22. Plays the Good Behavior Game by:  Dividing students into balanced teams.  Reviewing class rules and announcing the game.  Setting and adhering to time limits for the game.  Placing checks on the board when individual students break rules.  Providing rewards to winning teams. YYYYYYYYYYYY NNNNNNNNNNNN 1 2 3 4 5 6 Classroom Behavior Management Global Rating: Global Rating is not necessarily the sum of individual quality ratings. 1 2 3 4 5 6 Fidelity of Whole Day First Grade Program Implementation Checklist Excerpt

30. Patterns of Quality of GBG Implementation in 12 Intervention Schools: 2004-05 High 4 - 6 ▲▲▲ Low 1 - 3 11182536 Week of Implementation Level of Implementation

31. Patterns of Quality of GBG Implementation in 12 Intervention Schools: 2004-05 High 4 - 6 ● ● ● ●● ● Low 1 - 3 ● ● ● ● 11182536 Week of Implementation Level of Implementation

32. Patterns of Quality of GBG Implementation in 12 Intervention Schools: 2004-05 High 4 - 6 ■ ■ Low 1 - 3 ■ ■■ ■■ ■ ■■ ■ 11182536 Week of Implementation Level of Implementation

33. Patterns of Quality of GBG Implementation in 12 Intervention Schools: 2004-05 High 4 - 6 ▲▲▲ ● ■ ▲▲▲ ● ● ● ■ ▲▲▲ ● ● Low 1 - 3 ● ● ■ ● ●■●■ ■ ■■ ■ 11182536 Week of Implementation Level of Implementation

34. Professional Development Model as Moving to Scaling Up AIR (Carla Ford) transitioning from classroom/school based focus to multi-level professional development Carla monitoring coaches; coaches monitoring teachers 3 Area Lead Coaches being trained fall 2005; train teachers in spring 2006 Developing monitoring system

35. Summary— Prevention and Service Researchers Researcher/Community Institution Partnerships Defining intervention and practices with precision Defining and measuring multi-level structure for sustaining and scaling-up practices Continual monitoring of practices deliberately linked to professional development Dual design requires equal emphasis on measurement of effectiveness and going to scale

36. Summary— Statisticians and Methodologists Modeling mediation of teacher practices Relationship of observational data on teacher practices and checklist of teacher practices: thick to thin data Modeling multi-level structure data Help us prepare for missing data in follow-up!