1. Researchers Without Borders Webinar 4 A Framework and Suite of Instruments for Examining Fidelity of Implementation Jeanne Century Center for Elementary Mathematics and Science Education University of Chicago July 7, 2010 This work has been supported in part by the National Science Foundation Project ESI-0628052

2. Why Measure Fidelity of Implementation (FOI)? Determine if programs are effective Determine which parts of programs are effective Determine why and under what conditions parts of programs are effective

3. What is actually happening?

4. NSF Project Challenges Instruments that could be used across multiple programs A suite of instruments that could be used alone or in combination depending on resources and goals Provide detailed and specific information about FOI

5. Five Dimensions Approach Dane and Schneider (1998)  39/162 studies measured integrity  In the 39 studies, the authors defined “integrity” along one or more of five dimensions  They are: 1) adherence; 2) exposure; 3) quality of delivery; 4) participant responsiveness and 5) program differentiation Critical Components Approach Hall and Hord (1987), Bond et al. 2000; Huntley, 2005; Sabelli & Dede (2001) and others Some called them model dimensions, fidelity criteria, critical parts Structure and Process Approach Mowbray et al. (1987), Wang et al. (1984) Structure (composition of the intervention) Process (human interactions that take place during delivery) FOI Approaches…

6. Combining Frameworks Approach Ruiz Primo (2005) – matrix of 5 dimensions and structure/process Lynch and O’Donnell (2005); Lastica and O’Donnell (2007) – matrix of 5 dimensions and structure/process CEMSE Approach combine critical component approach and structure process approach show relationships to 5 dimensions approach use combination of approaches to create categories of critical components and a framework that can apply to multiple interventions

7. Questions?

8. FOI The extent to which an enacted program is consistent with the intended program model Critical Components The elements of a program model that are essential to its implementation FOI Operationalized The extent to which the critical components of a program are present when the program is enacted Definitions

9. Written Materials Developers Critical Component Identification theory… Users Challenges Theories aren’t always explicit Critical components very in “grain size” The “it” can vary, depending on how the intervention is framed Critical Component Identification evidence…

10. FOI Framework

11. FOI Framework and Critical Components

14. Questions?

15. Categories of Differentiation Mathematics And Science Science Only Mathematics Only Program Specific FOI Framework and Differentiation

16. C ategories of Differentiation (differentiation) Common Across Interventions Unique to Interventions CEMSE Framework Aligned with (Others’ Approaches)

17. Questions?

18. Instruments Teacher Questionnaire Teacher Instructional Observation Protocol Teacher Interview Protocol Teacher Instructional Log Teacher Attitude Questionnaire School Leader Questionnaire School Leader Interview Protocol School-wide Observation Protocol

19. Structural Procedural 12345678910101 1212 1313 1414 1515 1616 1717 Teacher Questionnaire XXXXXXXXXXXXXXXX School Leader Questionnaire XX Teacher Interview Protocol XXXXXXX School Leader Interview Protocol XX Classroom Observation Protocol XXXXXXXXXXXX School Wide Observation Protocol XXXXXX Teacher LogXXXXXXXXXXXX First Teacher LogXXXXX Teacher Attitude Questionnaire X Sample Instrument Construct Matrix

20. Categories of Differentiation Common Program A Program B Program C Impact of a single program Compare two or more programs Compare programs to “business as usual” Compare two versions of a program Uses of the Framework and Measures Developers U.S. Satellite, Inc. – additional elements Evaluators School–based change model – Lindblom High School Programmatic model – E2SP Researchers Trailblazers – look at a critical component in more depth

21. Programs Science and Technology for Children FOSS SEPUP IES Everyday Mathematics Science Companion

22. Jeanne Century Center for Elementary Mathematics and Science Education University of Chicago jcentury@uchicago.edu www.researcherswithoutborders.org FOI Supplement Work Researchers Without Borders Resources FAQs; Research; Tools and Instruments Learning Webinars; Collaboration Community Development www.researcherswithoutborders.org; This work has been supported in part by the National Science Foundation Project ESI-0628052