1. Progressing Toward a Shared Set of Methods and Standards for Developing and Using Measures of Implementation Fidelity Symposium Chair: Chris S. Hulleman, Ph.D. Center for Assessment and Research Studies James Madison University Society for Research on Educational Effectiveness Annual Conference March 5, 2010

2. Implementation vs. Implementation Fidelity Descriptive What happened as the intervention was implemented? A priori model How much, and with what quality, were the core intervention components implemented? Implementation Assessment Continuum Fidelity: How faithful was the implemented intervention (t Tx ) to the intended intervention (T Tx )? Infidelity: T Tx – t Tx Most assessments include both

3. Linking Fidelity to Causal Models Rubin’s Causal Model: – True causal effect of X is (Y i Tx – Y i C ) – RCT is best approximation – Tx – C = average causal effect Fidelity Assessment – Examines the difference between implemented causal components in the Tx and C – This difference is the achieved relative strength (ARS) of the intervention – Theoretical relative strength = T Tx – T C – Achieved relative strength = t Tx – t C Index of fidelity

4. Why is this Important? Construct Validity – Which is the cause? (T Tx - T C ) or (t Tx – t C ) – Degradation due to poor implementation, contamination, or similarity between Tx and C External Validity – Generalization is about t Tx – t C – Implications for future specification of Tx Statistical Conclusion Validity – Variability in implementation increases error, and reduces effect size and power

5. 5-Step Process 1.Specify the intervention model 2.Develop fidelity indices 3.Determine reliability and validity 4.Combine indices 5.Link fidelity to outcomes

6. What do we measure? (1) Essential or core components (activities, processes) (2) Necessary, but not unique, activities, processes and structures (supporting the essential components of Tx) (3) Ordinary features of the setting (shared with the control group)

7. Presentations Catherine Darrow – Measuring Fidelity in Preschool Interventions: A Micro-analysis of Fidelity Instruments Used in Curriculum Interventions Michael Nelson et al. – A Procedure for Assessing Fidelity of Implementation in Experiments Testing Educational Interventions Anne Garrison & Charles Munter – Evaluating Math Recovery: A Case of Measuring Implementation Fidelity of an Unscripted, Cognitively-Based Intervention Chris Hulleman & David Cordray – Achieved Relative Intervention Strength: Models and Methods Carol O’Donnell – Discussant