1. Conceptualizing Intervention Fidelity: Implications for Measurement, Design, and Analysis Implementation Research Methods Meeting September 20-21, 2010 Chris S. Hulleman, Ph.D.

2. Implementation vs. Implementation Fidelity Descriptive What happened as the intervention was implemented? A priori model How much, and with what quality, were the core intervention components implemented? Implementation Assessment Continuum Fidelity: How faithful was the implemented intervention (t Tx ) to the intended intervention (T Tx )? Infidelity: T Tx – t Tx Most assessments include both

3. Linking Fidelity to Causal Models Rubin’s Causal Model: – True causal effect of X is (Y i Tx – Y i C ) – RCT is best approximation – Tx – C = average causal effect Fidelity Assessment – Examines the difference between implemented causal components in the Tx and C – This difference is the achieved relative strength (ARS) of the intervention – Theoretical relative strength = T Tx – T C – Achieved relative strength = t Tx – t C Index of fidelity

4. Implementation assessment typically captures… (1) Essential or core components (activities, processes, structures) (2) Necessary, but not unique, activities, processes and structures (supporting the essential components of Tx) (3) Best practices (4) Ordinary features of the setting (shared with the control group) Intervention Fidelity assessment

5. Why is this Important? Construct Validity – Which is the cause? (T Tx - T C ) or (t Tx – t C ) – Degradation due to poor implementation, contamination, or similarity between Tx and C External Validity – Generalization is about t Tx – t C – Implications for future specification of Tx – Program failure vs. Implementation failure Statistical Conclusion Validity – Variability in implementation increases error, and reduces effect size and power

6. Why is this important? Reading First implementation results ComponentsSub- components Performance LevelsARS RFNon-RF Reading Instruction Daily (min.)105.087.00.63 Daily in 5 components (min.) 59.050.80.35 Daily with High Quality practice 18.116.20.11 Overall Average0.35 Adapted from Gamse et al. (2008) and Moss et al. (2008) Effect Size Impact of Reading First on Reading Outcomes =.05

7. 5-Step Process (Cordray, 2007) 1.Specify the intervention model 2.Develop fidelity indices 3.Determine reliability and validity 4.Combine indices 5.Link fidelity to outcomes Conceptual Measurement Analytical

8. Some Challenges Intervention Models Unclear interventions Scripted vs. Unscripted Intervention Components vs. Best Practices Measurement Novel constructs: Standardize methods and reporting (i.e., ARS) but not measures (Tx-specific) Measure in both Tx & C Aggregation (or not) within and across levels Analyses Weighting of components Psychometric properties? Functional form? Analytic frameworks Descriptive vs. Causal (e.g., ITT) vs. Explanatory (e.g., LATE) See Howard’s Talk Next! Future Implementation Zone of Tolerable Adaptation Systematically test impact of fidelity to core components Tx Strength (e.g., ARS): How big is big enough?

9. Treatment Strength (ARS): How Big is Big Enough? Effect Size StudyFidelity ARS Outcome Motivation – Lab 1.880.83 Motivation – Field 0.800.33 Reading First* 0.350.05 *Averaged over 1 st, 2 nd, and 3 rd grades (Gamse et al., 2008).

10. Thank You! And Special Thanks to My Collaborators: Catherine Darrow, Ph. D. Amy Cassata-Widera, Ph.D. David S. Cordray Michael Nelson Evan Sommer Anne Garrison Charles Munter

12. Extras and Notes

13. Achieved Relative Strength = 0.15 Infidelity “Infidelity” (85)-(70) = 15 t C t tx T Tx TCTC.45.40.35.30.25.20.15.10.05.00 Treatment Strength Expected Relative Strength = T Tx - T C = (0.40-0.15) = 0.25 100 90 85 80 75 70 65 60 55 50 Outcome

14. Linking Fidelity to Outcomes

15. Concerns and Questions Best practices vs. model-specific Fidelity – i.e., be wary of measures that find 100% fidelity! Fidelity as Achieved Relative Strength (ARS) How much ARS is enough? How to include ARS/fidelity into analytic framework – Weighting of core components – Combining (or not) of indices – Analytic framework Descriptive Causal (ITT) Explanatory (LATE, TOT, instrumental variables) ARS within multi-level models Combining indices within and across levels Zone of tolerable adaptation – Researcher vs. teacher perspective (fidelity is bad b/c I’m like everyone else and need to be creative) Fidelity to process vs. structure Fidelity as a moderator/mediator – When does fidelity = mediation, when does it not? – Tolerable adaptation vs. moderation Developmental studies – Develop valid and reliable indices – Determine which components matter – Implementation drivers