1. Implementation and Fidelity within a Theory-Driven Framework Greg Roberts, PhD. Vaughn Gross Center & National Center for Instruction The University of Texas at Austin

2. General Sources Theory-driven program evaluation  Theory types  Program modeling Implementation theory and research  Innovation diffusion, knowledge utilization, NIRN, CBAM, etc.  Predicting & managing change Measurement theory

3. Programs, Implementation, & Fidelity Programs, initiatives, funding mechanisms Adoption, implementation, use, evaluation Fidelity  Coverage  Dosage  Preserving causal mechanism  Scaling up  Lipsey meta-analysis (1999)

4. Notes on Programs and Fidelity Intended model: Implemented model Normative theory (program model)  The “what” and “how”  Activities, outputs, outcomes, moderators Causative theory (theory of change)  The “why”  Mediating factors

5. Theory-Driven Evaluation (Chen, 2004) Program Theory Impact Theory Intervening Mechanism Theory Normative Treatment Evaluation Outcome Theory Implementation Environment Theory Normative Theory Causative Theory Generalization Theory Normative Outcome Evaluation Normative Implementation Environment Evaluation Impact Evaluation Generalization Evaluation Treatment Theory Intervening Mechanism Evaluation

7. Program Models Models can not substitute for reality  Pattern matching Models can support:  Assessment of “evaluability”  Client needs identification  Program development, refinement, etc.  Monitoring and evaluation

8. Program Models Models can change  Formative processes  Standards & protocols regulating change  Embedded flexibility  Innovation

9. Notes on Implementation Models of systems change v. models of change inducement/management Stage-like nature Trends across theories  Change can be conceptualized at the individual, group, and social levels  Change is not an event  Change can not be mandated  Change can, at a very general level, be anticipated and managed

10. More Notes on Fidelity Implementation and evaluating fidelity Context for evaluating fidelity Considerations for conceptualizing fidelity  Multilevel nature of many interventions  Level and intensity of measurement aligned with need  Capacity for monitoring fidelity  Burden of monitoring fidelity  Alignment with desired outcomes

11. More Notes on Fidelity Tools for gathering fidelity data  Observation protocols  User logs  User self reports  Unobtrusive measures  Systemic indicators  Extant data

12. More Notes on Fidelity Analysis considerations  Structural models and measurement models  Latent variable models  Item response theory  Data quality

13. Questions?