Evaluating Behavior Change: Demonstrating the Effects of Training Patricia Gonzalez, OSEP

Big Picture Big Picture The need for program evaluation within Federal agencies, including OSEP, has never been more pressing The need for program evaluation within Federal agencies, including OSEP, has never been more pressing For IDEA Part D, OSEP depends on individual grantees for evidence of program effectiveness For IDEA Part D, OSEP depends on individual grantees for evidence of program effectiveness The vast majority of OSEP projects carry out at least one personnel preparation, professional development or training activity as part of their project; therefore… The vast majority of OSEP projects carry out at least one personnel preparation, professional development or training activity as part of their project; therefore… OSEP depends on grantees to produce evidence of the effectiveness of “training” activities OSEP depends on grantees to produce evidence of the effectiveness of “training” activities

Logic Model Elements INPUTSACTIVITIES OUTPUTSOUTCOMES

OUTCOMES Short-TermIntermediateLong-Term Change in: Knowledge Knowledge Skills Skills Attitude Attitude Motivation Motivation Awareness Awareness Change in: Behavior Behavior Practices Practices Policies Policies Procedures Procedures Change in: Environment Environment Situation Situation Condition Condition

Issues in Measuring Behavior Change Who: all participants or a sample Who: all participants or a sample What: What: Use of evidence-based practices Use of evidence-based practices Fidelity measures to ensure practice is performed as trained Fidelity measures to ensure practice is performed as trained Measures that demonstrate effectiveness of “training” (e.g., comparisons pre/post, with non-participants, against standard protocols or criterion) Measures that demonstrate effectiveness of “training” (e.g., comparisons pre/post, with non-participants, against standard protocols or criterion)

Issues in Measuring Behavior Change Where: in actual or simulated settings Where: in actual or simulated settings How: observations, scored video tapes, How: observations, scored video tapes, self-recorded data Setting Events: organizational support Setting Events: organizational support for change