Evaluating the Implementation of Child Welfare System Change
Introduction The evaluation of implementing system change is complex. The emerging science of implementation provides multiple frameworks that guide the evaluation
Why Evaluate? Document what IS implemented Effectiveness of Intervention(s) Effectiveness of Implementation Inform both through the stages of implementation
Why Evaluate? Development of ongoing Quality Assurance framework and process for sustainability Impact that implementation has on child welfare system
Complexity of IC Evaluation Evaluation of implementation of change at multiple levels: – Practice (EBP and non EBPs) – Organization – System
Complexity of IC Evaluation We also evaluate: – The process of implementation – Levels of fidelity – Outcomes of the intervention(s)
Evaluation System Change Matrix
Logic Model Each MCWIC implementation project includes a logic model that: – Provides an overall framework for implementation – Makes explicit the logical connections in a complex project – Identifies products, outcomes, and system impacts
Logic Model Logic model development is key to planning the evaluation The logic model is explicitly referenced in the development of: – Evaluation plans outlining all project evaluation activities – Work plans providing a timeline for project activities
Implementation Process Review Links implementation drivers to the logic model Measures fidelity of the implementation in reference to the logic model & work plan Provides ongoing monitoring of implementation Identifies emerging Project TA needs
Implementation Driver Driver Outputs Outputs from Logic Model Installation Activities During this Period Discussion / Next Steps/ Issues/ Technical Asst. Saliency Importance of driver during this time period TrainingRevision of existing DCF training to incorporate WICWA 2-day foundations training curriculum review began in February Teleconference between trainers, Intertribal TP Steering Committee & Green Bay TP has been held to discuss WICWA changes ID method that can best be used to determine the training needs of individual counties. MCWIC TA in development of Salience rating Lo Mid Hi
Evaluating Interventions & Implementation An implementation project often includes multiple specific interventions Each specific intervention has specific outcomes. Each intervention contributes to the implementation outcomes
Long Term Outcomes (examples) Organizational structure & functions identified to work more effectively with county service organizations (OH) Improved intake information collected, documented, & available (IN) Child Welfare staff will have increased understanding of the history and purpose of ICWA (WI)
System Impacts (examples) Improved working relationship between OFC and County Child Welfare Agencies (OH) Improved customer satisfaction with intake reporting experience (IN) Agency culture reflects a strength based perspective on parents (IA)
System Impacts / Child and Family Outcomes (examples) Increased number of cases maintaining tribal connection (WI) Shorter length of stay in foster care (IA) Increased use of relative care (IN)
Conclusions Evaluation is not a scary addition to an implementation project
Conclusions Rather, evaluation is an integral aspect of the implementation itself, providing: – A guiding logic to the implementation – A procedure for monitoring the progress of the implementation – A means of assessing implementation success as well as emerging needs and challenges
Questions? Comments? Tomatoes?