1.  Identification of Child Maltreatment: Public Child Welfare Worker Training Evaluation Outcomes Chris Lee, M.S.W. Maria Hernandez, M.S.W. California Social Work Education Center (CalSWEC) University of California, Berkeley (UC Berkeley)

2.  2 Acknowledgments CalSWEC – UC Berkeley California Regional Training Academies / Inter- University Consortium Training Evaluation Team at CalSWEC  Cindy Parry, Ph.D.  Jane Berdie, M.S.W.  Chris Mathias, M.S.W.  Barrett Johnson, LCSW  Leslie Zeitler, LCSW  Salonje Rochell, B.A.

3. 3 We Will Discuss…  Brief Review of Training Evaluation Literature  Overview and History of CalSWEC Training Model  CalSWEC’s Use of Embedded Evaluations  Evaluation Analysis Results  Future Directions

4. 4 Training Evaluation Literature  “Training evaluation” refers to a systematic method for assessing or evaluating training  Donald Kirkpatrick’s (1956) levels of evaluation are considered the foundation for many training models used today  The first to establish levels of evaluation for training  Proposed that in order to effectively evaluate training for its impact, it must be sequentially evaluated at 4 levels:  Reaction  Learning  Behavior  Results

5. 5 Training Evaluation Literature  Embedded training evaluation is one approach to evaluating job-related skills (knowledge/behavior) while in the training classroom  Embedded training evaluation is designed to blend instruction with evaluation  Little evidence is available in the literature regarding use and viability of embedded training evaluation in public child welfare training

6. 6 California’s Statewide Training Evaluation Framework  California needed a unified training evaluation system  In response, a statewide collaboration was formed and developed a multi-level evaluation framework  The foundation of this framework is the Common Core training currently required for all new public child welfare workers, which also emerged from this collaboration  California’s current framework is influenced by other training evaluation models

7. 7 Influence of Other Models The American Humane Association Levels of Training Evaluation Parry, C. & Berdie, J. (1999). Training Evaluation in the Human Services. Washington DC: American Public Human Services Association.

8. 8 California’s Statewide Training Evaluation Framework Parry, C. & Berdie, J. (2004). Training evaluation framework report. Berkeley, CA: California Social Work Education Center. Parry, C. & Johnson, B. (2005). Training evaluation in a large system: California Framework for child welfare training evaluation. Proceedings of the Eighth Annual National Human Services Training Evaluation Symposium, 15-32.

9. 9 Training: Identifying Child Maltreatment  There are 7 total core training modules for public child welfare workers in California  Depending on the region, some workers attend this training during pre-service (before beginning work in the field), while others attend in-service (shortly after starting work in the field)  Training module: Child Maltreatment Identification (I, II)  Identifying neglect, physical abuse, emotional abuse (CMI-I)  Identifying sexual abuse & exploitation (CMI-II)  Each CMI module is 1.5 days long, involving instruction and an embedded evaluation administered at the end

10. 10 Embedded Evaluations In The Classroom  Embedded evaluations (one for each CMI module) consist of 4 scenarios that may or may not describe a case of maltreatment  CMI-I evaluation assesses ability to identify physical abuse  CMI-II evaluation assesses ability to identify sexual abuse  Trainees read case scenarios on their own and answer a series of questions for each scenario, ultimately deciding in each case if maltreatment did or did not occur  Embedded evaluations are administered on non-carbon copy (NCR) paper

11. 11 Embedded Evaluations In The Classroom  Trainees are given an allotted amount of time to complete the embedded evaluation (EE)  Instructor collects completed EEs, then reviews case scenarios and test questions with trainees and engages them in discussion  EEs are generally used for evaluating skill; however the EEs used in the CMI modules evaluate both knowledge and skill

12. 12 Data and Methods  Current analysis: Explored associations between trainee characteristics (California Common Core Curricula Demographic Survey) and CMI embedded evaluation results  Utilized logistic regression to analyze trainee characteristics and their likelihood of passing CMI embedded evaluations  “Passing” was defined as correctly identifying whether maltreatment did or did not occur on at least 3 out of 4 case scenarios  Time frame of data collection: January 1, 2007 to December 31, 2008  Final Sample Sizes  CMI-I (n = 1282)  CMI-II (n = 896)

13. 13 Demographic Handouts

14. 14 CMI-I and CMI-II Evaluation Results

15. 15 Embedded Training Evaluation and “Chain of Evidence”  Review: As indicated in the literature, EEs are useful for instruction, evaluation and skill building  CalSWEC: EE results are used to inform revisions to curricula, training and future knowledge/skill evaluations  Use of EEs help to strengthen the skills level of the training evaluation framework, moving closer to the next level: transfer of learning

16. 16 Future Directions: Next Links In The Chain  Further explorations of existing data in order to strengthen foundational training components such as: curriculum, training in the classroom  Continue to expand the efficacy of each level (e.g., revise the CMI-I (EE) to also assess neglect)  Integrate training in collaborative research projects (e.g., CMI-II training & Child Forensic Attitude Scale)  Work through remaining levels of the training evaluation framework and eventually reach the final link in the chain: influencing client/agency outcomes

17. 17 References CalSWEC. (In Press). Evaluation of the California common core for child welfare training, implementation status, results and future directions [White paper]. Berkeley, CA: California Social Work Education Center. Kirkpatrick, D. (1959). Techniques for evaluating training programs. Journal of the American Society of Training Directors, 13 (3-9), 21-26. McCowan, R.J. & McCowan, S.C. (1999). Embedded evaluation: Blending training and assessment. Buffalo, NY: Research Foundation of SUNY/Center for Development of Human Services. Retrieved from http://www.eric.ed.gov/ERICWebPortal/custom/portlets/recordDetails/detailmini.jsp?_nfpb=true&_&ERI CExtSearch_SearchValue_0=ED501715&ERICExtSearch_SearchType_0=no&accno=ED501715 http://www.eric.ed.gov/ERICWebPortal/custom/portlets/recordDetails/detailmini.jsp?_nfpb=true&_&ERI CExtSearch_SearchValue_0=ED501715&ERICExtSearch_SearchType_0=no&accno=ED501715 Parry, C. & Berdie, J. (1999). Training Evaluation in the Human Services. Washington DC: American Public Human Services Association. Parry, C. & Berdie, J. (2004). Training evaluation framework report. Berkeley, CA: California Social Work Education Center. Parry, C. & Johnson, B. (2005). Training evaluation in a large system: California Framework for child welfare training evaluation. Proceedings of the Eighth Annual National Human Services Training Evaluation Symposium, 15- 32.

18.  18 For more information on training evaluation in California please visit the CalSWEC website: http://calswec.berkeley.edu/CalSWEC/CWTraining.html Leslie Zeitler, Training and Evaluation Specialist lzeitler@berkeley.edu Maria Hernandez, Graduate Student Researcher hernandezm@berkeley.edu Chris Lee, Graduate Student Researcher clee07@berkeley.edu lzeitler@berkeley.edu hernandezm@berkeley.edu clee07@berkeley.edu