1. What the NCWWI Evaluators have been up to… A love story Changing... Leading... Learning...

2. NCWWI Evaluation Team Updates | February 10, 2015 A Service of the Children’s Bureau Evaluation Team Anna de Guzman Research Coordinator for UP anna.deguzman@du.edu Robin Leake Lead Evaluator Robin.Leake@du.edu Salvador Armendariz Research Coordinator for LAS, LINKD (webinars), and OI Salvador.Armendariz@du.edu Shauna Rienks Research Analyst Shauna.Rienks@du.edu Amy Grenier Research Coordinator for LAMM and LADD amy.grenier@du.edu Kate Lawrence, PhD University at Albany Evaluation clawrence@albany.edu clawrence@albany.edu Ann Obermann Ph.D. Student Ann.obermann@du.edu 2

3. Still finishing NCWWI 1 Wrapped up the Leadership Academy evaluation and prepared summary reports – 12 month surveys for 64 LAS participants – 12 month surveys for 35 LAMM participants – Summary reports completed and posted online Traineeships – Last cohort of 36 students from 7 schools – Final report NCWWI Evaluation Team Updates | February 10, 2015 A Service of the Children’s Bureau 3

4. NCWWI Evaluation Team Updates | February 10, 2015 A Service of the Children’s Bureau LADD Evaluation Question Focus Working in partnership Improvement of individual leadership competencies Implementing systems change Implementing systems change in the WE sites 4

5. LADD 360 Feedback with 6 Deans and 7 Directors Coaching Notes Interviews in late spring, 2015 Analysis of jurisdiction change initiatives & alignment (flow down) in WE sites NCWWI Evaluation Team Updates | February 10, 2015 A Service of the Children’s Bureau5

6. NCWWI Evaluation Team Updates | February 10, 2015 A Service of the Children’s Bureau UP Evaluation Questions Focus Implementation of innovative strategies for: o Child welfare education o Workforce retention strategies University/Agency partnerships to support transition from school to work o Impact of facilitator support Support from LADD Stronger outcomes in WE sites? 6

7. University Partnerships Quantitative Data: Baseline SSI from 63 students across 12 programs (93%) Qualitative Data: Facilitator notes, Semi-annual reports Review of local workforce change efforts Annual SSI spring/summer – Inclusion of NCWWI 1 graduates Interviews in spring of 2015 with PIs, agency partners, facilitators, students, and field instructors Study of programs’ field placement competency assessments NCWWI Evaluation Team Updates | February 10, 2015 A Service of the Children’s Bureau 7

8. NCWWI Evaluation Team Updates | February 10, 2015 A Service of the Children’s Bureau Organizational Intervention Evaluation Question Focus Implementation of Org Intervention Changes in the health of the workforce to drive implementation of CIs Value-added of Org Intervention Improved readiness of the agency to implement systems change 8

9. NCWWI Evaluation Team Updates | February 10, 2015 A Service of the Children’s Bureau Organizational Intervention COHA: o Baseline and 3-yr follow-up for all WE sites Statewide Comparison group in Missouri 6-month surveys for site teams o Readiness to implement systems change o 6 and 12 month satisfaction and Progress Towards Goals surveys Review of implementation plans & facilitator notes 9

10. NCWWI Evaluation Team Updates | February 10, 2015 A Service of the Children’s Bureau LAMM & LAS Evaluation focus Effective and efficient strategies for developing leadership skills Impact of integrated and focused leadership development Impact of additional supports for Managers (360 feedback and coaching) 10

11. LAMM and LAS Competency and TOL Surveys Case studies with selected Managers and Supervisors (4 per site) o Interviews with participants and supervisors, in- depth study of change initiatives Analysis of Change Initiatives Review of LASLN and Coaching notes NCWWI Evaluation Team Updates | February 10, 2015 A Service of the Children’s Bureau11

12. Upcoming Interviews Schedule NCWWI Evaluation Team Updates | February 10, 2015 A Service of the Children’s Bureau Who?How Selected?When? LAMM Participants (SF Cohort)Random selection (5), 1-2 for case study July 2015 LADD Deans and DirectorsAllMarch-April 2015 LAS Participants (SF Cohort)Random selection (5), 1-2 for case study July 2015 LAS State Coordinator (SF Cohort)AllMarch 2015 UP Principal Investigators and Agency Partners All (except Arizona and Case Western Reserve) March-April 2015 UP FacilitatorsAllearly May 2015 UP Field InstructorsRandom selection from each schoolMarch-April 2015 UP StudentsRandom selection (10)March-April 2015 UP GraduatesRandom selection (10)June-July 2015 Org Intervention COHA InterviewsBy invitation, by job positionJune 8-12, 2015 12

13. WE Site: San Francisco COHA in summer of 2014 339 surveys, response rate of 83% 36 interviews and focus groups across Seneca and HSA Reports and dissemination of findings 13 NCWWI Evaluation Team Updates | February 10, 2015 A Service of the Children’s Bureau

14. WE Site: San Francisco LAMM: – Baseline surveys from 18 managers, response rate of 100% – Post-training surveys from 15 managers – 13 managers participated in 360 Feedback – Selection of managers for case studies (2 from Seneca, 2 from HSA) Case study interviews begin in Spring 14 NCWWI Evaluation Team Updates | February 10, 2015 A Service of the Children’s Bureau

15. WE Site: San Francisco LAS: – Baseline surveys from 16 supervisors response rate of 100% – LASLN Facilitation notes – SF plan to support implementation of Supervisor Change Initiatives – Selection of supervisors for case studies (2 from Seneca, 2 from HSA) Case study interviews begin in Spring 15 NCWWI Evaluation Team Updates | February 10, 2015 A Service of the Children’s Bureau

16. Workforce Snapshot: San Francisco HSA Highly educated, highly paid, stable workforce Underperforming Unclear roles and practice standards Low morale CalSWEC Resource-rich community Heavy union involvement 16 NCWWI Evaluation Team Updates | February 10, 2015 A Service of the Children’s Bureau

17. WE Site: Missouri COHA in January of 2015 – 307 surveys, response rate of 86% – 35 interviews and 17 focus groups across four counties – Survey administered statewide to more than 1,500 staff 38% statewide – Reports and dissemination of findings in May, 2015 17 NCWWI Evaluation Team Updates | February 10, 2015 A Service of the Children’s Bureau

18. Workforce Snapshot: Missouri Children’s Division University-Agency partnerships getting stronger Workforce with few social workers Poor training system Lowest pay in the nation High turnover No practice model 4 unique sites (morale, community resources, leadership) 18 NCWWI Evaluation Team Updates | February 10, 2015 A Service of the Children’s Bureau

19. NCWWI Evaluation Team Updates | February 10, 2015 A Service of the Children’s Bureau What have we learned so far about the 360 Feedback? Supports the LAMM o Aligns with the Strength Finder Incongruity between self and reporter ratings Reports plus coaching is the RIGHT approach Value added for LADD? o Challenged by lack of focus on individual Leadership skills 19

20. What have we learned so far about the COHA?  WDF and COHA fit together like a hand in a glove  COHA is FOUNDATIONAL to all of the WE activities  Need to find a way to better integrate the UPs  Need to include community partners/providers 20 NCWWI Evaluation Team Updates | February 10, 2015 A Service of the Children’s Bureau

21. What are we still figuring out? How to evaluate the alignment of the Change Initiatives – Focused on developing structured approaches to developing and documenting CIs across the program areas How/whether to push for standardized approaches to competency assessment of traineeship students in field placements Final design for the Case Studies 21 NCWWI Evaluation Team Updates | February 10, 2015 A Service of the Children’s Bureau