Findings from the National Evaluation Of Child Welfare Training Grants: Lessons and Implications Mary Elizabeth Collins, MSW, Ph.D. Maryann Amodeo, MSW, Ph.D. Cassandra Clay, MSW, EdM. Boston University School of Social Work May 2007 Grant #9OCT0124 funded by U.S. DHHS, Children’s Bureau

National Evaluation of Child Welfare Training Grants §Funded by the Children’s Bureau (Oct Sep 2006) §Large project: multiple data collection tasks and several research questions §Overall goal: Greater understanding of federally-funded CWT and how to use information from the evaluation to improve our training strategies.

National Evaluation of Child Welfare Training: Four Main Components §Case studies of 9 previously federally funded training projects – Independent Living §Comprehensive review of child welfare training literature §Survey of faculty of (mostly) schools of social work §Survey of state (and some county) child welfare training administrators

Update on Four Components §Multiple case study of IL projects – DONE, report available §Review of child welfare training literature – DONE, report available §Phone interview of state child welfare training directors – 90% done, response rate=48/51, great data! §Mail survey of faculty – 60% done, N=97; response rate=54%

Conceptual Model

Recommendations from Case Study Report: Children’s Bureau, Future Grantees, Public Agencies §Youth/Consumer Involvement e.g., Public agencies have been making progress in emphasizing more partnership with services users; training initiatives and strategies are another mechanism by which agencies can engage consumers in their work. §Evaluation e.g., Public child welfare agencies need to cooperate with the evaluation task. This will include allowing evaluators to conduct follow-up with trainees from the state agency. Examination of the transfer of learning to the agency setting and mastering skills in the work setting is sorely needed. This will nearly always require follow-up in the practice setting which will include data collection involving interviews, observations, case record reviews, and other methods. Additionally, access to comparison groups may be needed. §Collaboration e.g., Grantees should aim to establish collaborations with organizations rather than individuals to provide more stability to the collaborations. Collaborators should be chosen in part to facilitate long term institutionalization. Thus, at the start of projects, active advocacy should occur to secure organizational collaborators who will share the work and responsibility for outcomes

Recommendations from Case Study Report: Children’s Bureau, Future Grantees, Public Agencies §Institutionalization e.g. The Children’s Bureau should aim to fund the type of training projects that are of critical need to public child welfare agencies. If the funding priorities of the Children’s Bureau are not aligned with the needs of the field, public child welfare agencies have no reason to engage in long-term institutionalization of training programs. After projects are completed, the Children’s Bureau is the only entity with appropriate infrastructure to keep the products that have been developed at the forefront of child welfare practice. §Knowledge Development e.g., Grantees should recognize that their projects are opportunities to develop learning about the field of child welfare training that can and should be shared with wider constituencies. Although “lessons learned” are typically requested as a part of project reporting, the content is generally thin and lessons are not shared. Grantees should think conceptually about the core lesson of their project and disseminate the contributions via conference presentations and journal articles. These should be less focused on promoting projects and more focused on linking project innovations to the wider field of child welfare training.

Findings from the Literature Review §Adult Learning Theory e.g., Organizational realities that are barriers to training based on these theories need to be confronted head-on. These barriers include time and space for sophisticated training; professional orientation toward workers that recognizes their expertise; and development of a learning culture in which organizational learning and development is the norm. Without addressing these barriers, training programs will have limited impact. §Training Implementation e.g., The field should move toward the development of training systems rather than training courses. Training systems infuse training with an organizational context and have greater potential for viewing training holistically, with connection to achievement of organizational outcomes. This will add to better conceptualization of the purposes of training and the anticipated linkage of training content and training activities with expected outcomes

Findings from the Literature Review §Agency/University Partnerships e.g., As with other forms of training, better evaluation is needed to assess the results of partnership efforts. Most of the literature emphasizes the benefits of partnerships, but there are limitations. Sometimes these are discussed as logistical problems or the melding of academic and practice cultures. Partnerships may limit the voice of schools of social work to critique child welfare practice and policy, and may cause schools to shift toward vocational education rather than professional education. §Evaluation of Training e.g., The methodology for evaluating outcomes related to knowledge and attitude change are fairly well developed and easily implemented. The measurement of skill development is more complicated and requires more methodological sophistication. Measures such as case plans and action plans should be more fully developed for use in training evaluation. Also observational methods and measures should receive attention so that the field can begin to measure the interaction of worker and client.

Survey of State and County Child Welfare Training Administrators -- Content §Main challenges agency faces in providing effective CWT §Experiences with federally-funded CWT l Description l Strengths/Weaknesses l Impact §Involvement with IV-E partnerships l Description l Strengths/Weaknesses l Impact §State/county funded training programs

Survey of State and County Child Welfare Training Administrators -- Content §Administrative supports for training §Impact of CFSRs on training §Existing mechanisms to institutionalize training §Relevant historical factors that have influenced CWT §Future initiatives in CWT §Training evaluation §What could federal government do to help states and counties deliver effective CWT? §What should be federal priorities (topics) of CWT in the next few years?

Survey of State and County Child Welfare Training Administrators: Training Evaluation §Many (n=18) refer to “levels” of evaluation §Level 1/satisfaction, n= 8; “Would like to do more” §Level 2/pre-post change, n=13 with some moving toward more complex evaluation for some training. §Additional evaluation steps e.g., longitudinal follow-up, n=20 §Most developed training systems, n=4

Survey of State and County Child Welfare Training Administrators: Training Evaluation “The success of training is first measured when trainees take a test. … followed by a field based assessment piece, which is left up to various supervisors. The supervisor’s evaluation includes an evaluation form, document review and observation of workers in various case management activities. Also an online tracking system, which tells them who passes competencies and other information.”

Survey of State and County Child Welfare Training Administrators: Training Evaluation “Evaluation method varies based on the kind of program. Some Level 1 (happy sheets) that are standardized. Available electronically—when a course is offered, evaluation information is put in immediately, can access it. Use pre/post knowledge test for basic training and all specialty programs. Occasional attempts at level 3 & 4 for very special situations; it’s expensive and hard. Also have a system to do field evaluations – conduct an on- site field review, either scheduled randomly or a project manager requests it.”

Survey of State and County Child Welfare Training Administrators: Training Evaluation Utilization: Generic statement about reviewing results (n=9) Primarily to review courses and trainers (n=23) With information system to tract participation, link with retention, overall quality assurance (n=12)

Survey of State and County Child Welfare Training Administrators: Training Evaluation The information is compiled and examined quarterly. It is then utilized for the following: 1. to look at trainers; 2. are workers getting what they want? (i.e., if they need to change curriculum); 3. how is the environment? (are environmental changes needed). Future plans include a practice reform initiative with [the University] which will have its own formal evaluation and they are working towards developing a model that will include behavioral anchors for new workers.

Survey of Colleges/Universities: Relationship with Child Welfare Agency N=97 Your Relationship Your School’s Relationship Quality Amount Mutuality/Reciprocity Productivity Desire for Future Collaboration Scale: 1 (low) – 4 (high)

Survey of Colleges/Universities: Perceived Impact of Federally-Funded Projects (N=30) Moderate/Strong npercent Recruitment of new CW staff15 50% Development of current CW staff26 87% CW worker knowledge30100% CW worker attitude23 85% CW worker skills27 90% CW supervisor knowledge22 76% CW supervisor attitude16 59% CW supervisor skill18 69% Macro level CW policy/practice11 47%

Triangulation: Common Findings §Reluctance of state agency to fully support training and its evaluation §Potential and pitfalls of partnerships between agencies and universities §“Training” knowledge dominated by curriculum development and training delivery; expansion needed in evaluation, organizations, and systems §Training knowledge dispersed in variety of fields and ranges from highly micro (e.g., instrumentation) to macro (e.g., policy implementation)

Discussion Topics §Relative roles of faculty, state/county training agencies, and the federal government in advancing the state of training. §Enhancing the knowledge base of child welfare training. §Addressing conceptual and methodological challenges to conducting training. §Assessing what is known and unknown in child welfare training. §Challenges of linking training outcomes to performance outcomes.

For more information, contact: Mary Elizabeth Collins, Ph.D. Boston University School of Social Work 264 Bay State Road Boston, MA