Preliminary Examination of Data Christopher L. Watson, Ph.D., Karen J. Storm, Ph.D. and Ann E. Bailey, M.A. Building Infrastructure Capacity for Reflective Practice: An Evaluation Study of Minnesota’s Maternal, Infant, Early Childhood Home Visiting Expansion Project Overview and Background Significance The Maternal, Infant, and Early Childhood Home Visiting (MIECHV) program of the Health Resources and Services Administration (HRSA) of the U.S. Department of Health and Human Services facilitates collaboration and partnership at the federal, state, and community levels to improve health and development outcomes for at-risk children through evidence based home visiting programs. The Minnesota Department of Health (MDH) received funds from HRSA to expand MIECHV using two home visiting models, Healthy Families America and Nurse Family Partnership. These models incorporate reflective practice for professional development. Reflective practice (RP) is a form of ongoing professional development that takes place either one-to-one or in a group and is usually led by a supervisor within the agency or an outside consultant (Mann, Boss, & Randolph, 2007; Wightman, et al., 2007). The supervisor creates a safe space for staff to reflect on and learn from their own work through an ongoing, collaborative process with a trusted mentor or supervisor (Heffron & Murch, 2010). Minnesota’s project employs a tiered system of mentoring. MDH staff members and infant mental health consultants mentor county program supervisors at 19 sites across the state. Supervisors, in turn, mentor and support home visitors in building relationships with at-risk families to improve the learning and development of infants and toddlers. Minnesota’s MIECHV Project seeks to increase the capacity statewide for RP consultation and mentoring to support and sustain the work of home visitors with families at risk. Literature regarding RP exists mainly in the clinical mental health field and the multidisciplinary field of infant mental health; it is theoretical and consists largely of anecdotal evidence and case studies. There is virtually no body of qualitative or quantitative research that addresses RP. Reflective practice is theorized to foster supervisor and practitioner self-awareness, reflective functioning, perspective taking, and problem solving—and resulting in improved self-regulation, increased job satisfaction, and reduced burnout. Our evaluation will address the effectiveness of the capacity-building efforts and the self- reported impact of RP on supervisors and home visitors and their work. Goals of the Evaluation The study contains both process and impact evaluations. The goals of the process evaluation are to: 1) study the strength of relationships between factors that both influence implementation of RP and are affected by adoption of RP; and 2) understand perceptions about the intervention; and inform the MDH in order to improve the mentoring system. The goal of the impact evaluation is to describe the growth in RP as a result of the intervention. Discussion Points and Next Steps Design and Selected Research Questions Selected Research Questions: 2a. Do local home visiting supervisors who participate in RP sessions with state mentors, gain new knowledge and skills in RP? 2b. Do local home visitors who participate in RP sessions with state mentors, gain new knowledge and skills in RP? 3b. Do local home visiting supervisors report positive change in their beliefs and attitudes about RP? 3c. Do local home visitors report positive change in their beliefs and attitudes about RP? 5a. Do home visiting supervisors report less burnout and increased competence and successful achievement in their work? 5b. Do home visitors report less burnout and increased competence and successful achievement in their work? Design The process evaluation is a mixed methods design that includes interviews, three measures, and a CEED developed survey. The impact study is an interrupted time series with 3 measures given 4 times prior to the intervention and 7 times after the intervention. Surveys The CEED Survey for supervisors and home visitors collects demographics and perceptions of experiences with reflective practice. Semi-structured Interviews Supervisors, infant mental health consultants, and home visitors participate in semi-structured interviews with evaluators to explore their experiences with reflective practice. Sample Nineteen (19) sites across the state received MIECHV funding. Sites were grouped by experience with RP. Group 1 consists of the 8 sites new to RP, and Group 2 consists of the 11 sites with experience in RP. The impact study examines growth in RP, thus only Group 1 is included in that study. All 19 sites are participating in the process study. Supervisors Thirty-three supervisors responded to the measures. They ranged in age from 25-64, with 3 over age 65. Twenty have bachelor of arts degrees and 13 have a post-graduate degrees. Twenty-seven are trained as nurses, 5 as social workers, and one as a psychologist. Nineteen are in the metro area; 6 in suburban areas, 2 in small cities, and 6 in rural areas. Experience ranged from less than a year as a supervisor (n=2) to 19 with more than 10 years experience. All but 2 were home visitors before becoming supervisors. Home Visitors A total of 97 home visitors responded to the measures. They ranged in age from 18-64, with a majority between category. Seventy-two have bachelor’s degrees, and 13 have post-graduate degrees. Eighty-one are trained as nurses, 6 are social workers, and 9 have “other” experience. Twenty-seven percent (27%) have less than one year’s experience as a HV, 25% have between 5-10 years experience, and 26% have more than 10 years experience. Measures and Baseline Findings Kentucky Inventory of Mindfulness Skills (KIMS) (Baer, Smith, & Allen, 2004): A self-report inventory for the assessment of mindfulness skills including: observing, describing, acting with awareness, and accepting without judgment. Group 1, with little to no experience in RP, inexplicably scored higher in mindfulness. Home visitors as a group scored lower than supervisors. Maslach Burnout Inventory (MBI) (Maslach, Jackson & Leiter, 1996): A 22-item survey that measures burnout versus sense of accomplishment in one’s work, with three scales: emotional exhaustion, depersonalization, personal accomplishment. As implementation of MIECHV began, participants had moderate scores on burnout and high scores in personal accomplishment. Working Alliance Inventory-Short Revised (WAI-SR) (Munder, Wilmers, Leonhart, Linster, et al., 2010): The Working Alliance Inventory-Short Revised (WAI-SR) assesses three aspects of the alliance between (in this study) home visitors and their supervisor, goals, tasks, and bond. Home visitors scored higher on the working alliance relationship than supervisors. Baseline Perceptions of Reflective Practice The case presented here is an evaluation study made possible by the Affordable Care Act of 2010 as the creator of the Maternal, Infant, Early Childhood Home Visiting Program (MIECHV), with funding and administration from the US Department of Health and Human Services: Health Resources and Services Administration (USSDHHS/HRSA) and is being conducted in partnership with the Minnesota Department of Health (MDH). MIECHV funding for home visiting programs for at-risk families Minnesota Department of Health (MDH) mentors and infant Mental Health Consultants (IMHCs) to support local programs Evaluation staff at the Center for Early Education & Development (CEED) at the University of Minnesota Semi-structured interviews with state mentors, local program directors, and supervisors explore whether state support is adequate to implement and sustain RP Kentucky Mindfulness Survey (KIMS) measures the degree to which supervisors and home visitors perceive themselves as reflective Maslach Burnout Inventory (MBI) records the level of burnout and perceptions of accomplishments at work reported by supervisors and home visitors InputsActivities Outcomes MDH establishes what is needed to provide adequate support for implementation of reflective practice at the local program level Working Alliance Inventory (WAI-SR) measures the degree to which supervisors and home visitors perceive the existence of good working relationships for implementing RP Surveys ask home visitors about the support they are receiving to implement RP Leadership Self Assessment explores local program directors leadership style and the extent to which they apply a reflective leadership approach in their work Evaluation MDH staff provide an initial training in reflective practice (RP) and infant mental health for local program directors, supervisors and home visitors to establish a foundational understanding of reflective practice IMHCs provide ongoing support by conducting regular one-to-one RP sessions with supervisors for 18 months Supervisors provide regular RP sessions for the home visitors in their programs IMHCs partner with supervisors in co- facilitating several group RP sessions for home visitors in their programs Local program directors develop a reflective leadership style Supervisors and home visitors gain knowledge and skills in reflective practice, including mindfulness and therapeutic alliance, and receive ongoing support in their use of reflective practice Families with risk factors experience more effective home visiting services resulting in improved benchmark outcomes Implementation of reflective practice is successfully expanded and institutionalization in local home visiting programs throughout the state Supervisors and home visitors experience a reduction in emotional exhaustion and depersonalization and an increase in their sense of accomplishment. Reflective Practice Mentoring — Minnesota MIECHV Evaluation Logic Model Supervisor (n = 33)Home Visitors (121) Improvement of Communication 93.9% (31/33)74.4% (90/121) Helpfulness of Individual RP 93.9% (31/33)80.2% (97/121) Helpfulness of Case Conference 90.9% (30/33)79.2% (95/121) A majority of both supervisors and home visitors reported that, because of reflective practice, there were improved levels of communication with each other, and that both the one-to-one reflective practice and the group RP sessions were helpful to their work. Supervisors and home visitors were also asked to rate their own level of implementation with reflective practice. Supervisors evenly reported that their RP implementation practices are either emerging or fully implemented when supervising home visitors. Home visitors more often rated their RP implementation practices as “emerging” with families and within the case conference, but were equally distributed between “emerging” and “fully implemented” when engaging in RP with their supervisor. Home Visitors (n = 122) How would you rate your level of implementation of reflective practice principles? Not Yet in PracticeEmergingFully Implemented In my case conference group27046 With my supervisor individually06059 In my work with families28335 Supervisors (n = 33) Not ApplicableEmergingFully Implemented In my case conference group11417 With the home visitors individually51114 In my work with families* Baseline data demonstrate that self-reported rates of RP implementation across the state are variable. Both home visitors and supervisors report improved levels of communication with each other because of RP implementation. They also report that individual RP sessions and case conferences are helpful to their overall work. Home visitors’ and supervisors’ baseline scores on the MBI were similar across the 3 sub-scales (i.e., emotional exhaustion, depersonalization, and personal accomplishment). The home visitors scored higher on the WAI, reporting higher levels of agreement on tasks, goals, and bond development than those reported by supervisors. Conversely, supervisors scored higher on the KIMS, suggesting more experience with and greater use of mindfulness. Discussion Next Steps Currently, we are conducing qualitative analysis of interviews conducted with supervisors, administrators, Infant Mental Health Consultants, and MDH Mentors. A sample of home visitors will be interviewed closer to the end of the evaluation study. There will be 5 more administrations of the measures (i.e., KIMS, WAI, MBI: SR) for the home visitors participating in the impact evaluation. There will also be one more administration of the CEED survey to both supervisors and home visitors to examine any changes in RP implementation and knowledge over time. *Some supervisors also continue to work with families.