UC San Diego, Department of Family Medicine and Public Health BHICCI Health Care Organizations’ Learning: Review of Initial BHICCI Learning Evaluation Report Todd Gilmer PhD UC San Diego, Department of Family Medicine and Public Health

Evaluation Team Todd Gilmer, PhD – Evaluation Project Director Ben Henwood, PhD – Senior Associate Kimberly Center, MA – Evaluation Program Manager Amy Panczakiewicz, MA – Program Evaluation Specialist Lucyna Klinicka, BA – Program Evaluation Specialist

BHICCI Evaluation A primary goal of the BHICCI evaluation is learning. The term “evaluation” may conjure many different words in your mind. The evaluation design and methodology that we will use is tailored to meet the needs of IEHP and the BHICCI in consideration of the unique nature of the initiative. The evaluation is not meant to make comparisons between organizations but to document each organizations progress to becoming health homes to provide recommendations to IEHP for further spreading transformation changes to other organizations. A primary goal of the BHICCI evaluation is learning.

Evaluation Methods Formative Components Summative Components Allow for feedback to be incorporated during program implementation BHICCI Roadmap Checklist BHICCI Site Self-Assessment Qualitative site visits Summative Components End of the program cycle Overall description of program effectiveness PROMIS Global Health Registry and measurement-based practices Utilization and cost Cost-effectiveness Evaluation methodology includes both formative and summative components. Formative evaluations allow for feedback to be incorporated during the implementation of a program. Summative evaluations occur at the end of a program cycle and provide an overall description of program effectiveness. NOTE: We need to be careful to use the term measurement-based practices instead of T2T. The next cycle of reports will include summative data components.

BHICCI Roadmap Progress Foundation for Improvement Developing Complex Care Systems Sustaining Changes Practice changes are conceptualized as steps within three phases for practice transformation: Foundation for Improvement, Developing Complex Care Systems/Whole Health Homes and Sustaining Changes. While the goals and practice changes described in the Roadmap are not strictly sequential, leadership who supports an organizational culture that is committed to quality and providing whole person health care, which includes attention to employee engagement and wellness and using tools for quality improvement, provides the foundation that is necessary to sustain enhancements to the service system.

Roadmap Progress: Phase 1 Engaging Leadership Engaging Employees Adopting QI Methodology Use Data and Measurement to Guide Practice Changes Team Reports submitted in April 2016, responses to the baseline BHICCI Site Self-Assessment and qualitative interviews with the practice coaches conducted in June 2016 were used so evaluation team members could coded progress for each BHICCI practice change goal listed on the Roadmap as “not started or not included in report,” “implementation phases,” or “practice change made.” Coding within the implementation phase includes any key activities that were described as being planned or tested. Engaging Leadership: The majority of organizations (83%) have started implementing or have made at least one practice change. There is the most variability in organizations’ progress promoting an organizational culture that creates incentives for workforce engagement in quality, learning and development, and promotes pride and joy in work, and developing collaborative relationships with other systems. Only 25% of organizations are implementing this change at this point. Engaging Employees: Half of the organizations have started implementing all three employee engagement practice change goals. As described in the team reports, hiring and recruitment has been a challenge for many organizations. Despite these challenges, 83% have taken steps towards the development of a specific department or leader that measures employee experience and engagement as a core focus. Adopting QI Methodology: All of the organizations participating in the BHICCI started implementing at least one of the five quality improvement methodology practice change goals. Two-thirds of the organizations have started to use health information technology, and have made progress establishing and monitoring client/patient, population, and system-level metrics. Use of data and measurement: 25% of organizations have started to implement all four data and measurement practice change goals. The majority of organizations (83%) have taken steps to train and coach staff at all levels to use both system and clinical-level data to track and improve results over time.

Roadmap Progress: Phase 2 Identify Target Populations Build Multidisciplinary Complex Care Team Offer Integrated, Complex Care Management Engage Individuals and Families in Whole Health Identify target population: 92% of organizations have started to identify their target population, select individuals for testing complex care management and established practices to ensure that ROI are signed Build multidisciplinary complex care team: Every organization participating in the BHICCI has initiated implementation of at least two of the six practice change goals necessary to build the multidisciplinary, complex care team. 41% of organizations have initiated daily team communications, and half have increased knowledge of substance use and addictive disorders and intervention capacity within their multidisciplinary team. Offer integrated, complex care management: Half of the organizations have started the process of measuring and tracking screening, treatment, and wellness outcomes in a registry. 58% of organizations have started coordinating the care of clients/patients with complex conditions. Engage individuals and families in whole health: Two-thirds of organizations have started the implementation phase of linking patients and/or families to needed social services, housing, and community resources.

Roadmap Progress: Phase 3 Sustaining Practice Changes Most organizations participating in BHICCI have not started to implement any of the practice changes related to sustainability. However, it is early in the implementation process, and not really anticiatied that many organizations would have made progress sustaining changes at this point.

Roadmap Progress Summary Overall, each organization has started implementing at least some practice change goals At some organizations, some practice change goals were implemented prior to participation It is anticipated that organizations will move from “not started” and “implementation phase” to “practice change made” on these goals over time Overall, all of the organizations participating in the BHICCI have started the implementation phase of at least some of the practice change goals. Many of the organizations had implemented some of the elements of the Roadmap prior to participation in the BHICCI. BHICCI is supporting the pathway that organizations were already on. As the initiative continues, it is anticipated that organizations will move from the “not started” and “implementation phase” on these goals to “practice change made.”

BHICCI Site Self Assessment (SSA) Adapted from the Maine Health Access Foundation (MeHAF) SSA BHICCI SSA has 21 characteristics 11 related to services (more directly impacting clients/patients) 10 related to organizations (more directly impacting health care team members) Health care teams rate each characteristic using a 1-10 scale. 11 characteristics combined to create person-centered, integrated care domain 10 characteristics combined to create practice/organizational integration domain

Baseline Site Self Assessment Results Person-Centered Integrated Care Domain Most common average score was 5.09. Interpretation of average rating Practice/Organizational Integration Domain Most common average score was 4.10. A rating score of “5” suggests that behavioral health and physical health providers have started to work in consultation and are conducting screenings and measurement regularly, but clients/patients and family are only sometimes included in conversations about screening results and treatment planning. Average scores for the person-centered domain ranged from 3.45 to 7.91. A rating score of “4” suggests that engagement by the executive leadership and providers’ is limited, coordination of care is not always assured and data systems and registries are not consistently integrated. Average scores for the practice/organization domain ranged from 2.70 to 8.30.

Baseline Site Self Assessment Results Highlights There was high variability in responses for several items. Organization’s ratings on these items were distributed between almost every level from “1” to “10” Greatest variability for following items: - Individual/family involvement in shared care planning - assessment of clients/patients’ strengths, preferences and social determinants of health - services are provided by multidisciplinary team

Baseline Site Self Assessment Results Highlights Half (50%) of organizations are engaging in some aspects of: Average domain scores were more spread out between levels. Results suggest organizations may be implementing different components of BHICCI, dependent on their current capacity for change. Complex care management Integrated care Population health management Team-based care Measurement of health care team experience Integration of leadership

Baseline Site Self Assessment Summary Different Stages of Change Practice Change Goal Status Results and Future Reporting as evident by high ratings at baseline - Results show that organizations are currently at very different stages of BHICCI planning and implementation. - Some organizations have integrated several BHICCI components into practice, but most organizations are currently working towards integration and implementation in each BHICCI domain. - These results are promising, and future reporting will focus on changes in BHICCI domains within the first year of the initiative.

Fact Sheets It was important to the evaluation team to shared the results from the baseline SSA back with organizations to promote a learning culture. Baseline BHICCI Site Self Assessment Results are described in the Baseline Integration report, and summarized for each organization in their Fact Sheet. The 1st Fact Sheet includes a brief description of the organization, a summary of the Physical Health and Behavioral Health diagnoses included in the ACG report (only if IEHP has authorization to access client/patient information) and information from the Site Self Assessment. The Fact Sheets will be distributed to each organization tomorrow morning. Please feel free to contact evaluation team with any questions.

Documenting the BHICCI Learning Throughout the Process Ben Henwood, PhD University of Southern California (USC) School of Social Work

Methods to Document Learning Documentation of learning through qualitative interviews during site visits Key stakeholders Health care teams Documentation of learning collaborative and team meetings Capture challenges Identify best practice solutions A major component of the evaluation will be the documentation of learning from the implementation of the BHICCI. This will be achieved through qualitative interviews with key stakeholders’ and health care teams during site visits. Learning collaborative and team meetings will also be documented to capture challenges and best practice solutions encountered during the course of implementation at the organizational level. 

Learning from Qualitative Interviews Interviews with key stakeholders will help identify emerging themes and challenges. Site visits and interviews with health care team will promote learning about transformation process and best practices.

Early Lessons Learned Importance of Engaging Executive Leadership in BHICCI Implementation of Cross-System Leadership skill building workshops Intentional practice of sharing and communication

Moving Forward July 2016 January 2017 November 2016 August 2016 PROMIS Global Health pilot testing August 2016 Site visits to 3 or 4 organizations November 2016 Second administration of BHICCI SSA January 2017 Additional data in next set of reports Pilot testing of the PROMIS Global Health will begin in July 2016 First round of site visits at 3-4 organizations will occur in August 2016 Second administration of BHICCI SSA will occur in November 2016. November assessment period will be baseline for RUHS sites. Next set of reports (January 2017) will include data from the PROMIS Global Health and Measurement- based practices, describe SSA progress and learning from site visits.