Implementation, Evaluation and Getting to the Triple Aim Deborah J. Cohen. PhD Oregon Health & Science University Russell E. Glasgow, PhD University of Colorado School of Medicine

Overview Implementation and Implementation Science Evaluation, Learning and the Triple Aim Population Health Patient Experience Take Home Points

Traditional RCTs study the effectiveness of treatments delivered to carefully selected populations under ideal conditions. This makes it difficult to translate results to the real world. Even when we do implement a tested intervention into everyday clinical practice, we often see a “voltage drop”—a dramatic decrease in effectiveness. “If we want more evidence-based practice, we need more practice-based evidence.” Green LW. Am J Pub Health 2006 Rothwell PM. External validity of randomised controlled trials…Lancet 2005;365: Implementation Science Pragmatic Challenge: Much Research is Not Relevant to Practice

Implementation Science, Multi-level Research-Practice Contextual Systems Approach Adapted from Estabrooks P. et. al. AJPM, 2005, 31: S45 Research Design Team And Adaptive Design Organization Program Delivery Staff Delivery Site(s) Fit Design Appropriate for Question Partnership Critical Elements Program as Tested Evidence-Tested Program Non-critical Packaging Program as Marketed

Models – theories and frameworks What can they do: ◦ Make the spread and uptake of interventions more likely ◦ Provide systematic structure for the development, management, and evaluation of interventions/D&I efforts Wealth of existing models for D&I ◦ 61 identified by Tabak et al in a recent review ◦ Additional models with practitioner focus Tabak RG et al, Bridging Research and Practice: Models for Dissemination and Implementation Research Am J Prev Med, 2012, 43: ; Models for Implementation Science

RE-AIM To Help Plan, Evaluate and Report Studies 6 Glasgow, Klesges, Dzweltowksi, et al. Ann Behav Med 2004;27(1):3-12 R Increase R each E Increase E ffectiveness A Increase A doption I Increase I mplementation M Increase M aintenance

Precision (Personalized) Medicine Questions in IS Determine: What percent and types of patients are Reached; For whom among them is the intervention Effective; in improving what outcomes; with what unanticipated consequences; In what percent and types of settings and staff is this approach Adopted; How consistently are different parts of it Implemented at what cost to different parties; And how well are the intervention components and their effects Maintained? Pawson R, et al. J Health Serv Res Policy 2005;10(S1):S21-S39. Gaglio B, Glasgow RE. Evaluation approaches…In: Brownson R, Colditz G, Proctor E, (Eds). Dissemination and implementation research in health: Translating science to practice. New York: Oxford University Press; Pages

8 Broaden the criteria used to evaluate programs to include external validity Evaluate issues relevant to program adoption, implementation, and sustainability Help close the gap between research studies and practice by: Suggesting standard reporting criteria Informing design and evaluation of interventions Focus on contextual factors that may impact results RE-AIM Focus on Context

Advancing Care Together (ACT) – An example of use of RE-AIM in an integration program Program funded by The Colorado Health Foundation 11 practices funded to do demonstration projects  9 primary care practices  2 community mental health centers ◦ All focused on integrating behavioral health and primary care ◦ We had the privilege of following their journey; RE-AIM informed our work

Better Outcomes Systematic reviews and other rigorous, peer- reviewed studies show that integrated care leads to better patient outcomes 1-5 for: Depression Panic Disorder Tobacco cessation Alcohol Misuse Diabetes IBS GAD Chronic Pain Primary Insomnia Somatic Complaints 1. Butler et al., AHRQ Publication No. 09- E003. Rockville, MD. AHRQ Craven et al., Canadian Journal of Psychiatry. 2006;51:1S-72S. 3. Gilbody et al., British Journal of Psychiatry, 2006;189: Williams et al., General Hospital Psychiatry, 2007; 29: Hunter et al., Integrated Behavioral Health in Primary Care: American Psychological Association, 2009

11 Improved Patient and Provider Experience With a shift to integrated delivery models, patient experience with healthcare delivery improves 1-5 With a shift to integrated delivery models, primary care provider experience improves too 6,7 1.Chen et al., American Journal of Geriatric Psychiatry. 2006; 14: Unutzer et al., JAMA. 2002; 288: Katon et al., JAMA. 1995; 273: Katon et al., Archives of General Psychiatry. 1999; 56: Katon et al., Archives of General Psychiatry. 1996; 53: Gallo et al., Annals of Family Medicine. 2004; 2: Levine et al., General Hospital Psychiatry. 2005; 27:

Multifaceted Diabetes and Depression Program– medical savings of $39 PMPM observed over 18 months Pathways program for diabetes & depression - $46 PMPM saved, or about 5% over 2 years IMPACT program for depression among the elderly - $70 PMPM saved over 4-year period, or about 10% Missouri CMHC health homes in 2012 – independent living increased by 33%, vocational activity increased by 44%, overall healthcare costs decreased by 8% Observed savings of between 9% and 16% of value opportunity Lower Cost When Treated Melek, SP, Norris, DT, Paulus, J. Economic impact of integrated medical-behavioral healthcare: Implications for psychiatry. Prepared by Milliman. for the American Psychological Association, April 2014

Findings from an Analysis of Comorbid Chronic Medical & Behavioral Conditions in Insured Populations Melek, SP, Norris, DT, Paulus, J. Economic impact of integrated medical-behavioral healthcare: Implications for psychiatry. Prepared by Milliman. for the American Psychological Association, April 2014

Learning Evaluation – A RE-AIM Informed Evaluation Approach RE-AIM provides a framework for collecting relevant data when implementing an innovation Implementation of a new innovation occurs through rapid, short cycles of improvement Data are essential to ◦ Monitoring and refining the change process ◦ Ensuring that implemented changes result in expected outcomes ◦ Taking the innovation to to scale

What Data Do We Collect Practice Characteristics ◦ Practice Information Form Implementation / Context ◦ Documentation ◦ Online diaries ◦ Phone calls ◦ Site visits Reach ◦ % of target screened ◦ % of target positive ◦ % of target received services Effectiveness ◦ Patient outcomes ◦ Expenditures ◦ Utilization

What We Do With the Data We Collect

DimensionDefinitions REACH (Individual Level) 1.Participation rate among eligible individuals 2.Representativeness of participants RE-AIM Dimensions and Definitions for Population Health ADOPTION (Setting Level) 1.Participation rate among invited settings and staff 2.Representativeness of participating settings and staff

Key Lessons Learned about Reach and Adoption Both focus on importance of denominators Frequently confused: Same principles at different levels If exact denominator is unknown, estimate!

Types of Interventions Provided to Patients Brief counseling in primary care Referrals for traditional long-term counseling ◦ Within clinic ◦ From partner clinic ◦ Outside clinic Warm hand-off Joint PC and BH counseling

Screening Tools Used by ACT Innovators

Reach Over a 12 month period: ◦ 84,645 target patients ◦ 13,168 were screened ◦ 6,845 screened positive Over a three month period, on average: ◦ 21,149 target patients ◦ 3,292 were screened ◦ 1,711 screened positive 15.5% screened 52% of those screened, screened positive The ACT program included:

Screening Systematic Screening Clinical Discretion

Two stories Documenting referral or receipt of further counseling for patients who screened positive Mental model for how data is used in quality improvement process

DimensionDefinitions Effectiveness (Individual Level) 1.Effects on primary outcome of interest 2.Impact on quality of life, any negative outcomes RE-AIM Dimensions and Definitions for Patient Experience Maintenance (Setting Level) 1.Long-term effects of intervention 2.Sustained delivery and modification of intervention (setting level)

Key Lessons Learned about Effectiveness and Maintenance Are often unintended consequences of programs, can be either positive or negative Some of the programs that have the greatest effectiveness have the lowest reach (and vice versa) Setting level Maintenance is seldom sustained, but almost never is a program continued in the exact same way

Effectiveness – A Patient’s Perspective Patrick’s health story Sandy’s health story

Take Home Points “All models are wrong” Importance of ongoing evaluation, monitoring quality, learning from a range of data Interrelationship of RE-AIM dimensions Value of RE-AIM when implementing integration innovations

Questions?