Implementation Science, Context, and Health Equity: RE-AIM Applications and Ideas for Feedback Russell E. Glasgow, PhD Department of Family Medicine, University.

Implementation Science, Context, and Health Equity: RE-AIM Applications and Ideas for Feedback
Russell E. Glasgow, PhD Department of Family Medicine, University of Colorado School of Medicine VA Eastern Colorado QUERI and Geriatric Research Centers, and Adaptation, Fidelity and Tailoring Interest Group Dissemination and Implementation Science Program of Adult and Child Consortium for Outcomes Research and Delivery Science

Acknowledgments and Conflicts of Interest
Amy Huebschmann, Borsika Rabin University of Colorado SOM - ACCORDS D&I Science Program RE-AIM Colleagues FINANCIAL DISCLOSURE National Institutes of Health (NIH), Agency for Healthcare Research and Quality (AHRQ), and Robert Wood Johnson Foundation (RWJF) funding on various projects UNLABELED/UNAPPROVED USES DISCLOSURE None

Health Equity and How Pragmatic RE-AIM Research Might Help
Health Equity and How Pragmatic RE-AIM Research Might Help....need your feedback Considering context - PRISM extension of RE-AIM Planning programs - estimating RE-AIM impacts Adaptations - types and implications Representativeness and transparent reporting Be Fit Be Well example Your feedback and opportunities for D&I research

Please use Top Left one- photo of older coal cars

Using Systematic Reviews to Select Evidence-Based Programs that Are Pragmatic (using PRECIS-2 and RE-AIM) Level of Pragmatism Low Medium High Level of Effectiveness Low Medium High Pragmatic Traditional Efficacy Traditional Effective Pragmatic and Effective Luoma K, et al. (2017). Translational Behav Med, 7: 751

Pragmatic Models- RE-AIM

Other Models Many commonalities across models and theories
Consolidated Framework for Implementation Research Diffusion of Innovations Many commonalities across models and theories Tabek RG, et al. Bridging research and practice...Amer J Prev Med (2012); 43: Over ?? D&I Frameworks:

“D&I theories are kind of like toothbrushes: Everybody has one and no one wants to use somebody else’s” Cara Lewis via Anne Sales via ????

Purpose and History of RE-AIM Framework
Intended to facilitate translation of research to practice Balance internal and external validity, and emphasizes representativeness Individual (RE) and multi-level organizational (AIM) factors Public health impact depends on all elements (reach x effectiveness, etc.)

Pragmatic Use of RE-AIM- What is Feasible?
RE-AIM Dimension Key Pragmatic Priorities to Consider and Answer Reach WHO is (was) intended to benefit and who actually participates or is exposed to the intervention? Effectiveness WHAT is (was) the most important benefit you are trying to achieve and what is (was) the likelihood of negative outcomes? Adoption WHERE is (was) the program or policy applied WHO applied it? Implementation HOW consistently is (was) the program or policy delivered? HOW will (was) it be adapted? HOW much will (did) it cost? WHY will (did) the results come about? Maintenance WHEN will (was) the initiative become operational; how long will (was) it be sustained (setting level); and how long are the results sustained (individual level)? Glasgow R and Estabrooks P. Preventing Chronic Disease (2018) 15, E02.

RE-AIM—Health Equity Implications
RE-AIM Issue Disparity Overall Impact Reach 30% 70% benefit Effectiveness 0 (equal) Adoption 49% benefit Implementation 34% benefit Maintenance 24% benefit This slide uses hypothetical data to illustrate the importance of broad thinking about outcomes. The importance of focusing on all five dimensions of RE-AIM – effectiveness is not the only important factor, especially in disparities. Note if one only focused on effectiveness, one would likely conclude there were no health disparities associated with this program or policy

RE-AIM Summary Points RE-AIM is not a theory, but it tells you where to look; where things often break down RE-AIM is an outcomes framework that can be used for planning and evaluation....and (maybe?) iteratively Each dimension is an opportunity for intervention All dimensions can be addressed within a given study (though likely not all intervened upon) RE-AIM can be used for observational, efficacy, effectiveness, and implementation science projects

Evolution of RE-AIM Applicability to many different content areas…more than 420 articles Used for both planning and evaluation Exploring iterative use for adaptations Underreporting of key components Setting level factors reported much less often (e.g., adoption) Increasing use of qualitative measures* Gaglio, et al. The RE-AIM framework….AJPH 2013; 103: Holtrop, et al. BMC Health Serv Res Mar 13;18(1):177. doi: 10.

Crosscutting issues Proportion who benefit Representativeness of who benefits Reasons: how and why they benefit Adaptations made Costs incurred

Roadmap for Whirlwind RE-AIM & Equity Journey
Considering Context - PRISM extension of RE-AIM Planning programs- Estimating RE-AIM impact Adaptations - types and implications Representativeness and transparent reporting Be Fit Be Well example Your feedback and opportunities for D&I research

PRISM = Pragmatic Robust Implementation and Sustainability Model
Feldstein & Glasgow (2008). Joint Commission J on Qual. & Patient Safety, 34:

Context Issues from RE-AIM Perspective
People exist in the context of culture and places in which they work, live, study and play (Take home point #1) Context is multi-level and dynamic Challenges of studying and understanding context: So many factors…which are most important for which issues, for which settings, for which populations? - Need for brief, validated and quantitative pragmatic measures Chambers et al. DSF (2013) Imp Sci 8: Bayliss et al. Understanding context. Ann Fam Med (2014) 12: 3: 260-9

Practical, Robust Implementation & Sustainability Model
Addresses multi-level contextual factors impacting RE-AIM outcomes Feldstein and Glasgow, The Joint Commission Journal on Quality and Patient Safety 2008;34(4):

Key Aspects for Context in PRISM:
Practical Robust Implementation & Sustainability Model External Context- Multi-level and dynamic Polices/guidelines Incentives/reimbursement Resources History (and trust) Internal Context- consider fit Multi-level organization characteristics and perspectives Multi-level citizen/patient/end-user characteristics & perspectives Implementation and sustainability infrastructure

Roadmap for Whirlwind RE-AIM & Equity Journey
Considering Context - PRISM extension of RE-AIM Planning programs- Estimating RE-AIM impact Adaptations - types and implications Representativeness and transparent reporting Be Fit Be Well example Your feedback and opportunities for D&I research

Applying RE-AIM to Planning Interventions

Planning and ‘Evaluability’*
Do initial estimates of RE-AIM dimensions where you don’t have data (evaluability)* - with stakeholders Include multiple perspectives on an ongoing basis Expect different programs or interventions to do well on different RE–AIM dimensions Often helpful to compare two or more program or policy options (create RE-AIM ‘profiles’) Similar to P.S. Hovmand. Community-based system dynamics (2014) *Leviton L, et al. Annual Review Public Health, 2010, 31,

EXAMPLE OF USING RE-AIM to PLAN for EQUITY: My Own Health Report (MOHR) Program
Cluster randomized pragmatic trial of web-based, brief health behavior and mental health risk assessment and feedback intervention in nine diverse, low-resourced pairs of safety net primary care practices RE-AIM Outcomes included: Reach of the MOHR program across patients* Effectiveness of the MOHR program on behavior change goal setting Whether safety net practices would Adopt MOHR* How practices would Implement and adapt MOHR Little time, few costs and burden to deliver MOHR- MINC concept *Primary goals Glasgow, et al. (2014). Conducting rapid, relevant, research. Am J Prev Med, August;47(2):212-9. Krist, et al. (2016). The impact of behavioral and mental health risk assessments. Transl Beh Med Jun; 6(2):212-9.

Example of Planning for Equity and Context using RE-AIM in the MOHR Study: Designing for generalization Reach*: Options for clinics and patients in contact modalities (phone, , EHR, waiting room); in context of regular visits- remove barriers Effectiveness: Shared Decision Making on goals, including social determinants of health (most patients had 4 or more behavioral risk factors) Adoption*: Fit with reimbursement for annual wellness exams; CME; few resources required, largely automated intervention, English and Spanish Implementation: Options of where and when the BRIEF online session, real time feedback, and related counseling/goal setting occur and who does this Maintenance: Not well done- assumed success and reimbursement would be sufficient; needed to be integrated into the EHR

How Pragmatic RE-AIM Research Might Help Address and Improve Health Equity
Considering Context- PRISM extension of RE-AIM Planning programs- estimating RE-AIM impacts Adaptations- types, implications and guidance Representativeness and transparent reporting Be Fit Be Well example Your Feedback and opportunities for D&I research *Leviton L, et al. Annual Review Public Health, 2010, 31,

Kaiser Permanente Research
Evaluating complex interventions: Confronting and guiding (vs. ignoring and suppressing) heterogeneity and adaptation October 9, 2018 Brian S. Mittman, PhD Department of Research and Evaluation, Kaiser Permanente Southern California Quality Enhancement Research Initiative (QUERI), U.S. Department of Veterans Affairs Clinical and Translational Science Institute, University of California at Los Angeles

Implementing EBP and Implementation Strategies
Complex interventions usually can be, will be and should be adapted* Adaptation should be: embraced, studied and guided, rather than ignored and/or suppressed Bauman, AA, Cabassa, LJ, et al. Adaptation in D&I science et al. In Brownson R et al. DIRH in health, 2019 Casbassa, LJ. & Bauman, AA. A two-way street: Bridging IS and cultural adaptations.... Imp Sci 2013: 8; 90

PCORI Methodology Guideline SCI-3: Specify how adaptations to the form of the intervention and comparator will be allowed and recorded Researchers should specify: allowable adaptations in form and/or function a description of how planned and unplanned adaptations will be managed, measured and reported over time Any planned adaptations should: have a clear rationale ideally be supported by theory, evidence or experience maintain fidelity to the core functions of the intervention Upon study conclusion, researchers should provide guidance on: allowable adaptations, or unproductive adaptations

Types of Adaptations- Cultural, Resources, Local
Focus of Adaptation Timing of Adaptation (point in the study) Planning During Dissemination Intervention Implementation Strategy Setting

Adapting the Stirman, et al
Adapting the Stirman, et al. Framework Using the RE-AIM Model and Clinical Experience RUSS Rabin B, et al. Frontiers Pub Health 2018, 6:102

Adaptation, Fidelity and Tailoring Interest Group
Began January 2016 as part of the IRG 61 members currently ....YOU ARE INVITED TO JOIN Representation from many VA QUERI research programs Co-chaired by Borsika Rabin, MPH, PhD, PharmD and Russell Glasgow, PhD; Facilitated by Christine P. Kowalski, MPH Meet monthly to discuss topics related to adaptation, tailoring and fidelity with attention to clinical application. Discussions include how to define interventions and implementation strategies, as well as how to describe and document adaptations. For information or to join, contact:

Considering context - PRISM extension of RE-AIM Planning programs - estimating RE-AIM impact Adaptations - types and implications Representativeness and transparent reporting Be Fit Be Well example Your feedback and opportunities for D&I research

Equity Issues in Evidence-Based Research: Evidence on What
Equity Issues in Evidence-Based Research: Evidence on What? (take home point #2) External Validity/Pragmatic Criteria, Often Ignored Participant representativeness Setting and staff representativeness Multi-level context Adaptation/change in intervention and implementation strategies What outcomes and over what time period Reasons for participation and drop out

Replicability (and Generalizability)
Important to report conditions under which the program was delivered To what extent is the program replicable: In similar settings? In different settings? Bottom Line and ULTIMATE USE QUESTION “What program/policy components are most effective for producing what outcomes for which populations/recipients when implemented by what type of persons using what implementation strategies under what conditions, with how many resources and how/why do these results come about?”

Transparent Adoption Reporting: Setting and Staff Issues
Setting Level Adoption Factor Settings What Sites are invited/excluded Participation rate Which participate How representative are they Reasons for participating/declining Staff Level Adoption Factors Who is invited/excluded Who participates

Expanded CONSORT Figure – Adoption
Total Number Potential Settings Settings Eligible (n and %) Settings and Agents Who Participate Settings and Agents Who Decline Other Excluded by Investigator (n, %, and reasons) Critical Considerations: - Characteristics of Adopting Settings vs. Non - Characteristics of Adopting Staff vs. Non Glasgow RE, Huebschmann A, Brownson RC. (2018) American J Preventive Med 55 (3), 422–430

“Original” CONSORT criteria are found here in the interior of the expanded CONSORT diagram.
Expanded CONSORT Figure (Access at

Expanded CONSORT Figure: With Mapping for RE-AIM Domains
Adoption Reach Maintenance Expanded CONSORT Figure (Access at

Why is it Important to Use the Expanded CONSORT
Why is it Important to Use the Expanded CONSORT? (or similar reporting approach*) To understand and estimate health equity impact (Take home point #3) To enhance transparency and indicate generalizability factors (Take home point #4) To increase the frequency and quality of external validity reporting To address the replication failure crisis To align with evolving D&I reporting criteria (StaRI*) *Pinnock H, et al. Standards for reporting...(StaRI). BMJ 2017: 356; i6795

THE FUTURE OF RE-AIM Application to comparative effectiveness research
(CER- T) and ITERATIVE use of RE-AIM Transparency focus (Expanded CONSORT figure*) What it means to “Use RE-AIM” Possible directions - Merge with PRECIS-2 model*? - Your ideas welcomed! *Glasgow RE, Huebschmann A, Brownson RC. (2019) Making health research Ann Review Public Health. Jan 21. Kessler RS, et al. What Does It Mean to ''Employ'' the RE-AIM Model? Eval Health Prof., 2012 Mar;36, 44-46

All Models (and Methods) Are Wrong… Some Are Useful
“To every complex question, there is a simple answer… and it is wrong.” ~H. L. Mencken

Be Fit Be Well: 24-month randomized weight loss and hypertension self-management intervention trial among low-income urban clinics. RE-AIM used to plan for and assess reduction in disparities, as well as report outcomes Let’s take a concrete example of how REAIM was used to help design and evaluate an intervention for low income African Americans in inner city Boston. These were patients in 3 community health centers. BFBW, was a 24-month randomized weight loss and hypertension self-management intervention trial with an option for the intervention participants to choose either web-based or IVR/print delivery. The 365 participants in BFBW were primary care patients at one of three predominantly low-income community health centers (CHC) in Boston, Massachusetts. Eligibility criteria included: measured body mass index (BMI) kg/m2, pharmacologic treatment for hypertension, age 21 or older, receipt of primary care of at one of the three participating CHCs and ability to speak and read either English or Spanish. Participants received a $50 gift card at their baseline, 6-, 12- and 18-month visits and a $75 gift card at their 24-month visit. Intervention participants received a scale at their 6-months and a blood pressure monitor at their 12-months to aid their behavioral self-monitoring. Participants with difficulty getting to clinic sites for assessment visits were also offered taxi vouchers. Had high reach, good engagement, and moderate results for hypertension and diabetes. Bennett, et al. Obesity treatment for socioeconomically disadvantaged patients in primary care practice. Arch Intern Med ;172(7):565-74

Baseline Characteristics of Be Fit Be Well Participants
Usual Care (n=185) Intervention (n=180) Female 122 (66%) 128 (71%) Non-Hispanic Black/African-American 131 (71%) 129 (71%) Hispanic 23 (12%) 25 (14%) Language n(%) Spanish 22 (12%) 23 (13%) < High School Education 73 (40%) 47 (26%) Unemployed 87 (47%) 86 (47%) Medicare or Medicaid 99 (54%) 99 (55%)

Self-monitoring and feedback via CHOICE of
One in-person visit, introduction to website with follow-up phone calls Self-monitoring and feedback via CHOICE of - Web, - IVR and print The intervention was designed to be low cost and to appeal to African American patients- they could access it via either their cell phone or any Internet device

Results REACH EFFECTIVENESS*
60% of eligible population was invited to participate (604). Of those, 365 (60%) completed baseline and were randomized. Those who participated vs. those who did not, were younger and had a higher mean BMI. No other differences were found on demographics. EFFECTIVENESS* At 24 months, intervention participants had greater weight losses compared with those receiving usual care (difference, −1.03 kg; 95% CI, −2.03 to −0.03 kg) Mean systolic blood pressure was significantly lower in the Ix arm compared with usual care * No differential patterns in outcomes observed for minority vs. non-minority or disparity- related sub-groups. Intervention participants had greater 24-month weight losses compared with those receiving usual care (difference, −1.03 kg; 95% CI, −2.03 to −0.03 kg;). In addition, the intervention promoted larger mean weight losses in 24 months relative to usual care (AUC difference, −1.07 kg; 95% CI, −1.94 to −0.22 kg). The proportion of those who lost at least 5% of their initial body weight during the 24 months was 19.5% for usual care and 20.0% for intervention participants. Similar patterns were observed for percent weight loss and change in body mass index Bennett et al. Obesity Treatment for Socioeconomically Disadvantaged Patients…Arch Intern Med. 2012;172(7):

Results (cont.) ADOPTION IMPLEMENTATION MAINTENANCE
All three centers invited, participated. Four centers were excluded for lack of EHR system 19 of 20 primary care physicians (95%) referred their patients to the program IMPLEMENTATION 71% completion rate for counseling calls; 63% of participants completing >70% of their calls English speakers were more likely to have goals, barriers and strategies documented (P<0.0001), as were participants making more than $10,000 (P<0.001) MAINTENANCE Strong individual-level maintenance with no sub-group differences, but at the setting- level none of the centers maintained the program components. At the setting level- remember RE-AIM has dimensions at both the setting and individual level you can see that all 3 community health centers invited partcicpated and the intervention was =delivered with fair amount of consistency Glasgow RE et al. Use of RE-AIM to Address Health Inequities…Transl Behav Med 2013 Jun 1;3(2):

Considering Context- PRISM extension of RE-AIM Planning programs- estimating RE-AIM impacts Adaptations- types and implications Representativeness and transparent reporting Be Fit Be Well example Your feedback and opportunities for D&I research

WE'RE HIRING! Who: D&I Scientist at the Associate or Advanced Assistant Professor level Qualifications: Knowledge of D&I models, measures & methods, strong research record, experience in clinic/community. Ability to function independently, consult with/mentor junior faculty and those new to D&I, and some level of own funding Opportunities: Collaborate with D&I experts and others in diverse programs, departments and disciplines (SOM, SPH, VA, Cancer Center, CTSA, Children’s Hospital Colorado, etc.). Be part of a dynamic growing program focused on pragmatic research- both in D&I and health services outcomes research generally. For More Information: Google “ACCORDS D&I Science” Search Committee Chair:

Questions? ‘I am all ears!’

The 5 Rs to Enhance Pragmatism, D&I Science and Likelihood of Translation
Research that is: Relevant Rapid and recursive Redefines rigor Reports resources required Replicable Peek, CJ, et al. (2014). The 5 Rs: An emerging bold. Annals of Family Medicine, 12(5), doi: /afm.1688 deGruy, FV, et al. (2015). A plan for useful and timely family medicine and primary care research. Family Medicine, 47(8),

An Evidence-Based Cancer Prevention. or Depression Reduction
An Evidence-Based Cancer Prevention... or Depression Reduction... or Care Transition.....or (fill in blank) Story Even if 100% effective, it’s only as good as how and whether: it is adopted, and where it is not adopted practitioners are trained to deliver it - and who is not trained trained practitioners consistently deliver it - and who does not eligible populations receive it - and which do not it can be sustained - and where, why and when it is not If we assume 50% success for each step (even with perfect access/adherence/dosage/maintenance, and equal benefit throughout) Impact: .5x .5x .5x .5 x .5 = 3% benefit Glasgow RE, Vogt TM, Boles SM. Evaluating the public health impact ... Am J Public Health. 1999;89(9):1322. Battaglia, C, Glasgow RE. Pragmatic D&I research. (2018). Nursing Outlook

Pragmatic RE-AIM “Precision Implementation” and Health Questions
Determine: What percentage and what types of patients are Reached? For whom is the intervention Effective in improving what outcomes (including health equity) with what unanticipated consequences? In what percentage and in what types of settings and staff is this approach Adopted? How consistently are different parts of it Implemented, at what cost, to different parties? And how well are the intervention components and their effects Maintained? Gaglio B, Glasgow RE. Evaluation approaches…In: Brownson R, Colditz G, Procter E, (Eds). Dissemination and implantation research in health: Translating science to practice. New York: Oxford University Press; Pages

Why Is This Important? Consider Impact of Loss at Each RE-AIM ‘Stage’
Example of Translation of Interventions into Practice Dissemination Step RE-AIM Concept % Impact 50% of settings use intervention Adoption 50.0 50% of staff take part 25.0 50% of patients identified, accept Reach 12.5 50% follow regimen correctly Implementation 6.2 50% benefit from the intervention Effectiveness 3.2 50% continue to benefit after six months Maintenance 1.6

PCTs: Fewer Exclusions Allow for a Broader Subset of Settings, Staff and Participants
Traditional RCT PCT Efficacy, among a defined subset Eligible population Exclusions, non-response, etc. Effectiveness, in a broad subset Figure provided by Gloria Coronado, PhD, Kaiser Permanente Center for Health Research

Types of Outcomes in Implementation Research
Implementation Outcomes Service Outcomes Client Outcomes Acceptability Efficiency safety Satisfaction Adoption Effectiveness Function Appropriateness Equity Symptoms Costs Patient-centeredness Feasibility Timeliness Penetration Sustainability Proctor E, Silmere H, Hensley M, et al. Outcomes for implementation research: Administration And Policy In Mental Health [serial online]. March 2011;38(2):65-76.

Implementation Science, Context, and Health Equity: RE-AIM Applications and Ideas for Feedback Russell E. Glasgow, PhD Department of Family Medicine, University.

Similar presentations

Presentation on theme: "Implementation Science, Context, and Health Equity: RE-AIM Applications and Ideas for Feedback Russell E. Glasgow, PhD Department of Family Medicine, University."— Presentation transcript:

Similar presentations

About project

Feedback

Log in

Auth with social network:

Implementation Science, Context, and Health Equity: RE-AIM Applications and Ideas for Feedback Russell E. Glasgow, PhD Department of Family Medicine, University.

Similar presentations

Presentation on theme: "Implementation Science, Context, and Health Equity: RE-AIM Applications and Ideas for Feedback Russell E. Glasgow, PhD Department of Family Medicine, University."— Presentation transcript:

Similar presentations

About project

Feedback