1. The Promise and Challenge of Dissemination and Implementation Science (DIS)
Russell E. Glasgow, PhD
Director, ACCORDS DISSEMINATION AND IMPLEMETATION SCIENCE Program
University of Colorado School of Medicine
Acknowledgments:
Ross Brownson, David Chambers, PAUL ESTABROOKS, BRYAN FORD, Amy Kilbourne, Borsika Rabin and the great researchers cited throughout this talk

2. Everything I know (about DIS)
Dissemination and Implementation Science (DIS) is about:
Multi-level, contextual issues, and external validity
Relevant, pragmatic models, research methods and measures
Real world implementation and adaptation
Designing for dissemination, sustainability and equity
(Normal science (T1– T2) is necessary but not sufficient)

3. Objectives Why do we need DIS? How can DIS inform your work?
What might you expect if you engage in DIS? better outcomes: better fit to local context, better uptake, etc.
Current and Evolving Issues in DIS
Overview some current DIS research areas and future opportunities

4. “PUBLICATION PATHWAY”
Balas & Boren, 2000
Original research
18%
variable
Negative results
Dickersin, 1987
Submission
46%
0.5 year
Kumar, 1992
Koren, 1989
Acceptance
Negative results
0.6 year
Kumar, 1992
Publication
Expert
opinion
35%
0.3 year
Poyer, 1982
Balas, 1995
Lack of numbers
Bibliographic databases
Reviews, guidelines, textbook
9.3 years
Inconsistent
indexing
Implementation
“PUBLICATION PATHWAY”

5. It takes 17 years to transform 14% of original research
Balas & Boren, 2000
Original research
18%
variable
Negative results
Dickersin, 1987
Submission
46%
0.5 year
Kumar, 1992
Koren, 1989
Acceptance
It takes 17 years to transform 14% of original research
into the benefit of patient care
Negative results
0.6 year
Kumar, 1992
Publication
Expert
opinion
35%
0.3 year
Poyer, 1982
Balas, 1995
Lack of numbers
Bibliographic databases
Reviews, guidelines, textbook
9.3 years
Inconsistent
indexing
Implementation
“PUBLICATION PATHWAY”

6. Need for Pragmatic DIS Research
Traditional biomedical RCTs study the effectiveness of treatments delivered to carefully selected populations under ideal conditions
Even when we do implement a tested intervention into everyday clinical practice, we often see a “voltage drop”…a dramatic decrease in effectiveness
Most common reason evidence- based programs are not adopted…they are not seen as relevant
“If we want more evidence-based practice, we need more practice-based evidence.”
Green LW
Am J Pub Health 2006
Rothwell PM. External validity of randomised controlled trials. Lancet 2005;365:82-93.

7. Key Differences Between Traditional RCTs and Pragmatic DIS Controlled Trials (PCTs)
A traditional RCT tests a hypothesis under ideal conditions
A DIS Pragmatic Trial compares treatments under everyday clinical conditions
GOALS
To determine causes and effects of treatment
To improve practice and inform clinical and policy decisions
DESIGN
Tests the intervention against placebo using rigid study
protocols and minimal variation
*Tests two or more real-world
using flexible protocols & local customization*
PARTICIPANTS
Highly defined and carefully selected
More representative because eligibility criteria are less strict
MEASURES
Require data collection outside routine clinical care
Brief and designed so data can be easily collected in clinical settings
RESULTS
Rarely relevant to everyday practice
Useful in everyday practice, especially clinical decision-making

8. T1-T4 Research Which type (T) is most common?

9. Bench to Bookshelf
The end result of these challenges is that the most common type of research today is not ‘bench to bedside’; or ‘bedside to community’, but ‘bench to bookshelf’.

10. Some Remedies and Key Terms
Implementation science is the study of methods to promote the integration of research findings and evidence into healthcare policy and practice.
Dissemination research is the scientific study of targeted distribution of information and intervention materials to a specific public health or clinical practice audience. The intent is to understand how best to spread and sustain knowledge and the associated evidence-based interventions.
Implementation research is the scientific study of the use of strategies to adopt and integrate evidence-based health interventions into clinical and community settings to improve patient outcomes and benefit population health.
Pragmatic research is the use of real-world tests in real-world populations and situations.
*Brownson, Colditz & Proctor. Dissemination and Implementation Research in Health 2018

11. A Big Tent of Terms (and ovals)
Population Health (and Community) Services
Health Services Research
Implementation Science
Implementation Research
Dissemination Research
Pragmatic Research
Health Communication Research
Quality Improvement Science QI .
Adapted from Mitchell S, Chambers, D

12. Key Characteristics of D&I Science
Glasgow RE, Chambers D. Developing robust, sustainable, implementation systems using rigorous, rapid and relevant science. Clin Transl Sci. 2012;5(1):48

13. Too often, we have assumed, “If you build it…and if you have evidence”

14. An Evidence-Based Cancer Prevention. or Weight Loss. or Mental Health
An Evidence-Based Cancer Prevention... or Weight Loss... or Mental Health.....or (fill in blank) Story
Even if 100% effective...is only so good as how and whether:
it is adopted
practitioners are trained to deliver it
trained practitioners choose to deliver it
eligible populations receive it
it can be sustained
If we assume 50% threshold for each step…
(even with perfect access/adherence/dosage/maintenance)
Impact: .5x .5x .5x .5 x .5 = 3% benefit
Glasgow RE, Vogt TM, Boles SM. Evaluating the public health impact of health promotion interventions: The RE-AIM framework. Am J Public Health. 1999;89(9):1322.

15. Pragmatic DIS Use of RE-AIM
RE-AIM Dimension
Key Pragmatic Priorities and Key DIS Questions to Consider and Answer
Reach
WHO is/was intended to benefit, and who actually participates or is exposed to the intervention?
Effectiveness
Adoption
WHAT is/was the most important benefits you are trying to achieve and what is/was the likelihood of negative outcomes?
WHERE is/was the program or policy applied and WHO applied it?
Implementation
HOW consistently is/was the program or policy delivered, HOW will it be adapted, HOW much will/did it cost, and WHY will/did the results come about?
Maintenance
WHEN will/was the initiative become operational; how long will/was it be sustained (setting level); and how long are the results sustained (individual level)?
Glasgow R & Estabrooks P. Pragmatic applications of RE-AIM for health care initiatives in community and clinical settings. Preventing Chronic Disease. 2018; 15(1): E02.

16. The 5 Rs to Enhance Pragmatism, DIS and Likelihood of Translation
Research that is:
Relevant
Rapid and Recursive
Redefines Rigor
Reports Resources Required
Replicable
LINDA)_ PLEASE MAKE LARGER TO FILL SLIDE
Peek, C.J, et al. (2014). The 5 Rs: An Emerging Bold Standard for Conducting Relevant Research in a Changing World. Annals Of Family Medicine, 12(5), doi: /afm.1688
deGruy, F.V, et al. (2015). A plan for useful and timely family medicine and primary care research. Family Medicine, 47(8),

17. Reporting Resources Required
Reporting on cost and other resources in a standardized manner is useful in:
Demonstrating value
Promoting rigor, transparency and relevance to stakeholders
Present from perspective of stakeholders and decision makers
Simple is fine – sophisticated economic analyses are not needed
Report costs of conducting or replicating interventions
Beyond money, costs can include clinician and staff time, training, infrastructure, startup costs, opportunity costs
Ritzwoller, D P, et al. (2009). Costing behavioral interventions: A practical guide to enhance translation. Annals of Behavioral Medicine, 37(2),

18. How well do research
and practice connect?
Dissemination

19. Dissemination Disconnects
How local public health agencies learn about research findings?
How researchers perceive they most effectively reach practitioners?
Professional associations
Journal articles
Seminars/workshops
Face-to-face meetings
alerts
Media interviews
Press releases

20. Dissemination: What we know
Dissemination generally does not occur spontaneously and naturally;
Passive approaches to dissemination (diffusion) are usually ineffective;
Single-source prevention messages are generally less effective than comprehensive, multi-level approaches;
Stakeholder involvement in the research or evaluation process is likely to enhance dissemination;
Theory and frameworks for dissemination are beneficial; and
The process of dissemination needs to be tailored to various audiences
From a broad set of literature, both within and outside health, here are some key points on what we know about dissemination of evidence-based programs and policies. You will hear these themes addressed thru-out our course. Many of these apply as much to implementation science as to dissemination science.
Brownson RC, et al. Designing for dissemination among public health researchers......
Am J Public Health. 2013;103(9):

21. DIS Models + Frameworks are available!
Outlined in Tabak, et al., proliferation of models: 61 reviewed, now more than 150!!
Context is critical
Focus on external validity
Begin with stakeholders—take their perspective(s)
Find balance between fidelity to EB program and adaptation to local setting
Unlikely you need to create a new model
website by Rabin et al.
Tabak RG, et al. Bridging research and practice. Am J Prev Med. Sep 2012;43(3):
Mitchell S & Chambers, D. Leveraging Implementation Science...J Oncology Practice,

22. Mode

23. Types of Outcomes in DIS Research (Proctor, et al., 2010)
Implementation Outcomes
Service Outcomes
Client Outcomes
Acceptability
Efficiency
Satisfaction
Adoption
Effectiveness
Function and QoL
Appropriateness
Equity
Symptoms
Costs
Patient-centeredness
Feasibility
Timeliness
Penetration
Safety
Sustainability
Proctor E, Silmere H, Hensley M, et al. Outcomes for implementation research: Administration and Policy in Mental Health [serial online]. March 2011;38(2):65-76.

24. Key Outcomes Related to Health Equity
Penetration (Reach)...It is ‘about the denominator’
Representativeness- levels of community, setting, staff, patient
Costs and Feasibility- for setting; staff; and patients/citizens/users
Other

25. Evidence-based...but on what?
Sorely needed on external validity and pragmatic criteria…often ignored, including:
Participant Representativeness
Setting Representativeness
Context and Setting
Community/Setting Engagement
Adaptation/Change
Sustainability
Costs/Feasibility of Treatment
Comparison Conditions

26. Reporting for D&I vs. Usual HSR
Focus
Internal validity (some EV)
External validity (some IV)
Priority
Fidelity
Adaptation
Format
CONSORT
Expanded CONSORT; StaRI

27. Questions? “I am all ears!”

28. Future Opportunities
Examples of how DIS Research can help address current and future prevention, healthcare, health equity and population health

29. Example One: Learning Healthcare Systems (and Communities)
Big Data, e-Health/m-Health, geo-coding
Consumer-driven Healthcare policy reform
Exchanges/ACOs
Medicaid expansion
Learning Healthcare Systems
Aligning science with clinical priority goals
Emphasis on costs and value
Conducting more rapid and efficient studies
Leveraging existing data to deploy and evaluate innovations and best practices
ALSO VERY RELEVANT – VERY STRONG DESIGN LAB (USED TO BE LED BY KEVIN PATRICK NOW TAKEN OVER BY VERY TALENTED RELATIVELY NEW RECRUIT ERIC HECKLER -

30. The Long Road to Learning Healthcare Systems and Communities (and why we need DIS and pragmatic application)
New research takes too long for adoption (if at all)
Research is often not aligned to address critical health and health care problems or feasible
Research is often not designed with stakeholders
Providers lack practical tools/technical assistance and strategies to implement EB treatments (data are not enough)
Large programs are often rolled out without adequate planning to maximize effectiveness, learning and sustainability
Variation and patient-centered care
Treatments work differently for different people and different settings

31. Example Two: Precision Medicine (Health)
VERY RELEVANT – WE HAD JUST 2 WEEKS AGO THE PUBLIC HEALTH RESEARCH DAY PLENNARY SPEAKER TALK ABOUT : “
Putting Genomics in Context to Promote Health ”
Ellen Wright Clayton, MD, JD
Rosalind E. Franklin
Professor of Genetics & Craig -
Weaver Professor of Pediatrics
Department of Health Policy, Vanderbilt University School of Medicine
Professor of Law, Vanderbilt University School of Law
How does clinical practice incorporate PMI findings?
How do you implement evidence that will evolve?
What does it cost and what are opportunity costs?
How do you train and support the workforce?
How do you ensure that PM findings reduce, not exacerbate, health inequities?

32. Convergence of Precision Health, DIS & Learning Health Care Systems and Communities
GREAT!!
Chambers DA, Feero WG, Khoury MJ. Conversion of implementation science, precision medicine, and the learning health care system. JAMA. 2016, 315:

33. RE-AIM Precision Medicine (PM- or Precision Health) Questions
Determine
What percent and types of patients are Reached (equity);
For whom among them is the PM intervention Effective, in improving what outcomes, with what unanticipated consequences (is health equity improved or decreased )?
In what percent and types of settings is this approach Adopted;
How consistently are different PM Implemented at what cost to different parties;
And how well are the intervention components and their effects Maintained at both setting and individual levels?
Gaglio B, Glasgow RE. Evaluation approaches…In: Brownson R, Colditz G, Proctor E, (Eds). Dissemination and implementation research in health. New York: Oxford University Press;

34. Current DIS Funding Opportunities
NIH: PAR ; ; (18 ICs & Offices)
Purpose: To support innovative approaches to identifying strategies for the adoption, adaptation, integration, scale-up and sustainability of evidence-based interventions, tools, policies, and guidelines.
Also benefit in studying how to “de-implement” or reduce the use of strategies and interventions that are not evidence-based, yield sub- optimal benefits for patients, and are harmful or wasteful.
PCORI: Dissemination and Communication and Large Pragmatic Trials
NHLBI: Center on Translational Research and Implementation Science (training and research); upcoming discussion of new RFA
Add Fogarty?

35. Key Science Questions- New and Old
Traditional science and evidence question: (necessary but not sufficient): “What intervention produces the largest average effect in tightly controlled trials on the major (clinical) outcome?
Pragmatic DIS question: (contextual)
“What program/policy components are most effective for producing what outcomes for which populations/recipients when implemented by what type of persons using what strategies under what conditions, with how many resources and how/why do these results occur?”
Should I include this here...or not...save for longer talk? Or good to point ow how DIS relates to T2- T3???
ABSOLUTELY YES – WE CAN EVEN MOVE IT UP AND HAVE ONE SLIDE ON T1-T4 AS WELL

36. Everything I know (about DIS)
Dissemination and Implementation Science (DIS) is about:
Multi-level, contextual issues, and external validity
Relevant, pragmatic models, research methods and measures
Real world implementation and adaptation
Designing for dissemination, sustainability and equity
(Normal science (T1– T2) is necessary but not sufficient)

37. USE FOR DAY 1 AS WELL!!!

38.
University of Colorado School of Medicine ACCORDS D&I Program
SOME FROM BELOW SHOULD MOVE UP

39. Future Opportunities (and biased speculations)
*Getting serious about CONTEXT: Measures; designs; international collab.
*Real integration of health policy, public health, and biomedicine; clinical and community; impact on policy
Focus on social determinants of health, health equity, generalizability of results to low resource settings, individuals and LMIC (and vice versa)
Focus on COSTS, resources, de-implementation, and comparative effectiveness research
Precision medicine, big data, AI and learning health systems

40. Expanded CONSORT Figure
Existing CONSORT Figures for RCTs focus almost exclusively on internal validity
Use of CONSORT- required by most medical journals, has improved quality and reporting on internal validity
Until now, no comparable resource has addressed external validity URL:
Glasgow, Huebschmann, Brownson. Amer J Prev Med, 2018, DOI: 

41. General Resources
Brownson, RC, Colditz, GA, & Proctor, EK (2018). Dissemination and implementation research in health: translating science to practice. Oxford University Press.
re-aim.org
RUSS- something on pragmatic

42. ACCORDS Dissemination and Implementation Science Program, University of Colorado Denver School of Medicine
Our goal is to:
Provide local consultation on D&I related research to increase funding and publication success
Create collaborative learning partnerships with embedded research settings to translate research into practice more quickly and successfully
Conduct cutting edge T3-T4 research on: pragmatic research and measures, adaptation of interventions, designing for dissemination, shared decision making, planning for and evaluation of reach, implementation and sustainability
Use interactive on-line resources and support for patients, medical and public health students, trainees and faculty researchers
Communicate the latest information on DIS related conferences, articles, grant opportunities, events, webinars, talks, and training opportunities

43. Bethany- want to do this
Bethany- want to do this?- either live or maybe better to just use screen shot- and we could have the website online during Q&A... Up to you. 

44. T1 – T4 Extra