1. Finding Evidence-based Interventions
Talking Points:
In this session we will cover online resources for finding evidence-based packaged programs, policies, and/or strategies.
This presentation was supported by Cooperative Agreement Number 3 U48 DP S8 from the Centers for Disease Control and Prevention. The findings and conclusions in this presentation are those of the author(s) and do not necessarily represent the official position of the Centers for Disease Control and Prevention.
The Cancer Prevention and Control Research Network is supported by Cooperative Agreement Number 3 U48 DP S8 from the Centers for Disease Control and Prevention’s Prevention Research Centers Program and the National Cancer Institute. The content of this curriculum is based upon findings and experiences of workgroup members and does not necessarily represent the official position of the funders. 

2. Where Do We Stand? Establishing Goals & Objectives
Assessing your Community
Finding EBI
Selecting Best Fitting EBI
Adapting EBI
Talking Points:
In this session you will find resources that will be useful tools for your work. Please also share about other tools where you’ve found useful evidence.
Once you know your audience and the disease or behavior you want to address, it is time to start the search for an evidence-based strategy that may already be available and can be adapted to fit your needs.
We have defined evidence and talked about the importance of using evidence. Now we will focus on how and where to find those evidence-based programs, policies and recommendations.
Implementing & Evaluating
*Adapted from Brownson et al. (2017). Evidence-Based Public Health. 3rd ed. New York, NY: Oxford University Press.

3. Session Objectives Identify approaches to:
Find evidence-based interventions (programs, policies, and strategies)
Look for additional resources to assist in planning and implementing an intervention  
Apply criteria to evaluate sources of evidence-based interventions 
*What we expect to accomplish during this session
*Questions welcome throughout.

4. Importance of Using Evidence
Builds on what others have found works
Adds credibility to your work
Remember to cite the source for your strategy!
Talking Points:
The benefits of using evidence are many – not only does it help share and build upon what is working in public health, but it also adds credibility to your work (which is really important for various stakeholders, community members, and funders).
To maintain this credibility it helps to cite the source of your strategies as you are working with partners and stakeholder.

5. Challenge Practitioners Face Finding EBIs
“Honestly the frustration right now for me is you’ve got Robert Wood Johnson, Policy Link, CDC, Convergence Partnership. You name it and everyday somebody is coming out with something.”
A public health practitioner working in obesity prevention
Talking Points:
Here is a quote from a public health practitioner working in obesity.
It’s easy to get overwhelmed with the various sources and amount of evidence. The goal of today’s session is to show you options of where to go and what to look for.

6. Expert opinion or personal experience
Sources of Evidence
Systematic reviews
Research studies
Practitioner reports
Expert opinion or personal experience
Scientifically Supported
“Best” or “Proven”
Some Evidence
Practice-Tested:
“Promising”
Strategies             
Programs
Talking Points:
Evidence to guide practice can be derived from different types of research as well as practical experience. Health professionals make decisions based on various types of evidence, spanning the spectrum from more subjective evidence—such as their direct experience with the populations they work with—to more objective sources of evidence—including the results of well-designed research studies.
Systematic Reviews are generated by a team of researchers and practitioners who examine methods and findings of multiple, rigorously conducted studies to summarize evidence into recommendations for what works, such as recommendations related to worksite wellness programs to reduce obesity. The Community Guide, a collection of evidence-based findings from the Community Preventive Services Task Force, is one example.
Research studies refer to individual studies that test the efficacy of a specific strategy. The results might be packaged for dissemination or described in a peer-reviewed journal publication.
Practitioner reports of implementation: include reports on the findings of an evaluation of a strategy. As a practitioner, this could also be your own program evaluation. These might be shared through reports, briefs, or success stories.
Expert opinion or personal experience: could be recommendations made by credible groups in support of strategies that have not yet been tested. An example of this is “Complete Street” policies, which are policies and design strategies that require streets to be planned, designed, operated, and maintained to be safe and convenient for all users regardless of their mode for transportation. Many credible experts recommend “Complete Street” policies, but it will take years for these strategies to be tested. You might also use personal experience. For example, you might have discovered that text message reminders are more effective than phone calls for reminding parents in your population to bring adolescents in for vaccinations.
It is important to note that only the top two tiers—Systematic Reviews and Research Studies—are generally referred to as “evidence-based interventions”.

7. Resources for Evidence
Talking Points:
A number of resources will be covered in this presentation. We will not cover all resources that disseminate evidence-based program, policy, or strategy. But, we will provide an overview of useful resources that are easily accessible.

8. Types of Evidence-Based Interventions
Environment or Behavior
Strategies
(recommended after systematic review)
Policies
Packaged Programs
Evidence-Based Interventions
Talking Points:
In the session on “Defining Evidence” that you did as an online module prior to the workshop, you learned about 3 categories of evidence-based interventions: Programs, Policies, and Strategies.
Packaged programs;
Policies – at both public policy and at organizational levels;
Strategies, primarily those derived from systematic reviews, such as the Community Guide.
In this session, we are going to talk about sources of packaged programs and strategies.

9. Activity: Evidence Scavenger Hunt
Discussion
What are the key differences between the Community Guide and RTIPS?
Using the Scavenger Hunt Activity
*Can tailor depending on audience.
Activity Instructions (5 minute discussion review):
Spend time going back to review RTIPS and Community Guide
Will review some key questions –
1. Key differences between Community Guide and RTIPS – RTIPS includes interventions/programs (including materials), and is more clinically focused. Community Guide includes broad strategies.
2. Similar in that they both use peer-reviewed journals for evidence. Both do not include strategies with evidence from the practice community.
Many of the programs in RTIPs are also directly linked to associated Community Guide strategies.

10. Key Criteria for Finding Evidence
What types of evidence and other information are they disseminating?
What is the potential for bias?
How current is the information?
What are their methods for reviewing the evidence?
What criteria do they use to rate the evidence? 
Talking Points:
As we go through each of these websites, we will be addressing the following questions.
Consider writing this questions on a flip chart or white board.
Then as you proceed through each of the websites that follow, point out where they would find the answers to each of these questions.

11.
Sponsor: Taskforce for Preventive Services, Centers for Disease Control and Prevention
Health topics: Multiple
Resources available: Strategies (Recommendations from systematic reviews)
Talking Points:
The first resource is the Community Guide to Preventive Services. It covers a spectrum of public health documents.
It presents strategies for prevention from systematic reviews.

12. Ask the Audience:
Who has used the Community Guide to Preventive Services?
How have you used it?
Refer to the Criteria for Evaluating the resource – Note that you can go to “about us” to see who created the resource (CDC). Do they have profit or ideological motives? Note that you can go to the methods menu tab to learn about the Community Guides methods and criteria for systematically reviewing the literature.

13. Methods – systematic review of the literature
Criteria 
What criteria does the Community Guide use to rate evidence?
Reports using the following criteria:
Recommended: Strong or sufficient evidence that the strategy is effective; Based on study design, number of studies, and consistency of effect across studies. Not based on the expected magnitude of benefits.
Insufficient Evidence: The available studies do not provide sufficient evidence to determine if the strategy is, or is not, effective. More research is needed.

14. Talking Points:
Let’s look at what the recommendations look like for the physical activity topic.

15. Talking Points:
The Task Force provides recommendations for behavioral/social strategies, campaigns and informational strategies, and environmental/policy strategies. This is an example of the environmental and policy strategies.
Also note date recommended.  The Community Guide is a great source for strategies but some of its reviews are old - (point to criteria – is it current). When the last review as along time ago, you may want to search for more recent research studies or better yet systematic reviews and use those in addition to the community guide. Shortly we will be looking at a great website where you can get more current systematic reviews.

16. Talking Points:
The Community Guide provides a snapshot, findings, supporting materials, and considerations for implementation.
You can see here under Supporting Materials for Point-of-Decision Prompts to Encourage Use of Stairs, a summary evidence table summarizing the included studies.

17. Other Sources of Systematic Review Recommendations
9-Sep-18
Other Sources of Systematic Review Recommendations
Cochrane Collaboration Center for Reviews and Dissemination Campbell Collaboration Guide to Clinical Preventive Services Using What Works for Health National Guideline Clearing House
Talking Points:
Some of the additional sources of strategies available on the Internet. Mention Cochrane specifically and Guide to Clinical Preventive Services.
Note Handout providing a list of sources and other websites for topic-specific evidence.

18. What Works for Health http://www.countyhealthrankings.org
Sponsor: Robert Wood Johnson Foundation
Health topics: Multiple
Resources available: Recommendations and programs
Talking Points:
Let's spend a few minutes looking at one more source of recommendations from systematic reviews.

19. Talking Points:
Using our checklist, review home page: Who (RJWF), methods, etc.

20. Talking Points:
NOte the 2nd sentence: review systematic reviews and also review unpublished literature (called grey lit) in addition to systematic reviews. 
Grey literature could be white papers, briefs, reports – lots of organizations share evidence in that form vs. submitting papers for peer review. 
The methods here are not as rigorous as the Community Guide – There goal is to get the latest information out there quickly and they do a great job of it. 

21. “What Works for Health” Rating Criteria
Scientifically Supported
It works: numerous studies or systematic review(s) with strong positive results
No substantial contradictory evidence
Some evidence
Likely to work, but further research is needed to confirm effects
Expert Opinion
Recommended by credible groups
Research evidence limited
Insufficient Evidence
Study quality varies
Findings vary
Mixed Evidence
Some studies have significant findings
Findings inconclusive
Findings mixed leaning negative
Evidence of Ineffectiveness
Significant negative/ineffective findings
Strong evidence of harm
Talking Points:
Note that the terms are different than we saw for the Community Guide and this is one of the challenges.
Ask the Audience: Why would you want to use a strategy that’s only expert opinion?
May be something that is emerging but the research has not caught up with the practice. Example is complete streets policies – experts recommended that this would help increase physical activity, but took several years to start to see research findings that supported a connection between complete street policies and physical activity)

22. Talking Points:
Browse All Policies & Programs from home page
Also note New Policies & Programs – these are released within the past 30 days so very current

23. Demo: County Health Rankings & Roadmaps
Talking Points:
Use this link to access their webpage about places for physical activity. 
Note that the website references the Community Guide (CG) and also provides easy access to more recent systematic reviews that you can use to make the case that this strategy is evidence based. When making a case for the evidence-base, cite both the COmmunity GUide and these more recent webistes. You can use this website to do this for many of the community guide sites.
Note also the implementation examples and the links to other websites that provide resources to support implementation of this strategy.

24. Sources of Packaged Programs
Environment or Behavior
Strategies
(recommended after systematic review)
Policies
Packaged Programs
Evidence-based Strategies
Talking Points:
Now we are going to talk about sources of packaged programs

25. http://rtips.cancer.gov/rtips Sponsor: National Cancer Institute
Health topics: Multiple
Resources available: Programs
Talking Points:
This next resource is the Research-tested strategies Programs (RTIPS) website. It is sponsored by National Cancer Institute and the Substance Abuse and Mental Health Services Administration.
It covers many cancer topics. The type of evidence available here are packaged evidence-based programs.

26. Ask the Audience:
Who here has heard of RTIPs?
RTIPS from National Cancer Institute
Provides a lot of programs for behavioral health change
There are 175+ programs (use strategies only in past 10 years)
Criteria
Who created the resource? Are they credible and relatively unbiased?
See RTIPS Submission and review process: A guide for program developers to see their methods.
Click on the strategy program links to start our search.

27. Talking Points:
This is the search engine for RTIPs.
RTIPs gives you a tool to refine your search and once you run that query on Colorectal cancer (CRC), for example, the list of possible programs is reduced to that that have a focus on CRC. You could further narrow the search by clicking on items under setting or gender or race/ethnicity for example.

28. Talking Points:
Before submitting the colorectal cancer search it looks like we could have narrowed the search even further. Had we searched on CRC and African Americans (AAs), the search would have yielded only 4 results, those for which AAs were the target population. Including materials available on RTIPs yields only three results.

29. Criteria
Criteria for rating – point out the RTIPs Scores. Research integrity relates to the design and execution of research testing the strategy. Unlike the Community Guide, they do look at size of benefits (strategy impact).
Also look at how ready is for others to use (dissemination capability). Also note the RE-AIM criteria. Looks at evidence related to reach to intended population, adoption by providers and settings for which it was designed, and quality of implementation.
Ask the audience:
Is the information current? Look at date under purpose (2010).

30. Talking Points:
Additional guidance on implementation
For each RTIP, there is a page that provides basic information on that program such as its descriptions, results, and what Community Guide Preventive Services strategy type that it represents.

31. Activity Use the resource list to search for an EBI
Apply the five criteria to evaluate the website disseminating the EBI
What is the type(s) of EBI?
What are the sources of bias?
Is the information current?
What are the methods for selecting EBIs?
What are the criteria for rating EBIs?
Talking Points:
There are many different sources of evidence, just remember organizations vary on criteria they use.
Important to use list of questions we shared as you are going through a source of evidence.

32. Take-home Points
There are no agreed-upon criteria for assessing an EBI
Organizations vary on criteria used
There are numerous sources of EBIs, but it’s buyer beware!
Talking Points:
There are many different sources of evidence, just remember organizations vary on criteria they use.
Important to use list of questions we shared as you are going through a source of evidence

33. Questions?
Ask the Audience:
Do you have any questions?

34. References
Jacobs JA, Jones E, Gabella BA, Spring B, Brownson RC. (2012). Tools for Implementing an Evidence-Based Approach in Public Health Practice. Prev Chronic Dis, 9:
Brownson RC, Baker EA, Leet TL, Gillespie KN. (2003). Evidence-Based Public Health. New York: Oxford University Press.
Brownson RC, Fielding JE, Maylahn CM. (2009). Evidence-Based Public Health: A fundamental concept for public health practice. Annu. Rev. Public Health, 30: