1. Understanding the term “Evidence-based”
What is it and and why do I care?

2. Practical Exercise
Look at each scenario and decide if the example give is Evidence-based.
Don’t be scared.

3. Practical Exercise
Your community coalition develops a logic model for the problem of youth binge drinking. It has been found through your community assessment that parents often host parties at their homes for local teenagers. The coalition plans to educate local lawmakers with the goal of implementing a city wide social host ordinance
Evidence based Not Evidence Based

4. Practical Exercise
Recently, your coalition has been approached by a concerned citizen who stated that, “2 kids were recently found with steroids in the school and your coalition needs to do something about it.” As the sole strategy, your coalition decides to put up 32 brochure sites within the middle school and high school. These sites have brochures that talk about the dangers of steroids.
Evidence based Not Evidence Based

5. Practical Exercise
Recently, your coalition has done its community assessment. The committee developing the logic model has asked you to implement your 6B APPROVED program to address the risk factor of “low perception of risk for alcohol use in youth under the age of 18”. You advise them that your 6B program for this situation is:
Evidence based Not Evidence Based

6. Evidence-based defined:
Programs or practices that effectively integrate the best research evidence with clinical expertise, cultural competence and the values of the persons receiving the services. These programs or practices will have consistent scientific evidence showing improved outcomes for clients, participants or communities.
Evidence-based practices may include individual clinical interventions, population-based interventions, or administrative and system-level practices or programs.
Simply put, a program is judged to be evidence-based if (a) evaluation research shows that the program produces the expected positive results;
(b) the results can be attributed to the program itself, rather than to other extraneous factors or events; (c) the evaluation is peer-reviewed by
experts in the field; and (d) the program is “endorsed” by a federal agency or respected research organization and included in their list
of effective programs.
Oregon Department of Human Services, (2007). Operational Definition for Evidence-Based Practices. Addictions and Mental health Division. Retrieved on February 1, 2011 from

7. 4 Criteria Program or Strategy produced expected positive results
Results can be attributed to the program
Peer reviewed
“Endorsed” by federal program or respected research organization

8. Research-based defined:
What do you think it should be?
Some of the common labels used to describe programs that meet the criteria of evidence-based include: exemplary programs, effective programs, model programs, proven programs, research-based or science-based and what works. The growth of evidence-based prevention programs is a relatively recent phenomenon.
“evidence-based” has been interpreted differently by different groups and has been called by different names, to include “research-based”. Most of these definitions however, share a common set of core characteristics that make up a common definition for evidence-based and set it apart from “research-based”. For a program to be considered evidence-based, it must have been tested in a well-designed and methodologically sound study and have been show to produce significant reductions in poor outcomes or associated risk factors. Further, the body of research demonstrating evidence of effectiveness must have undergone review by experts in the field in order to identify possible methodological flaws in the research or statistical analyses. This is usually, though not exclusively, accomplished through publication of the findings in a peer-reviewed scientific journal. Programs that meet these criteria can be assumed to have a strong THEORETICAL FOUNDATION, making them RESEARCH BASED. Thus, all evidence based programs are research-based, but not all research based programs are evidence based. Not all research based programs have shown evidence of effectiveness. Each organization requires different levels of “research” to make a program evidence based.

9. Society for Prevention Research (SPR)
The Society for Prevention Research is an organization dedicated to advancing scientific investigation on the etiology and prevention of social, physical and mental health, and academic problems and on the translation of that information to promote health and well being. The multi-disciplinary membership of SPR is international and includes scientists, practitioners, advocates, administrators, and policy makers who value the conduct and dissemination of prevention science worldwide.

10. SPR
Started in 1991
747 members, most with PhD after their name; Hawkins, Botvin, Sloboda, etc – you probably recognize these
Prevention Science Journal
Standards of Evidence published in Prevention Science Journal in 2004

11. Standards of Evidence Criteria for Efficacy, Effectiveness and Dissemination
Efficacy – the extent to which an intervention (technology, treatment, procedure, service, or program) does more good than harm when delivered under optimal conditions
Effectiveness –Effectiveness trials test whether interventions are effective under “real-world” conditions or
in “natural” settings.
Dissemination –can be appropriately used by providers (coalitions, prevention specialists, teachers, counselors, social workers, service agencies, etc.).

12. Efficacy Specificity of Efficacy statement
Intervention Description and Outcomes
Clarity of Causal Inference
Generalizability of Findings
Precision of Outcome
Specificity of Efficacy Statement - Efficacy is the extent to which an intervention (technology, treatment, procedure, service, or program) does more good than harm when delivered under optimal conditions (Flay, 1986; Last, 1988). Efficacy is distinguished
from effectiveness, which refers to program effects when delivered under more real-world conditions. Because outcome research results are specific to the samples (or populations from which they were drawn) and the outcomes measured, it is essential that conclusions from the research be clear as to the population(s) and outcomes for which efficacy is claimed. Therefore,
a statement of efficacy should be of the form that “Program X is efficacious for producing Y outcomes for Z population.” The remaining standards pertain specifically to the four areas of validity described by Cook and Campbell (1979): the description of the intervention and outcomes, the clarity of causal inferences, the generalizability of findings, and the precision of outcome measurement. Standards might change over time as methods develop and prevention science and practice advance.
Intervention Description and Outcomes –
Description - The intervention must be described at a level that would allow others to
implement/replicate it (this may require a more detailed description than what is presented in most research journals). An adequate description of an intervention includes a clear statement of the population for which it is intended and a description
of its content and organization, its duration, the amount of training required, etc.
Outcomes - The stated public health or behavioral outcome(s) of the intervention must be measured. For example, a measure of attitudes about violence cannot substitute for a “measure” of actual violent behavior.
ii. For outcomes that may decay over time, there must be at least one long-term
follow-up at an appropriate interval beyond the end of the intervention (e.g., at
least 6 months after the intervention, but the most appropriate interval may be
different for different kinds of interventions).
• It is also desirable, though not necessary, to include measures of proximal outcomes (i.e., mediators). The analysis of program effects on theoretical mediators is essential for establishing causal mechanism.
• It is desirable to measure implementation.
• It is desirable to measure potential side-effects or iatrogenic effects.
Outcomes must be measured properly (leave it to the egg-heads).
Clarity of Causal Inference –
Comparison Condition-The design must have at least one comparison condition that does not receive the
tested intervention.
Assignment- The assignment to conditions needs to be done in such a way as to maximize confidence that the intervention, rather than some other alternative explanation, causes the reported outcomes.
Generalizability of Findings-The report must specify what/who the sample is and how it was obtained. It needs to be clear how well the sample does or does not represent the intended population. This is an essential component of the efficacy statement (see #1). An intervention shown to be efficacious can claim to be so only for groups similar to the sample on which it was tested.
Precision of Outcomes-
-Statistical analysis must allow us to unambiguously establish the causal relations between the intervention and the outcomes (main effects).
-Statistically significant effects
-Practical value
-Duration of effect (at least 6 months)
-Replication (2 or more, preferably more)

13. Effectiveness Must meet all standards of Efficacy
Program or Strategy Description and Outcomes
Clarity of Causal Inference
Generalizability of Findings
Precision of Outcome
Must meet standards of efficacy
Program Description and Outcomes
Program Definition - Manuals and, as appropriate, training and technical support must be readily available.
Intervention Delivery- The intervention should be delivered under the same types of conditions as one would
expect in the real world (e.g., by teachers rather than research staff).
Theory- A clear theory of causal mechanisms should be stated.
ii. A clear statement of “for whom?” and “under what conditions?” the intervention is expected to be effective should be made.
Measures- Level of exposure should be measured, where appropriate, in both treatment and control conditions. Effectiveness trials generally have much more variation in these elements than efficacy trials; therefore the level of variation should be documented, as it affects ultimate program impact. Level of exposure is determined by two factors, both of which it is essential to measure:
Integrity and level of implementation/delivery of intervention.
ii. Acceptance/compliance/adherence/involvement of target audience and
subgroups of interest in the intervention activities.
Clarity of Causal Inference
The same standards as stated for efficacy apply, though the challenges are greater. Randomization is still the best approach, but the other alternatives suggested (regressiondiscontinuity, time-series, high quality matched controlled designs) may be used.
Generalizability of Findings
The real-world target population and the method for sampling it should be explained in order to make it as clear as possible how closely the sample represents the specified real-world target population. The degree to which findings are generalizable should be evaluated. One of the objectives of effectiveness studies is to establish for whom the intervention is effective.
Precision of Outcome
Practical value – must have it.
Replication- Consistent findings are required from at least two different high-quality trials that meet all of the above criteria and each of which has adequate statistical power. Effectiveness can be claimed only for those outcomes for which there are similar
effect sizes in all trials (which may be lower than the effect sizes found in efficacy studies). This reduces chance findings.

14. Broad Dissemination
1. To claim readiness for broad dissemination, a program must meet all of the criteria for effectiveness plus the following: 2. The program must have the ability to go to scale, including providing all program materials and necessary services (e.g., manual, training and technical support). 3. Clear cost information must be readily available. 4. Monitoring and evaluation tools must be available to providers.

15. Why does this matter?
Why the big deal about evidence-based lists? Do we understand what they mean?

16. National Registry of Evidence Based Programs and Practices (NREPP)
searchable online registry of mental health and substance abuse interventions that have been reviewed and rated by independent reviewers
NREPP ratings do not reflect an intervention's effectiveness. Users should carefully read the Key Findings sections in the intervention summary to understand the research results for each outcome.
You can use NREPP as you look for prevention strategies to address issues found in the community assessment. For the Civil Operator, this tool can help you suggest evidence-based programs to address specific issues in specific populations. Remember, just because it is on the list, does NOT make it an EFFECTIVE program for your community’s situation. It is important to understand what specific effect the program or strategy has, and on what population.

17. Coalition Strategies
Most coalition strategies focus on large populations. Many of these strategies are termed “Environmental”
Environmental strategies target the broader physical, social, cultural, and institutional forces that contribute to problem behaviors. These strategies are found in the outer layers (or levels) of the social-ecological model.
Iowa Department of Public Health (2010). Iowa Strategic Prevention Framework State Incentive Grant: Evidence-Based Practice Selection and Planning Workbook. Retrieved on January 21, 2012 from

18. Social Ecological Model
Environmental Strategies
Individual level: This level encompasses the knowledge, attitudes, and skills of the individuals within the target population. This level can be influenced by individual-level interventions (such as educational and skill-building programs) as well as county-wide media and social marketing campaigns. An example of an individual level intervention would be a 6-week program targeted at high-risk students to improve their self-confidence and teach them the skills needed for resisting alcohol and drug use.
Relationship level: This level includes the family, friends, and peers of the individuals within the target population. These persons have the ability to shape the behaviors of the individuals in the target population. This level can be influenced by enhancing social supports and social networks as well as changing group norms and rules. An example of a relationship level intervention would be an educational program targeted at parents of year olds to teach them how to better communicate with their children and establish rules around alcohol use.
Community/County level: This level includes the unique environments in which the individuals in the target population live and spend much of their time, such as schools, places of employment and worship, neighborhoods, sports teams, and volunteer groups. This level can be influenced by changes to rules, regulations, and policies within the different community organizations and structures. An example of a community level intervention would be the adoption of an “alcohol free” policy by a local company for all of their work-related functions and events.
Societal level: This level includes the larger, macro-level factors that influence the behaviors of the individuals in the target population, such as laws, policies, and social norms. This level can be influenced by changing state and local laws, policies, and practices, as well as other initiatives designed to change social norms among the target population as a whole, such as a media campaign. An example of a societal level intervention would be requiring mandatory beverage server training.

19. Evidence-Based Environmental Strategies
Four Guidelines (SAMHSA):
Guideline 1: The intervention is based on a theory of change that is documented in a clear logic or conceptual model; and
Guideline 2: The intervention is similar in content and structure to interventions that appear in registries and/or the peer-reviewed literature; and
Are you seeing any differences in these requirements and the requirements for programs from SPR or NREPP?

20. Evidence-Based Environmental Strategies
Guideline 3: The intervention is supported by documentation that it has been effectively implemented in the past, and multiple times, in a manner attentive to scientific standards of evidence and with results that show a consistent pattern of credible and positive effects; and
Guideline 4: The intervention is reviewed and deemed appropriate by a panel of informed prevention experts that includes: well-qualified prevention researchers who are experienced in evaluating prevention interventions similar to those under review; local prevention practitioners; and key community leaders as appropriate, e.g., officials from law enforcement and education sectors or elders within indigenous cultures.
SAMHSA (2009) . Identifying and Selecting Evidence-Based Interventions. Retrieved on January 22, 2012 from
Who are these people?
Answer: Scientists at SAMSHSA, CDC, Universities, etc.

21. SPF-SIG Definition of Evidence Based (according to SAMHSA)
The SPF SIG Program specifically requires implementation of evidence-based interventions.
Evidence-based interventions are defined in the SPF SIG Program by inclusion in one or more of the three categories below:
A. Included in Federal registries of evidence-based interventions;
B. Reported (with positive effects on the primary targeted outcome) in peer-reviewed journals; or 14 Identifying and Selecting Evidence-Based Interventions
C. Documented effectiveness supported by other sources of information and the consensus judgment of informed experts
See pages (19-25 adobe) after clicking on the link.

22. Criteria for Use
Not only does it need to be “Evidence-Based”; it needs Conceptual Fit!
Need to be practical within the needs and resources!
Must have evidence that applies to the logic model and community!
SAMHSA (2009) . Identifying and Selecting Evidence-Based Interventions. Retrieved on January 22, 2012 from

23. Evidence-Based Environmental Strategies
SAMHSA (2009) . Identifying and Selecting Evidence-Based Interventions. Retrieved on January 22, 2012 from
(SAMHSA, 2009)

24. Which are environmental?
SAMHSA (2009) . Identifying and Selecting Evidence-Based Interventions. Retrieved on January 22, 2012 from
(SAMHSA, 2009)

25. Evidence-based environmental strategies
In developing the logic model from the community assessment, there should be a logical progression from the issue or outcome to the intervening variable (the why) to the local condition (why here). An evidence based strategy should be selected that has been found through research to be effective FOR THAT LOCAL CONDITION

26. Evidence-based environmental strategies
See the Community Toolbox’s list of resources on websites that explain different evidence based strategies CTB EVIDENCE BASED RESOURCES
These are harder to define than “boxed” programs. Strategies that cannot be researched in a controlled setting make it difficult for researchers to prove effectiveness
Strategies are only evidence based if they address the problem (see next slide)

27. Evidence-based environmental strategies
Click on the link to review a great product for selecting environmental strategies from Maryland
The key is to DO YOUR HOMEWORK
Don’t take for granted that a strategy will work in your situation. Do your research
Don’t take for granted that because some says its evidence based, that it is evidence based or it applies to your situation
Be sure that your strategies match your conditions
Implement with FIDELITY!

28. Links to Good Evidence Based Practices Information
Iowa SPF SIG
Nebraska’s Strategy Approval Guide
SAMHSA’s Identifying and Selecting Evidence-based Interventions
SAMHSA’s list of Registries
Indiana’s planning Toolbox
Community Toolbox Databases of Best Practices
For the Iowa document, click on the link. When you arrive at the webpage, select the “ Environmental Strategies - Selection Guide, Reference List and Examples of Implementation Guidelines” document

29. Summary of Evidence-based
Definitions and criteria for Evidence Based change amongst organizations, individuals, and purpose
Programs and Strategies need to be scrutinized for what their evidence-based outcomes are
There are many lists, NREPP is not the only list
It is only evidence based if it being used with fidelity
Strategies should meet Logic Model needs…LOGICALLY!