Theory of Change and Evidence 1:00 PM – 2:45 PM Wednesday, August 21, 2013 Broadway A/B Presenters: Cheryl Leménager – Serve Wisconsin Sarah Hinzman – Volunteer Iowa Mary Cannon – Serve Ohio

AGENDA Welcome Pre-test TOC Break Evidence Q and A Post-test

Designing Effective Action for Change Welcome to the module, “Designing Effective Action for Change.” This module will describe how a Theory of Change helps you clarify the cause-and-effect relationship at the heart of your program. How a Theory of Change helps you clarify the cause-and-effect relationship at the heart of your program Copyright © 2012 by JBS International, Inc. Developed by JBS International for the Corporation for National & Community Service

Learning Objectives By the end of the module, you will be able to: Describe the benefits of a Theory of Change Define the three elements needed to construct a Theory of Change Identify how the Theory of Change informs the program design This module is designed to give you an overview of Theory of Change. By the end of the module, you will be able to: Describe the benefits of a Theory of Change Define the three elements needed to construct a Theory of Change Identify how the Theory of Change informs the program design

Theory of Change Elements Community Problem/Need Specific Intervention Intended Outcome While the term theory of change may be new to you, the elements are not new to National Service programs. National Service Programs have addressed questions related to these elements in developing and managing strong programs. You might think of this as a set of organizing principles. A theory of change has three main elements: There is the community problem or need to be addressed. There is the intended outcome – if you deliver the intervention according to plan, it will bring about a measurable positive change in the community in relation to your targeted need. There is a specific intervention, a set of activities that you have chosen to address the need based on evidence that a similar approach has worked in the past. As you develop a theory of change for an issue that your program will address, you identify where you are now, as well as the change for which you are programming. You will choose certain activities based upon their link to your desired outcomes.

Theory of Change Perspective the most effective set of activities for volunteers and participants Community Problem/Need Specific Intervention Intended Outcome Developing a theory of change requires you to articulate assumptions underlying your choice of activities by documenting why you think the activities will lead to the intended change. Looks at cause and effect relationships Adapted from www.theoryofchange.org/about/

Theory of Change Perspective the most effective set of activities for volunteers and participants Community Problem/Need Specific Intervention Intended Outcome Identifies specific interventions to achieve the desired result Adapted from www.theoryofchange.org/about/

Theory of Change Perspective the most effective set of activities for volunteers and participants Community Problem/Need Specific Intervention Intended Outcome Clarifying your theory of change is also critical for existing programs as it offers you an opportunity to compare and contrast your program’s approaches and context to similar programs and allows you to consider where to make changes to strengthen your program’s outcomes. Constructing a theory of change also offers you the opportunity to revisit your choice of outcome(s) to insure there is a logical and practical relationship between the service you provide (your intervention) and the change you intend to make. Shifts thinking from “what we are doing” to focus on “what we want to achieve” Adapted from www.theoryofchange.org/about/

Theory of Change Elements Community Problem/Need Specific Intervention Intended Outcome Evidence Guides choice of intervention Supports cause-effect relationship Statistics documenting the need Underlying elements to main elements are: Data that documents the extent and severity of the community problem/need you will address; and Evidence that informs your understanding of why a particular intervention will result in the change identified in the outcome. [Note: Evidence will be covered in more depth in the next module.]

Everyday Life Example I have strep throat (problem) I will take antibiotics (intervention) I will get better (outcome) Let’s walk through an example of what a theory of change looks in everyday life. Science provides us with some of the most familiar examples of how theories of change show us a pathway from problem to solution. If you have strep throat, then the intended outcome is wellness. Taking antibiotics is a way to get there. Strep throat Antibiotics Be Healthy Evidence Guides choice of intervention Supports cause-effect relationship

Everyday Life Example Evidence:  Guides choice of intervention  Supports cause-effect relationship But every antibiotic is not equally effective with strep, so you need to consider the evidence from past studies to determine the best possible intervention or which antibiotics are best to fight strep. Penicillin has been proven very effective in fighting strep when taken a certain number of times a day for a period of time – making it a good choice. Tetracycline is also a powerful antibiotic – but it isn’t effective for strep throat. It has proven to help with acne. So it would not be a good choice of intervention. You take 500 mg of penicillin twice a day for 10 days – and you get well. But which antibiotic(s) fight strep the best? (Look at evidence to make the choice)

Example: Riverton Literacy Corps Community Problem/need Children reading below grade level in 3rd grade Now let’s look at an example for a national service literacy tutoring program. The community problem is that third graders at Riverton Elementary are not reading at grade level as demonstrated by high failure rates on the state reading exam. This is supported by data documenting the failure rates of Riverton students on the state exam and research on the importance of mastering reading by the end of the third grade for future academic success. Statistics on the number of students at below grade level in program’s service area; Research on why reading proficiency by 3rd grade is important.

Example: Riverton Literacy Corps Community Problem/need Intended Outcome Children reading below grade level in 3rd grade Students are able to read at 3rd grade level (as measured by 3rd grade reading exam) The intended outcome sought by the program is for students to be able to read at third grade level as demonstrated by passing the state reading exam. Statistics on the number of students at below grade level in program’s service area; Research on why reading proficiency by 3rd grade is important.

Example: Riverton Literacy Corps Community Problem/need Specific Intervention Intended Outcome Evidence: Research on building block skills leading to reading proficiency. Research on design, frequency, duration of tutoring sessions. Children reading below grade level in 3rd grade Individualized tutoring 3 times/week for 20 min on five “building block” literacy skills through reading, writing and verbal communication activities Students are able to read at 3rd grade level (as measured by 3rd grade reading exam) The intervention will provide individualized tutoring to students identified as “at risk”. This tutoring will focus on five “building block” literacy skills through integrated reading, writing and verbal communication activities. Research has shown this curriculum model to be linked to demonstrable improvement in reading skills. The research outlines the specifics of the intervention that yielded the intended outcome. Statistics on the number of students at below grade level in program’s service area; Research on why reading proficiency by 3rd grade is important.

Theory of Change Elements Community Problem/Need is the specific issue your project, with its specific intervention (service activity), is designed to address. What is the extent and severity of this need in the community? Community Problem/Need is the specific issue your project, with its specific intervention (service activity), is designed to address. What is the extent and severity of this need in the community? Now let’s go into a bit more detail for each element using a different example. Community problem is the negative condition that exists; community need describes the extent/severity of the problem in your community. A community has a number of teens that are overweight. You would ask questions such as what is the problem; i.e. what does the research say about the health effects of being overweight? How extensive is the problem in the community; i.e. 5%, 15%, 25% of the teens? And what does this mean to our community? The description of the problem as well as the specific community need is supported by statistics, facts and figures from current and reputable sources. As you read about and study the problem, you gain an understanding of it which will help lead to your chosen intervention.; i.e., do health classes cover nutrition and fitness? Does the school offer healthy eating options? Does the community offer physical activity programs (team sports) in which teens could become involved? How is the problem described and analyzed in the literature and in conversations? What has been tried, whether or not it has worked? It is useful to know what doesn’t work as well as what does. Community Problem/Need Specific Intervention Intended Outcome Evidence Guides choice of intervention Supports cause-effect relationship Statistics documenting the problem/need

Theory of Change Elements Statistics documenting the problem/need The best data: Come from reputable primary sources, such as government agencies, institutes, foundations, and universities that have conducted their own research. News reports are NOT primary sources. Data from multiple (reputable) sources increases reliability. Are as current (up to date) as possible. Are local; national-level and state-level statistics don’t tell us about the problem as it affects the community where your intervention will be implemented. If you operate in multiple communities, be sure your data describe these communities. Multi-site programs or programs covering large geographic areas, may rely on county or statewide data. It needs to be clear why you selected particular communities for your intervention. The goal here is to find and use data as close to the source as possible.

Community Problem/Need Data documenting problem/need should answer these questions: Scope: Who and how many are directly affected? How severe is this? significance: What makes this a compelling need? Is it likely to become worse? What will happen if we do nothing? Cause(s): Why does the need exist? How is it perpetuated? Data describing the community problem/need should address three broad areas. Scope: The extent of the issue in your local community. What is the severity/intensity of the need for those affected by it? For example, how many individuals or families are directly affected by the need, either in absolute terms or as a percentage of the entire community? Significance: This is the “so what” question. Why we ought to care about this particular need? This includes data telling us how the need affects the community more generally, and how this is connected to other community issues. Sharing data on trends or the likelihood that the need will worsen also speaks to significance. Causes: Why the need exists and how it came about , such as historical data showing how the need started and how long it has been around. Data on the scope and relevance pertain to describing the community need. Data on the causes of a problem will also inform your choice of intervention or service activity.

Documenting Community Need Healthy Futures Program Example Scope: National Survey of Children and Health found childhood obesity in State X increased 23% between 2003 and 2007 – the 2nd fastest rate of increase in US (2010). CDC data show nearly one-third of children and teens are obese or overweight in Webb County (2009). Significance: Obese children found to become obese adults Links between childhood obesity and early onset of cardiovascular disease, and Type II diabetes Webb County rising in state and national averages for cardiovascular disease Long term health care costs rising Causes: Diet, sedentary lifestyle If a health education program is aimed at reducing rates of obesity, then data from reputable sources showing that obesity is a growing problem in the local community should be provided. These data should inform the reader about the extent to which the community is affected by the problem (scope), why it matters (significance), and why it exists (causes).

Intended Outcome What change are you hoping to make related to the identified need? Community Need Specific Intervention Intended Outcome Evidence Guides choice of intervention Supports cause-effect relationship Statistics documenting the need What change do you want your program to make? Intended Outcome: After you identify the problem/need, the next step is to consider what change you want your program to make. What is your intended outcome, your desired result?

Identifying the Intended Outcome Healthy Futures Program Example Possible outcomes to measure: Increased knowledge of what is “healthy food” More frequent choice of healthy foods to eat More frequent involvement in physical activity or exercise Improved physical condition Let’s see what an intended outcome might look like for the Healthy Lifestyles program example. There are several possible intended outcomes to measure: Increased knowledge of what is “healthy food” More frequent choice of healthy foods to eat More frequent involvement in physical activity or exercise Improved physical condition Based on the need, this program selected “Improved physical condition” as its intended outcome.

Intervention An intervention is the specific set of activities in which participants and volunteers will be engaged. What is the best way to achieve the intended outcome? Once you know what change you intend to make (your outcome), then you can determine the most effective way to intervene. An intervention is the specific set of activities in which your participants and volunteers will be engaged. It is the strategy you employ to bring about the change for those who will benefit from the services. You are approaching this step with an understanding of the scope, significance and causes of the problem/need and an identified outcome or desired result. To determine what could be an effective intervention, you will look for evidence from performance measurement, evaluations, or research studies that show your proposed intervention is likely to achieve your intended outcome. What interventions have others tried? Have they proven successful, or have they not led to the expected outcome? Community Need Specific Intervention Intended Outcome Evidence Guides choice of intervention Supports cause-effect relationship Statistics documenting the need

Intervention An intervention is the specific set of activities in which participants and volunteers will be engaged. What is the best way to achieve the intended outcome? Here’s an example of how evidence is useful when designing your intervention or modifying an existing intervention to make it more effective. If evidence from an evaluation shows that some specific number of hours of physical activity for teen girls led to improved physical condition and health . Then your intervention should offer a similar number of hours of physical activity if you are seeking to improve physical condition and health in teen girls your program serves. It is important to note here that physical activity is a broad category of intervention in the same way that antibiotics from our earlier example, are a broad category. Therefore, as you review evidence of what has worked, you also need to pay attention to both the design of the intervention and the dosage received by the teen girls in the program that was studied. For more detailed information on evidence – what it is, how it helps you select an intervention, where to find it, etc., please see the Evidence Module. Community Need Specific Intervention Intended Outcome Evidence Guides choice of intervention Supports cause-effect relationship Statistics documenting the need

Intervention Describe the design and dosage of your intervention (service activity): Design (who does what with whom?) Dosage Frequency (how many sessions a week?) Intensity (length of each session) Duration (how many total weeks of sessions?) As part of your theory of change, you need to describe your intervention or service activities clearly. You need to identify the: Design (who does what with whom?) Dosage Frequency (for example, how many sessions a week?) Intensity (for example, length of each session) Duration (for example, how many total weeks of sessions?)

Intervention Example Healthy Futures Program Example Design: national service participants implement the Shape Up curriculum with economically disadvantaged urban girls ages 14-16 to increase physical activity (30 minutes/session) and educate them on healthy eating Frequency: twice a week afterschool Intensity: 60 minutes per session Duration: 12 weeks Following our Healthy Futures program example, here is the intervention description for the after-school community-based Shape Up program: Design: the Shape Up curriculum serves small groups of 3-5 economically disadvantaged urban girls ages 14-16 to increase physical activity and educate them on healthy eating Frequency: twice a week Intensity: 60 minutes per session Duration: 12 weeks

Testing Your Theory of Change: Is Your Theory of change: Plausible: Does the logic of the model seem correct: “if we do these things, will we get the results we expect?” FEASIBLE: Are resources sufficient to implement the chosen intervention? Meaningful: Are intended outcomes important? Is the magnitude of expected change worth the effort? After you have laid out all three elements of your theory of change, you need to ask yourself the following questions. Is your Theory of Change Plausible: Does the logic of your model seem correct: “if we do these things, we will get the results we expect?” Is it Feasible: Do you have enough resources to implement the intervention you have chosen? Is it Meaningful: Are your intended outcomes important? Is the magnitude of change you expect worth the effort?

Summary of Key Points A theory of change identifies cause/effect The three elements of a theory of change; community problem/need, intervention, intended outcome are supported by data and evidence Data documenting community need should show scope, significance, and causes To summarize the key points of this module: Having a well conceived theory of change helps identify the pathway linking your service activities to your intended outcome The three elements of a theory of change; community problem/need, intervention, intended outcome are supported by data and evidence Data documenting community need should show scope, significance, and causes

Additional Resources CNCS Priorities and Performance Measures: http://www.nationalserviceresources.gov/npm/home Program-Specific Notices of Funding Opportunities and Application Instructions: http://www.nationalservice.gov This concludes the Theory of Change Module. For more information on CNCS national performance measures, go to http://www.nationalserviceresources.gov/npm/home. Also see http://www.nationalservice.gov for more information on Program-specific notices of funding opportunities and application instructions.

Group exercise Case Study of Rose Blossom ABC Club Answer: Question 2

Evidence: What It Is and Where to Find It Welcome to the module, “Evidence: What It Is and Where to Find It.” This module will describe how using evidence can help you select an effective intervention. How evidence helps support your intervention Copyright © 2012 by JBS International, Inc. Developed by JBS International for the Corporation for National & Community Service

Learning Objectives By the end of the module, you will be able to: Describe evidence-based interventions Identify the approved sources of evidence Find evidence for your intervention Assess the utility of evidence you have located Use evidence for your intervention This module will provide detailed information on evidence and evidence-based interventions. By the end of the module, you will be able to describe evidence-based interventions; identify the approved sources of evidence; find evidence for your intervention; assess the utility of evidence you have located; and use evidence for your intervention.

Module Overview What is evidence? What are evidence-based interventions? What are sources of evidence? What are methods to assess evidence? This module will answer a number of questions: What is evidence? What are evidence-based interventions? What are sources of evidence? What are methods to assess evidence?

Theory of Change Elements Community Need Specific Intervention Intended Outcome Evidence Guides choice of intervention Supports cause-effect relationship Statistics documenting the need As you will remember from the Theory of Change module, an intervention must address the identified community problem or need and be able to generate the intended outcome. The Corporation for National and Community Service (CNCS) emphasizes that all national service programs should develop and implement evidence-based interventions to insure that scarce resources create the most positive change possible in local communities. Note that “statistics documenting the need” are completely separate from “evidence”. Evidence supports why you chose your intervention (what the AmeriCorps members will do to decrease the need). Evidence does not support why there is a need itself.

Evidence-Based Intervention Service activities that are supported by research to demonstrate a high likelihood that these activities will address the need, and result in intended changes. Evidence demonstrates that the proposed intervention (design, dosage) is likely to lead to the outcome. Evidence-based intervention is a term used by CNCS to describe service activities that are supported by research, indicating a high likelihood that these activities will address the need, and result in intended changes. In other words, evidence should show the intervention is likely to work. Non-evidence-based interventions are those that have no track record of data and it is unknown whether they can produce the desired change.

Everyday Life Example of a Theory of Change Problem/Need Strep Throat Intervention Antibiotics Outcome Healthy Evidence  Penicillin  Amoxicillin  Tetracycline Let’s revisit the example of a theory of change from everyday life. If you have strep throat, the goal or outcome you want is to be healthy, that is, to get rid of the strep throat. Taking antibiotics is a great way to do that. But every antibiotic is not equally effective with strep. Doctors know which antibiotics are most effective from past research and experience so they use that information to prescribe a course of action, in this case which antibiotic will be the best in fighting strep. Penicillin and amoxicillin both are evidence-based interventions that have been proven very effective in fighting strep when taken a certain number of times a day for a period of time – making either of these a good choice. Tetracycline is also an antibiotic, but it isn’t effective for strep throat, though it will help with acne. So it would NOT be a good choice of intervention. Looking at the evidence helps avoid trial and error using scarce resources.

Evidence EVIDENCE: Information that supports your choice of a specific intervention and its ability to produce your intended outcome. It is NOT enough to just say “we believe” our intervention is “likely” to be successful. You need to justify your choice with evidence. Evidence provides a sound basis to choose an intervention to meet a specific community need. Once you know what change you intend to make (your intended outcome), then evidence helps you determine the best intervention to use to make it happen. It is not enough to just say “we believe” an intervention is “likely” to be successful. You need to justify an intervention with evidence. While “unproven” strategies or interventions may be valid, a lack of evidence makes it harder to know if it’s a good investment of resources. CNCS is choosing to invest in proven strategies.

Lessons from the Evidence Evidence provides a reality check for theory of change elements For new programs: What is the recommended design (specific program activities) and dosage (frequency, intensity, and duration) to achieve an intended outcome? For existing programs: Is there sufficient evidence for the intervention to continue its use? Based on the evidence, are there modifications to the intervention that should be considered to have greater impact? Do you need to choose a new intervention? Evidence provides a reality check for your theory of change. Therefore, all programs need to find and use evidence. Some of new are planning or implementing a new programs. In this case, evidence helps you choose an appropriate intervention design (the specific activities provided) and dosage (frequency, intensity, and duration) based on your intended outcome. Others of you are from existing programs and are either trying to justify an intervention you have already been using or make modifications to an intervention to improve its effectiveness. As you investigate the evidence for your intervention, some of you may find you will need to choose an entirely new intervention. The information about evidence presented in this module – where to find, assess, and use it – is equally applicable no matter which scenario best fits your program.

Knowledge check! C

You should look for evidence that program models or interventions like yours have been successful. CNCS has identified four possible sources of evidence that can help you make the case that your intervention is likely to achieve the desired result. The various National Service Programs may have different requirements concerning the use of evidence so be sure to check the appropriate notices of funding opportunities for details. The four sources of evidence that CNCS has identified are: Your past performance measurement outcome data; Results from an impact evaluation of your program; Research that documents the outcomes of similar programs; Impact evaluations that document outcomes of similar programs. Any of these sources that collected data using rigorous methodologies can be used. However, it makes for an even stronger case if you have more than one type of evidence to present that show similar findings. Now let’s look at each of these sources in more detail.

Evidence Source: Your Performance Measurement Data One source of evidence is your own past performance measurement data on outcomes. Past outcome findings from performance measurement do not show causality but they do provide data about whether or not a change occurred. In other words, they show only that a change occurred but they do not explain why. Therefore, while outcome data from performance measurement can't show that the change occurred because of your specific intervention, it is useful because it shows the intended result was achieved. If you are looking at your past performance data as evidence for an intervention, consider: What do your data tell you? For example, did you achieve your intended outcome? Do you have multiple years of outcome data you can aggregate? Using data from multiple years makes a stronger case for the effectiveness of your intervention. If you have it, use it!

Evidence Source: Your Program Evaluation Results from impact evaluation of your program: Is it an impact evaluation? Does it document change? Does it clearly show that your intervention is what caused the change? Another source of evidence can be the results of an impact evaluation done on your project or program: Impact evaluation uses rigorous methodologies in order to prove that the intervention caused the outcome. An impact evaluation tries to draw a connection between the cause and effect relationship between your intervention and the change that occurred. So the key questions to ask are, if you have an impact evaluation, does it document change? And, does the evaluation clearly show that your intervention is what caused the change? A process evaluation, on the other hand, looks at how your program was implemented and maybe at benefits it provided sites or beneficiaries, but it does not assess outcomes or causality. So as evidence to support your intervention, a process evaluation does not help – it needs to have been an impact evaluation.

Evidence Source: Research Studies and Evaluations Research studies: Focused on increasing knowledge or understanding of a particular group, problem or issue Impact Evaluations: Look at intervention success and outcomes produced The first two sources of evidence we have just discussed (blue circles) come from your programs. The next two sources to be covered (red circles) are research studies and impact evaluations from other programs. Research studies tend to focus on increasing knowledge or understanding of a particular group, problem or issue. Impact evaluations look at how well specific programs worked and what outcomes were produced by an intervention. It is important to remember that these two sources provide evidence about the interventions used by other organizations. Therefore you need to assess whether certain criteria are met – before you use it.

Assessing Evidence Criteria include: Relevant: Cites comparable intervention with similar beneficiaries and results Compelling: Persuasive, shows clear likelihood of success Up-to-date: Recently published or most recent available Objective: Source with no stake in outcome and published in a peer reviewed journal or by credible organization You need to be sure that the intervention that was studied in the article or report is relevant. Was the intervention similar in design, dosage, the type of beneficiaries served, and outcome achieved? If not, it may not be relevant enough to use as evidence for your Theory of Change. Evidence that is not relevant would not refer to the same population or may not be a comparable intervention (for example, participants receive intensive ongoing case management services as opposed to a one-time assessment and referral session). Is the evidence compelling? Is it persuasive and does it indicate the intervention produced a significant result? Is the evidence up-to-date? That is, what it published recently and has it utilized current thinking around the issues? There is no specific recommendation for what is considered “recent.” The intent here is to find timely information that is as recent as possible so it will reflect the latest thinking about what works. Is the evidence objective? Is it from a source that had no stake in insuring a positive outcome and was published in a peer reviewed journal or as a research study from a credible organization? Peer reviewed journal articles are great sources of information because they have been screened and selected based on their merit by other professionals in that field.

Evidence Source: Research Studies and Evaluations Sources of Strong Evidence University or research organizations (national or local) Known professionals/ experts in the field Similar sounding programs/ descriptions Articles that review multiple studies (meta studies) When looking for evidence from other organizations, be on the look out for articles or reports from a university, research organizations, or known professionals or experts in the field of interest. Also be on the lookout for articles about similar sounding programs or that review multiple studies, also known as “meta-studies”

Knowledge check! A, B, E

Web Search Tool Google Scholar is one web search tool that helps narrow the search for your keywords to professional journals http://scholar.google.com Tip: Use quotation marks around key terms - may help narrow the number of search results Now let’s talk about where to look to find the evidence from other programs. A first step in looking for evidence is doing a web-based search. One option is to use Google’s Scholar tool at http://scholar.google.com Google Scholar is a great shortcut to finding articles from peer reviewed, professional journals on high quality research studies and impact evaluation reports, without having to surf through many general documents that won’t help you find the evidence you are looking for. You may need to try a few variations on search terms to see what yields the most useful articles. The more specific you are when choosing your terms, not only will you get better results, you will save time. For example, using “afterschool obesity prevention program evaluations” as your search term will yield fewer and more relevant results than using more general terms like “girls health programs.” One tip is using quotation marks around your primary search terms as this also helps narrow your search. You may want to try it both ways – with and without quotation marks – to see which yields the best list of articles. Access to research librarian at college or local library could also be very helpful. Not all publications are available via Google Scholar. To help save time with your evidence search, review the introduction and conclusion before deciding if the rest is worth your time.

Search Result Questions Next, as you look through the search results, if a listing looks promising follow the link to the article. Use the criteria discussed earlier to see if the study suits your needs. For example, when looking for evidence to support an afterschool obesity prevention intervention for girls, here’s how the criteria would be applied. First, does the article about an evaluation article appear relevant? As potential evidence for an afterschool obesity prevention program, this article does appear relevant at first glance as the intervention, beneficiaries, and results seem similar. Is it compelling and does it indicate a significant result? Get a clear understanding of the results they got before choosing the article as evidence. Is the evidence up-to-date? The article is fairly recent – from 2003. However something more recent might be better, but this article is worth reviewing anyway. Does the evidence appear objective? It does because the evaluation was done by the University of Minnesota, a large state university with a specialized School of Public Health. The article authors were external evaluators – they were not involved in running the program and had no stake in the results. So, based on these four criteria, this article seems like it could provide useful evidence. You will likely find several studies which meet these criteria. But, before choosing one to use as evidence, take a careful look at the intervention specifics – the design and dosage – as well as the results achieved, and how that aligns with your theory of change. Is the article: Relevant? Compelling? Up-to-date? Objective?

Knowledge Check! Relevant = B, Up-to-date = A, Compelling = D; Objective source = C

Evidence Continuum Causality Low High Preliminary Moderate Strong Doesn’t show causality Outcome results from performance measurement Show causality, compares intervention recipients to non-recipients Results from studies/impact evaluations Comparison groups: Quasi-experimental Design Randomly-assigned control groups: Experimental Design Another consideration when reviewing evidence is where it lies along the CNCS evidence continuum (low quality to high quality evidence). The primary distinction between preliminary evidence and moderate or strong evidence relates to causality. If the evidence shows the intervention caused the outcome it is considered stronger. Performance measurement outcome results can only show that a change occurred and that the intended outcome was achieved. Performance measurement is not able to prove that the intervention caused the change. Therefore, outcome results are considered by CNCS to provide “preliminary” evidence. Impact evaluations and research studies use rigorous data collection methodologies that go beyond just documenting that a change occurred to determine causality. They are designed to collect data from people that received the intervention, as well as from similar people that did not. This allows a comparison between the two groups and sheds light on causality. In some cases, researchers and evaluators are able to randomly assign people to one of the two groups – those that receive the series and those that don’t. This is called an experimental design and is considered top of the line among data collection methodologies. It is highly rigorous and can produce very high quality data documenting that the intervention caused the outcome. Studies using random assignment are considered strong evidence. However, when unable to assign people randomly to groups, researchers often use comparison groups – groups of people similar to those receiving the service – as a way to get at causality. This is called a “quasi-experimental design.” Use of comparison groups is a rigorous data collection methodology, but because such studies don’t use random assignment, they are considered moderate evidence on the continuum.

Evidence Continuum Shape Up: afterschool obesity prevention program Preliminary Moderate Strong Performance measurement shows that 75% of girls age 14-16 participating in the Shape Up program made more healthy food choices A 2005 impact evaluation by internal evaluators (using a quasi-experimental design of girls participating in a similar program to Shape Up) found that after 12 weeks, the girls in the program made 50% more healthy food choices than the comparison group. A 2010 impact evaluation of the GEMS program by University of MN using experimental design/ random assignment found after 12 weeks, the girls in the experimental group made 50% more healthy food choices than control group girls Here’s an example of what evidence might look like along the continuum for an afterschool obesity prevention program called Shape Up. Preliminary evidence is from the program’s own past performance measurement findings: the program found that 75% of girls participating in the Shape Up program made more healthy food choices. This evidence is considered preliminary because it relies on performance measurement data on outcomes so while it shows a change occurred we can’t be sure that change happened because of the intervention. At a moderate level along the evidence continuum, you might use something like this: A 2005 impact evaluation using a quasi-experimental design of girls participating in a similar program to Shape Up, found that after 12 weeks, the girls in the program made 50% more healthy food choices than the comparison group. This is considered moderate because it used a quasi-experimental design which means girls were not put into groups randomly. So the data are considered less rigorous than if random assignment had been used. Strong evidence might look like this: A 2010 impact evaluation of a similar program by University of Minnesota using experimental design with random assignment found after 12 weeks, the girls in the experimental group made 50% more healthy food choices than the girls in the control group. This evidence is considered to be on the strong end of the continuum because it is used random assignment or an experimental design. The goal is to find the strongest evidence possible to justify your intervention choice. However, sometimes even after spending the necessary time to do online searches and read articles, you will only be able to access preliminary evidence for your theory of change. In that case, use whatever strength evidence you were able to find to show the evidence-basis for your intervention.

Knowledge Check! Preliminary = C, Moderate = B, Strong = A

Key Points Summary The term “evidence-based intervention” describes service activities that are supported by research to indicate a high likelihood that the intervention will address the need and result in intended outcomes. Evidence is information or documentation that a specific intervention (design and dosage) will produce your intended outcome. Here is a summary of the key points for this module: The term “evidence-based intervention” describes service activities that are supported by research to indicate a high likelihood that the intervention will address the need and result in intended outcomes. Evidence is information or documentation that a specific intervention (design and dosage) will produce your intended outcome.

Key Points Summary The four sources of evidence that CNCS has identified include: Past performance measurement data Results from a program impact evaluation Research studies that document the outcomes of similar programs Evaluations that document outcomes of similar programs When assessing evidence, check to be sure it is relevant, compelling, up-to-date, and objective. The strongest evidence clearly proves that the intervention caused the positive change by using an experimental design. The four sources of evidence that CNCS has identified include: Past performance measurement data Results from a program impact evaluation Research studies that document the outcomes of similar programs Evaluations that document outcomes of similar programs When assessing evidence, check to be sure it is relevant, compelling, up-to-date, and objective. The strongest evidence clearly proves that the intervention caused the positive change by using an experimental design.

Group exercise 20 minutes including debrief! Case Study of the Rose Blossom ABC Club Answer: Question 3 Question 6

Final Concerns & Questions Plus post test! 

Resources Additional Resources CNCS Priorities and Performance Measures: http://www.nationalservice.gov/resources/npm/home Program-Specific Notices of Funding Opportunities and Application Instructions: http://www.nationalservice.gov This concludes the Evidence Module. For more information on CNCS national performance measures, go to http://www.nationalservice.gov/resources/npm/home. Also see http://www.nationalservice.gov for more information on Program-specific notices of funding opportunities and application instructions.