PFS Evaluation Presented by Alaska PFS State Evaluators: Bridget Hanson and Jodi Barnett

WELCOME! WHO’S JOINING US? Let’s introduce ourselves Name Community Project role Please use the chat box

Meeting Guidelines We encourage participation! Share your comments, questions, ideas. Be positive, be problem solvers. Keep chat box comments on-topic.

“One of the great mistakes is to judge policies and programs by their intentions rather than their results” Milton Friedman

Webinar Objectives Alaska PFS staff will understand: project evaluation expectations importance of baseline data collection the difference between the federally required versus Alaska PFS evaluation

Your Turn (Raise Your Hand) How many of you have read the evaluation section of the guidance document in the past month?

Alaska PFS timeline Strategic plans are due on June 30th which will include an evaluation plan for approved strategies Step 3  Vision statement  Capacity building plan  Community logic model  Strategy action plans  Evaluation plan

Key Elements of the Alaska PFS Project Data driven All 5 Steps of the SPF process are driven by data – The process is designed to be Purposeful. In the evaluation step (step 5), data collected will be used to determine if mid-course changes to strategies are necessary and if they are having the anticipated effect on community factors.

Ultimate goal is to prevent/reduce Key Elements of the Alaska PFS Project Ultimate goal is to prevent/reduce non-medical use of prescription opioids (12-25 year olds) use of heroin (18-25 year olds) Prescription opioids is the priority substance - PFS grants were required to address alcohol or prescription drugs Heroin is an addition included by the Alaska PFS as we know the substances are related Change in consumption is unlikely to occur from this grant alone over the next 2-3 years. However the more targeted our efforts and the efforts of others working on this issue, the more likely it is that we will see a change in consumption over time. Strategy evaluation data related to community factors can help to show whether PFS strategies are moving communities towards the goals of decreasing NMUPO and use of heroin.

Reminder: Federal vs Alaska PFS evaluation Community Level Instrument (CLI-R) is a federal evaluation requirement Strategy evaluation efforts (using time series graphs) are specific to the Alaska PFS project Data collected and reported for the federal evaluation requirements may be helpful but they are designed to be most useful for SAMHSA in order to answer national evaluation questions. Data you collect for strategy evaluation efforts will be most informative and useful for your project.

Community-Specific Logic Model Elements Community Strategies  “What strategies will we choose to address the community factors” Community Factors “Why does Rx opioid misuse/abuse occur specifically in our community?” Social Availability Easy access to Rx opioids through social sources   Retail Availability Easy access to Rx opioids through providers & dispensers Perceived Risk of Harm from Rx opioid misuse or heroin use Community factors were a primary focus during the data assessment process and will be the primary focus of evaluation efforts as well. The more targeted the community factors are, the easier it will be to choose strategies and evaluate your efforts.

Your Turn! (Chat Box) Describe 1 or 2 community factor(s) that rose to the top in your data assessment? Please use the chat box

Alaska PFS Evaluation Planning Step 1: Finalize Community Logic Models snapshot of what the project is trying to accomplish communication tool help guide and focus planning efforts

Alaska PFS Logic Model Components Community Strategies   “What strategies will we choose to address the community factors” Community Factors “Why specifically does Rx opioid misuse/abuse occur specifically in our community?” Intervening Variables “Why does Rx opioid misuse/abuse occur?” Consumption “Rx opioid misuse/abuse among 12-25 year olds” Consequences “Consequences of Rx opioid misuse / abuse (i.e. Why do we care?)” Direction of influence once implementation begins Alaska PFS Logic Model Components  If-then statements

Example Community Logic Model State-level evaluators at CBHRS have identified indicators and will collect data for intervening variables, consumption, and consequences related to NMUPO and heroin use at the state and funded community levels. This data will be provided to communities in order to monitor outcomes. Communities will focus on local data related to community factors for the PFS outcome evaluation. Overview of your community approach to reduce Rx opioid misuse and heroin use A good time to use “if-then” statements to test your logic

Your Turn! (Raise Your Hand) Has your coalition discussed any potential strategies to target community factors?

(for time series graphs) Alaska PFS Evaluation Planning Step 2: Develop Strategy Evaluation Plans (for time series graphs) Identify outcomes, indicators, and measures to answer: How much are you doing? How many are you reaching? Is change on community factors occurring? A time series graph can be used use to visually evaluate patterns in data collected over time.

Intervening Variable: Strategy Target Population: STRATEGY: multi-level campaign to increase knowledge about HARMs of social access and ways to prevent social access to rx opioids Community Factor: Lack of community knowledge among adults about harms associated with easy access to Rx opioids Intervening Variable: Social Availability of Rx opioids CSAP Category: Information Dissemination Strategy Target Population: Adults 18+ Community Factor: Lack of knowledge among adults about how to prevent social access to Rx opioids KEY strategy outcomes Indicators method / measure Multi-level campaign to:   Increase knowledge about harms associated with easy access to Rx opioids Increase knowledge about ways to reduce social access to Rx opioids Increase safe storage, monitoring, and disposal of Rx opioids # and reach of awareness products disseminated (posters, articles, etc) (P) % of adults who have seen campaign messages about safe storage and disposal of Rx opioids (P) % of adults reporting safe storage and monitoring of Rx opioid prescriptions (or intentions) (O) % of adults who report disposing of Rx opioids safely (or intentions) (O) Dissemination tracking Quarterly random survey of 100 adults (e.g., DMV) Strategies that are focused on well-defined community factors will greatly facilitate this process. Strategies that do not align with community factors will not work well. A good way to determine if the logic works is to create IF-THEN statements and ask yourself whether they are true. Example 1 Pick an indicator or indicators that best reflect the process and outcome indicators that are most important to you. An indicator is anything that is measurable that can be used to identify a change in trends. The indicator needs to be relevant: it should tell you what you need to know about your strategy and expected outcomes. Find measurements on those indicators. Once you've chosen indicators, decide exactly what you're going to measure, and for how long. Ask yourself : Is this measure showing what it's supposed to measure and not some by-product? (O) Outcome indicators; (P) Process indicators

STRATEGies: 1) increase the Number of safe disposal sites and 2) Implement A campaign to increase awareness and use of safe disposal sites Community Factor: Lack of convenient and/or recognized sites for adult community members to dispose of Rx opioids safely Intervening Variable: Social Availability of Rx opioids CSAP Category Strategy Target Population: Adults 18+ Create safe disposal sites: Environmental Campaign to promote safe disposal sites: Information Dissemination KEY strategy outcomes Indicators method / measure Increase safe disposal sites to:   Increase access to safe disposal sites Multi-level campaign to: Increase awareness of safe disposal sites Increase use of safe disposal sites # of disposal sites created and # of days open (P) # and reach of awareness products disseminated (posters, radio ads, etc.) (P) % of adults aware of at least one local safe disposal site (O) # visitors of visitors and/or pounds of Rx opioid pills discarded at disposal sites (O) Activity tracking Dissemination tracking Quarterly random survey of 100 adults (e.g., DMV) Example 2 Most strategies will address one community factor, but sometimes one strategy may address two community factors or two strategies will address one community factor. Strategy Evaluation Templates, which will be provided on the PFS website, can be adapted accordingly to develop your evaluation plans. The DETAL is available to provide further training and technical assistance as you complete the Strategy Evaluation Templates. (O) Outcome indicators; (P) Process indicators

Step 3: Collect Strategy Data (for time series graphs) Alaska PFS Evaluation Planning Step 3: Collect Strategy Data (for time series graphs) collect data on a regular basis over time graph results Interpret results revise strategy efforts as needed

CF: Lack of convenient and/or recognized sites for adult community members to dispose of Rx opioids safely Time series graphs make trends easy to spot. Having baseline data help you interpret results. For the Alaska PFS, we are looking for trends both before and after strategy implementation occurs. Strategies to Address community factor above: Increase safe disposal sites to:   Increase access to safe disposal sites Multi-level campaign to: Increase awareness of safe disposal sites Increase use of safe disposal sites *Process indicators

CF: Lack of convenient and/or recognized sites for adult community members to dispose of Rx opioids safely Collecting data on a regular basis is important. If data is collected to infrequently, it’s hard to be respond and make mid course corrections OR see if you are on the right path. Excel is the easiest way to enter and graph data over time. Increase safe disposal sites to:   Increase access to safe disposal sites Multi-level campaign to: Increase awareness of safe disposal sites Increase use of safe disposal sites *Outcome indicators

Rule of thumb for PFS: Two quarters prior to implementation Baseline Data Rule of thumb for PFS: Two quarters prior to implementation Without multiple data points you would have no way to interpret any pattern once implementation begins. Remember that a good baseline will include information gathered at several points over a period of time, rather than simply a snapshot of information gathered over, say, a single weekend. We are certainly available to help review strategy evaluation plans or work with you to help identify appropriate indicators and measures. Please don’t hesitate to contact Suzanne as a first point of contact. She will assist you and certainly pull in the state evaluation team if additional support is necessary.

Quantitative Data Collection Ideas Policy Change progress towards policy establishment compliance with policy efforts to ensure policy compliance # of individuals reached

Telling Stories Quantitative Data vs Stories “head” “emotions” This isn’t a required part of the evaluation but certainly an important component for communities to consider. Qualitative and quantitative methods are, in fact, complementary. Each has strengths and weaknesses that the other doesn’t, and together, they can present a clearer picture of the situation than either would alone. Often, the most accurate information is obtained when several varieties of each method are used. That’s not always possible, but when it is, it can yield the best results. Quantitative data is in the “head” – facts, figures, and graphs “big picture” Stories are about individuals Stories are about emotions – what we want people to “think, feel, and act upon”. They should be true and relevant now. They speak to a broad audience. When we are motivated by what we care about we take action. Consider case studies. Stories are about values …. What we value, what our project values, and how we want to live. About why the program is important or how we are making a difference.

“The measure of our success will not be determined by the number of evaluations done or even upon the quality of the findings … success will depend on the ability to use evaluation findings to strengthen our efforts and sharpen our decision-making” U.S. Agency for International Development. (2011). Evaluation learning from experience: USAID evaluation policy. Washington, DC: USAID.

When choosing strategies, consider: relevance to your chosen CF/IV time needed to plan and implement evaluation requirements (baseline) budget available potential for community level change sustainability Next cohort call: Discuss strategic planning & strategy selection - May 22nd at 10:00am

Your Turn! (Multiple Choice) How confident do you feel in your team’s ability to fulfill the PFS evaluation expectations? Not at all confident Somewhat confident Mostly confident Very confident

Questions? Please use the chat box