1. PFS Evaluation Presented by Alaska PFS State Evaluators:
Bridget Hanson and Jodi Barnett

2. WELCOME! WHO’S JOINING US?
Let’s introduce ourselves
Name
Community
Project role
Please use the chat box

3. Meeting Guidelines We encourage participation!
Share your comments, questions, ideas.
Be positive, be problem solvers.
Keep chat box comments on-topic.

4. “One of the great mistakes is to judge policies and programs
by their intentions
rather than their results”
Milton Friedman

5. Webinar Objectives Alaska PFS staff will understand:
project evaluation expectations
importance of baseline data collection
the difference between the federally required versus Alaska PFS evaluation

6. Your Turn (Raise Your Hand)
How many of you have read the evaluation section of the guidance document in the past month?

7. Alaska PFS timeline
Strategic plans are due on June 30th which will include an evaluation plan for approved strategies
Step 3
 Vision statement
 Capacity building plan
 Community logic model
 Strategy action plans
 Evaluation plan

8. Key Elements of the Alaska PFS Project
Data driven
All 5 Steps of the SPF process are driven by data – The process is designed to be Purposeful.
In the evaluation step (step 5), data collected will be used to determine if mid-course changes to strategies are necessary and if they are having the anticipated effect on community factors.

9. Ultimate goal is to prevent/reduce
Key Elements of the Alaska PFS Project
Ultimate goal is to prevent/reduce
non-medical use of prescription opioids (12-25 year olds)
use of heroin (18-25 year olds)
Prescription opioids is the priority substance - PFS grants were required to address alcohol or prescription drugs
Heroin is an addition included by the Alaska PFS as we know the substances are related
Change in consumption is unlikely to occur from this grant alone over the next 2-3 years. However the more targeted our efforts and the efforts of others working on this issue, the more likely it is that we will see a change in consumption over time.
Strategy evaluation data related to community factors can help to show whether PFS strategies are moving communities towards the goals of decreasing NMUPO and use of heroin.

10. Reminder: Federal vs Alaska PFS evaluation
Community Level Instrument (CLI-R) is a federal evaluation requirement
Strategy evaluation efforts (using time series graphs) are specific to the Alaska PFS project
Data collected and reported for the federal evaluation requirements may be helpful but they are designed to be most useful for SAMHSA in order to answer national evaluation questions. Data you collect for strategy evaluation efforts will be most informative and useful for your project.

11. Community-Specific Logic Model Elements
Community Strategies 
“What strategies will we choose to address the community factors”
Community Factors
“Why does Rx opioid misuse/abuse occur specifically in our community?”
Social Availability
Easy access to Rx opioids through social sources
Retail Availability
Easy access to Rx opioids through providers & dispensers
Perceived Risk of Harm from Rx opioid misuse or heroin use
Community factors were a primary focus during the data assessment process and will be the primary focus of evaluation efforts as well. The more targeted the community factors are, the easier it will be to choose strategies and evaluate your efforts.

12. Your Turn! (Chat Box)
Describe 1 or 2 community factor(s) that rose to the top in your data assessment?
Please use the chat box

13. Alaska PFS Evaluation Planning
Step 1: Finalize Community Logic Models
snapshot of what the project is trying to accomplish
communication tool
help guide and focus planning efforts

14. Alaska PFS Logic Model Components
Community Strategies
“What strategies will we choose to address the community factors”
Community Factors
“Why specifically does Rx opioid misuse/abuse occur specifically in our community?”
Intervening Variables
“Why does Rx opioid misuse/abuse occur?”
Consumption
“Rx opioid misuse/abuse among year olds”
Consequences
“Consequences of Rx opioid misuse / abuse (i.e. Why do we care?)”
Direction of influence once implementation begins
Alaska PFS Logic Model Components
 If-then statements

15. Example Community Logic Model
State-level evaluators at CBHRS have identified indicators and will collect data for intervening variables, consumption, and consequences related to NMUPO and heroin use at the state and funded community levels. This data will be provided to communities in order to monitor outcomes. Communities will focus on local data related to community factors for the PFS outcome evaluation.
Overview of your community approach to reduce Rx opioid misuse and heroin use
A good time to use “if-then” statements to test your logic

16. Your Turn! (Raise Your Hand)
Has your coalition discussed any potential strategies to target community factors?

17. (for time series graphs)
Alaska PFS Evaluation Planning
Step 2: Develop Strategy Evaluation Plans
(for time series graphs)
Identify outcomes, indicators, and measures to answer:
How much are you doing?
How many are you reaching?
Is change on community factors occurring?
A time series graph can be used use to visually evaluate patterns in data collected over time.

18. Intervening Variable: Strategy Target Population:
STRATEGY: multi-level campaign to increase knowledge about HARMs of social access and ways to prevent social access to rx opioids
Community Factor: Lack of community knowledge among adults about harms associated with easy access to Rx opioids
Intervening Variable:
Social Availability of Rx opioids
CSAP Category: Information Dissemination
Strategy Target Population:
Adults 18+
Community Factor: Lack of knowledge among adults about how to prevent social access to Rx opioids
KEY strategy outcomes
Indicators
method / measure
Multi-level campaign to:
Increase knowledge about harms associated with easy access to Rx opioids
Increase knowledge about ways to reduce social access to Rx opioids
Increase safe storage, monitoring, and disposal of Rx opioids
# and reach of awareness products disseminated (posters, articles, etc) (P)
% of adults who have seen campaign messages about safe storage and disposal of Rx opioids (P)
% of adults reporting safe storage and monitoring of Rx opioid prescriptions (or intentions) (O)
% of adults who report disposing of Rx opioids safely (or intentions) (O)
Dissemination tracking
Quarterly random survey of 100 adults (e.g., DMV)
Strategies that are focused on well-defined community factors will greatly facilitate this process. Strategies that do not align with community factors will not work well. A good way to determine if the logic works is to create IF-THEN statements and ask yourself whether they are true.
Example 1
Pick an indicator or indicators that best reflect the process and outcome indicators that are most important to you.
An indicator is anything that is measurable that can be used to identify a change in trends. The indicator needs to be relevant: it should tell you what you need to know about your strategy and expected outcomes.
Find measurements on those indicators.
Once you've chosen indicators, decide exactly what you're going to measure, and for how long.
Ask yourself :
Is this measure showing what it's supposed to measure and not some by-product?
(O) Outcome indicators; (P) Process indicators

19. STRATEGies: 1) increase the Number of safe disposal sites and
2) Implement A campaign to increase awareness and use of safe disposal sites
Community Factor: Lack of convenient and/or recognized sites for adult community members to dispose of Rx opioids safely
Intervening Variable: Social Availability of Rx opioids
CSAP Category
Strategy Target Population:
Adults 18+
Create safe disposal sites:
Environmental
Campaign to promote safe disposal sites:
Information Dissemination
KEY strategy outcomes
Indicators
method / measure
Increase safe disposal sites to:
Increase access to safe disposal sites
Multi-level campaign to:
Increase awareness of safe disposal sites
Increase use of safe disposal sites
# of disposal sites created and # of days open (P)
# and reach of awareness products disseminated (posters, radio ads, etc.) (P)
% of adults aware of at least one local safe disposal site (O)
# visitors of visitors and/or pounds of Rx opioid pills discarded at disposal sites (O)
Activity tracking
Dissemination tracking
Quarterly random survey of 100 adults (e.g., DMV)
Example 2
Most strategies will address one community factor, but sometimes one strategy may address two community factors or two strategies will address one community factor. Strategy Evaluation Templates, which will be provided on the PFS website, can be adapted accordingly to develop your evaluation plans. The DETAL is available to provide further training and technical assistance as you complete the Strategy Evaluation Templates.
(O) Outcome indicators; (P) Process indicators

20. Step 3: Collect Strategy Data (for time series graphs)
Alaska PFS Evaluation Planning
Step 3: Collect Strategy Data
(for time series graphs)
collect data on a regular basis over time
graph results
Interpret results
revise strategy efforts as needed

21. CF: Lack of convenient and/or recognized sites for adult community members to dispose of Rx opioids safely
Time series graphs make trends easy to spot. Having baseline data help you interpret results. For the Alaska PFS, we are looking for trends both before and after strategy implementation occurs.
Strategies to Address community factor above:
Increase safe disposal sites to:
Increase access to safe disposal sites
Multi-level campaign to:
Increase awareness of safe disposal sites
Increase use of safe disposal sites
*Process indicators

22. CF: Lack of convenient and/or recognized sites for adult community members to dispose of Rx opioids safely
Collecting data on a regular basis is important. If data is collected to infrequently, it’s hard to be respond and make mid course corrections OR see if you are on the right path. Excel is the easiest way to enter and graph data over time.
Increase safe disposal sites to:
Increase access to safe disposal sites
Multi-level campaign to:
Increase awareness of safe disposal sites
Increase use of safe disposal sites
*Outcome indicators

23. Rule of thumb for PFS: Two quarters prior to implementation
Baseline Data
Rule of thumb for PFS: Two quarters prior to implementation
Without multiple data points you would have no way to interpret any pattern once implementation begins. Remember that a good baseline will include information gathered at several points over a period of time, rather than simply a snapshot of information gathered over, say, a single weekend.
We are certainly available to help review strategy evaluation plans or work with you to help identify appropriate indicators and measures. Please don’t hesitate to contact Suzanne as a first point of contact. She will assist you and certainly pull in the state evaluation team if additional support is necessary.

24. Quantitative Data Collection Ideas
Policy Change
progress towards policy establishment
compliance with policy
efforts to ensure policy compliance
# of individuals reached

25. Telling Stories Quantitative Data vs Stories “head” “emotions”
This isn’t a required part of the evaluation but certainly an important component for communities to consider.
Qualitative and quantitative methods are, in fact, complementary. Each has strengths and weaknesses that the other doesn’t, and together, they can present a clearer picture of the situation than either would alone. Often, the most accurate information is obtained when several varieties of each method are used. That’s not always possible, but when it is, it can yield the best results.
Quantitative data is in the “head” – facts, figures, and graphs “big picture”
Stories are about individuals
Stories are about emotions – what we want people to “think, feel, and act upon”. They should be true and relevant now. They speak to a broad audience. When we are motivated by what we care about we take action. Consider case studies.
Stories are about values …. What we value, what our project values, and how we want to live.
About why the program is important or how we are making a difference.

26. “The measure of our success will not be determined
by the number of evaluations done
or even upon the quality of the findings …
success will depend on the ability to use
evaluation findings to strengthen our efforts
and sharpen our decision-making”
U.S. Agency for International Development. (2011).
Evaluation learning from experience:
USAID evaluation policy. Washington, DC: USAID.

27. When choosing strategies, consider:
relevance to your chosen CF/IV
time needed to plan and implement
evaluation requirements (baseline)
budget available
potential for community level change
sustainability
Next cohort call: Discuss strategic planning & strategy selection - May 22nd at 10:00am

28. Your Turn! (Multiple Choice)
How confident do you feel in your team’s ability to fulfill the PFS evaluation expectations?
Not at all confident
Somewhat confident
Mostly confident
Very confident

29. Questions?
Please use the chat box