1. Step 5: Grantee Evaluation
State Evaluation Team at the University of Alaska Anchorage
Bridget Hanson, PhD
Jodi Barnett, M.A.
Oliver Smith, B.A.

3. Step 5
Most grantees have transitioned into implementation and evaluation after full or partial strategic plan approval. We will spend the next hour of time focused on evaluation.
Step 4

4. Overview Review PFS grantee evaluation structure
Refine evaluation plans in communication spreadsheets
Breakout discussion of evaluation plans by strategy type - media, safe disposal/storage, safe prescribing
MIS introduction and begin development
CLI-R review for Oct.15 deadline

5. 1. PFS Grantee Evaluation
Community Readiness Assessment
Pre
Post

6. 1. PFS Grantee Evaluation
Intervening Variables
Strategies
Community Factors
Social Availability of Rx Opioids
Strategies
Community Factors
Retail Availability of Rx Opioids
Through the data assessment process (step 1), grantees have identified community factors that influence the three main intervening variables prioritized for the Alaska PFS. Strategies have been selected by grantees to address the community factors in step 3. Evaluating the strategies and community factors are a priority of the PFS for step 5 (evaluation) grant and the focus of our time for the next hour.
Strategies
Community Factors
Perceived Risk of Harm from Rx Opioids/Heroin

7. 1. PFS Grantee Evaluation
Communication spreadsheet
In order to summarize chosen strategies and evaluation plans in one brief document, the evaluation team has created a communication spreadsheet for each grantee based on their strategic plan.  Grantees have also provided input. We see this as a useful resource for state staff, grantee staff, and perhaps coalition members to have a common understanding of overall strategy efforts and to help with CLI-R completion. An electronic copy has been included on thumb drives distributed and hard copies can be found in your folder.
Talk through components of the spreadsheet.

8. 1. PFS Grantee Evaluation
Outcome – What you expect to change among your target population from strategy efforts
Indicator – Evidence to show that the change is occurring among the target population
Measure/Method – How specifically you will measure the indicator
These are three components of the communication spreadsheet to be familiar with during our work today.

9. 1. PFS Grantee Evaluation
Outcome – Reduce sharing of prescribed opioids with friends or family members among adults 18 and older
Indicator – % of adults prescribed opioids over the past year who report giving or selling them to a friend or family member
Measure/Method – Quarterly online survey of 100 people prescribed opioids a year ago at a local pharmacy (Did you have pills leftover? If so, what did you do with your leftover pills?)
This in an example of the three key components.

10. 1. PFS Grantee Evaluation
Process
Indicators
Outcome Indicators
How much you are putting out to the target population with your strategies - “outputs”
Change you expect as a result of your strategies in the target population

11. 1. PFS Grantee Evaluation
Process
Indicators
(1-2)
Outcome Indicators
(3-4)
PFS strategy
evaluation
Awareness outcomes are likely to change first, followed by knowledge and attitudes. Behavior change is likely to happen last (perhaps 3-5 years down the road).
Quantity
Reach
Awareness
Knowledge
Attitudes
Behaviors
How much
So what?

12. 1. PFS Grantee Evaluation
INDICATOR PRIORITIZATION TIPS:
show whether early strategy modifications are needed
likely to change early, not 3-5 years down the road
consider target audience which may be different from target population
There is not enough time or funding to evaluate everything. Decisions must be made based on a number of factors.

13. Keep it manageable and focused!
1. PFS Local Evaluation
Keep it manageable and focused!
Doing less but doing it well is better than trying to do too much. There is limited time and funding.

14. 1. PFS Grantee Evaluation
MEASUREMENT TIPS:
Include baseline
Measure frequently (at least initially)
Measure consistently
Baseline measures enable a community to determine not only the level of an indicator but also the trend in the level of the indicator prior to implementing prevention strategies.
A single measure of any intermediate variable is not very useful in monitoring local environmental prevention nor detecting changes. Therefore, it is most desired that any measures be collected over an extended period of time to provide a series of values for intermediate variable, i.e., a time series of data.
Consistency in the sample is more important than being fully representative in the community.

15. 1. PFS Grantee Evaluation
Strategy 1: High school cartoons to increase perceived risk of harm (opioid misuse)
Example of process evaluation data

16. 1. PFS Grantee Evaluation
Strategy 1: High school cartoons to increase perceived risk of harm (opioid misuse)
Example of outcome evaluation data

17. 1. PFS Grantee Evaluation
Strategy 2: Increase disposal locations and promotion of locations
Example of process evaluation data

18. 1. PFS Grantee Evaluation
Strategy 2: Increase disposal locations and promotion of locations
Example of outcome evaluation data

19. 2. Breakout by Grantee Refine communication spreadsheets
Focus on indicators & methods
Breakout into grantee teams to review and finalize communication spreadsheets
15-20 minutes

20. 3. Breakout by Strategy 3 GROUPS Media Safe disposal/ storage
Safe prescribing
Breakout into 3 groups to discuss strategies and evaluation plans
30 minutes

21. 4. MIS and Graphs

22. 4. MIS and Graphs Why use an MIS for indicator data? Organize data
Visually display data
Communicate & make decisions about data
There is an MIS template in Excel included on the thumb drive you received. You also have a tip sheet which gives you step by step instruction on how to create graphs and dashboards from the data you enter into your MIS.

23. 4. MIS and Graphs
Strategy 1: High school cartoons to increase perceived risk of harm (opioid misuse)
Example of a strategy dashboard

24. 4. MIS and Graphs
Strategy 2: Increase disposal locations and promotion of locations
Example of a strategy dashboard

25. 4. MIS and Graphs
Demo MIS and graph making

26. 4. MIS and Graphs
Strategy 1: High school cartoons to increase perceived risk of harm (opioid misuse)
Line graph
-trend lines, ongoing
-frequent, consistent measurement
Bar graph
-points in time
-less frequent, inconsistent measurement

27. 4. MIS and Graphs
Strategy 1: High school cartoons to increase perceived risk of harm (opioid misuse)
Simple graph
Layered graph

28. 4. Breakout by Grantee Start building an MIS
Enter baseline data and practice making graphs
15-20 minutes

29. 5. CLI-R Review Contents: Q33: Implemented Prevention Interventions
Keys to decreasing burden
Multi-component strategy entry
Prevention Intervention Costs (Q )
CSAP Strategy Type Sub-Forms
General requirements & types
Environmental & Information Dissemination Basics
Live Tutorial

30. 5. CLI-R Review Q33: Implemented Prevention Interventions
Completing questions correctly the first time will decrease burden.
Divided into 8 questions (a-h):
Intervention Name
Service Type
Service Type Name (concise)
Date Started
CSAP Strategy Type
Institute of Medicine (IOM) Category
Intervention Targets
Status
Recommendations for these answers are provided on your Communication Spreadsheets
PEP-C has provided an extensive list of pre-defined Intervention Names and Service Types, but if your intervention strategy does not fit any of the pre-defined names or types, please select “Other.”
The drop down list for Intervention Name should only have one pre-defined “Other.”
The drop down list for Service Type has several “Other” types to choose from, and should be chosen based on CSAP strategy type.
Service Type Name should concisely describe the purpose of the intervention. Ex. Increase Perception of Risk of Harm Media Campaign. The CLI-R guidance manual warns away from listing multiple or as individual activities.
Date Started would be considered when your coalition began putting resources (e.g. funding, time, supplies, and personnel) towards this specific prevention intervention. An example could be when a member/employee of the coalition was given the assignment or hired to start developing media for a media campaign or when funds were used to purchase material for specifically for the strategy (i.e. Prescription Drug Drop Box was purchased.) Different than actual implementation date, which will be asked on the sub-form.
Most coalition’s strategies fall into the Environmental or Information Dissemination CSAP strategy types. It is important that the strategy or component is assigned to the correct type to trigger the correct sub-form.
Most coalitions’ strategies IOM category should fall into the Universal indirect category since most intervention are environmental strategies and not directly targeting individuals (having direct interaction with the target population).
A strategies Intervention Targets will highly depend on each coalitions population they want to reach with their individual strategies.
The status will depend on if strategies are being actively implemented. Since this is the first time submitting your CLI-R, some or all of your strategies may be inactive. The status should be checked and/or changed for every CLI-R report based on the prevention intervention strategies last 6 months activity.

31. 5. CLI-R Review Q33: Keys to Decreasing burden
Intervention Name: Controls set-up for single or multi-component strategy.
Service Type: Can only use once per multi-component strategy.
CSAP Strategy Type: Triggers the associated sub-form.
Intervention Name, Service Type, & Service Type Name cannot be changed nor the strategy deleted after the first CLI-R submission a strategy is reported on.
Service time is also directly related to the CSAP Strategy Type, and in the MRT is divided CSAP Strategy Type categories in the drop down menu.
It is incredibly important that the correct sub-form be triggered to ensure that questions relevant to your strategy are triggered.
If after implementation, your coalition has deemed a strategy to be ineffective or have another reason for discontinuing the strategy, coalitions may choose to set the status as inactive while it is adapted or if the strategy is abandoned entirely the status should be set discontinued.

32. 5. CLI-R Review Q33: Multi-component Strategies
If the same pre-defined intervention name is used the MRT will be automatically triggered to nest strategies.
Advantage: Decreases the number of questions.
Disadvantage: If “Other” is selected as the Intervention Name, the MRT nesting feature will not work.
If your coalition has a multi-component strategy/ies that uses “Other” as the Intervention Name and would like the components to be nested, PEP-C’s recommendation was to enter them separately then contact me so I can inform PEP-C of the Intervention Name/s and Service Types that you want to be nested together.
If the multi-component strategy nests, then your coalition will only have to answer the Service Type Questions and CSAP Strategy Type Questions for each component.
Within a multi-component strategy, the MRT will not allow multiples of the same Service Type, so to work around this issue, “Other” will be selected if the same Service Type is used.
This feature is also a problem because if you have multiple strategies that are suitable to use a pre-defined intervention name you will not be able to use it more than once. This is due to the MRT being set up to automatically nest if the Intervention Name is the same. This is why on your coalition’s communication spreadsheet, if you have multiple media campaigns, the Intervention Name appears as “Other ‘________’.”

33. Prevention Intervention Costs (Q172-189)
5. CLI-R Review
Prevention Intervention Costs (Q )
Requires completion per strategy.
Annual questions (April 1 – September 31 reporting period)
Divided into 3 sections:
Ongoing Costs
In-kind contributions
Start-up costs
SAMSHA Cost Template
Only necessary to answer per strategy, and for multi-component strategies, should have to be reported per component.
Most cost questions are only answered on an annual basis, and will be reported on the CLI-R for the reporting period for April 1 to September 31.
The CLI-R guidance manual emphasizes that there are clear distinctions between Start-up and Ongoing costs, and grantees should be careful to consider which is which.
The PEP-C team provides the SAMSHA Cost Template as a free resource for help you track your project related costs. Keep in mind this template was not originally developed for the purpose of complete the CLI-R, so there may be information that does not match up.

34. CSAP Strategy Type Sub-forms:
5. CLI-R Review
CSAP Strategy Type Sub-forms:
6 types:
Prevention Education
Alternative Drug-Free Activities
Problem Identification and Referral
Community-Based Processed
Environmental Strategies
Information Dissemination and Communication
General Requirements:
Activity tracking
Total estimated population # reached
Demographic estimates of population # reached
There are 6 types of CSAP Strategy Types to choose from, but the majority of the coalitions’ strategies fall into Environmental or Information Dissemination strategies. Grant focus on environmental change.
There are 3 general data reporting requirements:
Activity tracking: Types and # of activities done/information disseminated. (Ex. # brochures distributed, # participants completed training, # community meetings presented at, # participants at the community meeting your presented at, # promotional activities)
Demographic estimates are fairly detailed and lengthy.

35. 5. CLI-R Review Information Dissemination Sub-Form (Q142-171)
Environmental Strategies Sub-Form (Q )
3 Main Sections:
Training of environmental influencers (Q )
Policy enactment, establishment, or implementation (Q )
Enforcement (Q )
Other environmental interventions (Q132)
Covers the majority of methods for information dissemination:
Ex. Television ads, website, etc.
If the strategy does not fit any of them, Q162

36. 5. CLI-R Review
CLI-R Live Tutorial