1. A Tool to Monitor Local Level SPF SIG Activities
Indicator Matrix
A Tool to Monitor Local Level SPF SIG Activities

2. History of the Indicator Matrix
Developed by the SPF SIG Statewide Epidemiology Workgroup.
Based on the review of all SPF SIG community strategic plans.
Goal is to assist communities with local level data and strategy monitoring.
Natalie
3/13/09

3. Course Overview
Agenda
House Keeping
Natalie
3/13/09

4. Training Work Agreement
What behaviors can we all agree to do to make this training enjoyable for everyone attending?
Natalie
3/13/09

5. Indicator Matrix Training Objectives
Define the term “indicator”.
Explain process and outcome measures.
Describe the relationship between contributing factors and outcomes.
Explain how to use the Indicator Matrix to identify local process and outcome measures.
Natalie
3/13/09

6. Indicators
The factor, variable, or observation that will be used to determine that an immediate or direct effect of a program has occurred. (Getting to Outcomes)
A numerical measure of a quality or characteristic of some aspect of a program; evidence that something is occurring, that progress is being made. (
Paula
3/13/09

7. Indicators
Is the unit of measurement (or pointers) that is used to monitor or evaluate the achievement of project objectives over time. Indicators can include specification of quantifiable targets and measures of quality. (
Paula
3/13/09

8. Indicators
Ways of quantifying objectives: for example, road accident numbers are one indicator of safety (
Established quantifiable objective measures
There are both process and outcome indicators
Paula
3/13/09

9. Indicators Activity
In a small group, review the “Is This an Indicator?” worksheet.
Decide if the statement is an indicator. If it is, put an “X” on the line next to it.
Review as a larger group.
Paula
3/13/09

10. Indicators-Activity 2
In a small group, without looking at the Indicator Matrix, list 5 indicators your SPF SIG project would monitor.
4 minutes.
We will review as a large group
Natalie
3/13/09

11. Long Term Outcomes
Consumption Measures and Risk Behaviors are Long Term Outcomes (see first two pages of the Indicator Matrix).
Long-Term outcomes (also known as impacts): Changes that occur as a result of many interventions. Long-term outcomes are likely to be changes in behaviors, conditions (e.g., risk factors and status (e.g., poverty rates). (Getting to Outcomes)
Each Long-Term outcome has indicators communities and the state can use to measure the success of efforts.
Natalie
3/13/09

12. SPF Key is a data driven process.
Strategies are identified based on local data (aka contributing factors).
Local data will help monitor the effectiveness of the strategies.
Natalie
3/13/09

13. Process evaluation
Assessing what activities were implemented, the quality of the implementation, and the strengths and weaknesses of the implementation. Process evaluation is used to produce useful feedback for program refinement, to determine which activities were more successful than others, to document successful processes for future replication, and to demonstrate program activities before demonstrating outcomes.
Chinman, M., Imm, P., Wandersman, A.. (2004). Getting to Outcomes: Promoting Accountability Through Methods and Tools for Planning, Implementation, and Evaluation. Rand, Santa Monica.
Natalie
3/13/09

14. Process Measures/Indicators
Indicators that the intended process or plan is “on track.” One process indicator showing success in developing a collaborative effort may be the development of an interagency agreement. (Getting to Outcomes)
Natalie
3/13/09

15. Process Measures
Documentation of staff time, resources invested in delivering strategy, and counting the number of services delivered.
Quantitative and Qualitative
Examines the effort required to implement programs, practices, and policies.
Documenting process of strategies helps identify what did or did not produce results.
Hawkins, D.J. &Nederhood, B.. (1987). Handbook for Evaluating Drug and Alcohol Prevention Programs: Staff/Team Evalution of Prevention Programs (STEPP). U.S Department of Health and Human Services.
Natalie
Refer to process measures on the indicator matrix.
3/13/09

16. Outcomes
Immediate results or long term results associated with strategies.
Outcomes determine the effectiveness of strategies.
An immediate or direct effect of a program; outcomes are frequently stated, by a specified date, there will be a change (increase or decrease) in the target behavior, among the target population. (Getting to Outcomes)
Paula
3/13/09

17. Identifying Process Measures- Activity
Select 1 strategy your SPF SIG is implementing.
Fill out the following sections of the “Process Measures/Indicator Worksheet”:
Strategy
Activities (Make sure to list each activity)
Process Measures (use the indicator matrix and list any that may not be on the worksheet)
You will have 15 minutes.
We will discuss as a large group.
Paula
3/13/09

18. Identifying Outcome Indicators- Activity
Using the Indicator Matrix, identify Local Level Outcome Indicators for your strategy.
You will have 10 minutes.
We will review as a large group.
Natalie
Refer back to Indicator Matrix and identify outcomes for strategies they are currently working on. Circle the outcomes on their worksheets.
3/13/09

19. Using Process and Outcome Indicators to Monitor Your Project
Identifying the Local Level Outcome Indicators and monitoring them is crucial to the success of your project.
If you are not achieving these immediate outcomes, you will most likely not to be able to achieve long-term outcomes. These strategies should be addressed and a decision to modify them, continue them, or change strategies needs to be made.
Use your process measures to evaluate the effectiveness of your strategies.
Natalie
3/13/09

20. Achieving Outcomes # MIP citation LOA/MOA with Local Law Enforcement
Reduction of Crashes
Behavior Changes
# MIP citation
LOA/MOA with Local Law Enforcement
# of Targeted areas identified
# of nuisance calls
# of UAD party visits by Police
# of Party Patrols/month
# of arrests for providing to minor
Long Term Outcomes
Local Level Outcomes
Process Measures
Natalie
3/13/09

21. Achieving Outcomes Natalie 3/13/09 Long-Term Outcomes
Local Level Outcomes
Process Measures
Strategy Activities
Natalie
3/13/09

22. Local Level Outcome Indicators
Achieving Outcomes
Decrease in binge drinking
Decrease in current drinking
Long-Term Outcomes
# of citations for sales to minors
# of arrests for selling to minors
# of SID compliance checks
Decrease in youth self-reporting not being carded
Local Level Outcome Indicators
# of merchants trained
Frequency of trainings received
Encourage collaboration of SID and Local Law Enforcement
Process Measures
Media awareness (before compliance checks)
Media awareness (after compliance checks)
Alcohol outlet/server training
Compliance Checks
Activities
Example usingRetail Availability
Natalie
Refer to page 4 in Participant Handouts
3/13/09

23. Monitoring Data Using the Indicator Matrix
Local Epi Workgroups can use the Indicator Matrix as a guide for the data that needs to be collected.
Process measures will be collected and stored by the SPF SIG project.
Local Level Outcomes require collaboration in the community and assist in identifying what data the local epi workgroup needs to collect and from whom.
Paula
3/13/09

24. Identifying Data Sources and Partners
On your “NM SPF SIG Key Indicators” worksheet identify the following:
Local level outcomes your SPF SIG project needs to collect based on all of your strategies.
Local data sources
Partners (agencies or specific people)
Paula
3/13/09

25. Monitoring Local Level Data
Identify 1 outcome from your strategies and create a poster for the group that illustrates:
The outcome
Your data source
How will you monitor it over time? (Will you put the data into a table, chart, graph, maps, etc.)
Your poster will be shared with the entire group.
Paula
3/13/09

26. Objectives Define the term “indicator”.
Explain process and outcome measures.
Describe the relationship between contributing factors and outcomes.
Explain how to use the Indicator Matrix to identify local process and outcome measures.
Natalie
3/13/09 26

27. Please fill out your evaluations and leave them on your table.
Questions or Comments?
Please fill out your evaluations and leave them on your table.