Data-driven Approaches to Monitoring and Evaluating Environmental Strategies Training Workshop for Vermont Community Prevention Coalitions March 20, 2012 PIRE

TODAY’S TRAINERS AND PRESENTERS…  VDH:  Lori Uerz  Suzanne Kelley  PIRE:  Bob Flewelling  Amy Livingston  Vermont Center for Rural Studies:  Erin Roche

AND WORK GROUP #1 FACILITATORS….  Cindy Hayford  Beth Crane  Maryann Morris

And now…. Introductions

Just What Is Environmental Prevention, Anyway? Environmental strategies in a community seek: Environmental strategies in a community seek: 1.To bring about system-level change (including physical space, local community policies, availability of drugs and alcohol, etc.) in order 2.To reduce substance abuse problems (increase health and safety) at the population level. That is Public Health. Both conditions must be met. Both conditions must be met.

One of many examples…..

What about Strategies Designed to Change Community Norms? Yes, norms are an attribute of the community environment. Communications strategies (e.g., media campaigns) can contribute to normative change But are most effective when combined with other environmental change strategies; for example: – Alcohol free events – Bans on outdoor advertising – Stricter enforcement of alcohol laws

Environmental Strategies: Implications for Evaluation  Long-term outcome measures should be at the population-level;  The short-term outcome measures (intervening variables) will typically be environmental conditions or perceptions of such  Implementation often is less well prescribed, creating challenges for planning, implementation, and process evaluation

Evaluating Community-Based Prevention Strategies: For Who?  First and foremost, the evaluation serves the community organization that’s doing the prevention work:  By documenting activities  By monitoring implementation and making adjustments as needed  By indicating if short- and long-term objectives are being achieved (or going in right direction)  By providing empirical data on activities and progress to be shared with stakeholders

Evaluation Components  Evaluation involves collecting and interpreting data on:  Process (information regarding implementation of strategies)  Short-term outcomes (also referred to as intervening variables)  Longer-term outcomes (indicators of the target problem being prevented or reduced)

Assessing Outcomes  Short and long term outcomes need to be measured at multiple points in time (or at least two points)  Focus is on whether and how measures change over time  Comparisons to the state (or nation) may enhance interpretation

Challenges and Limitations of Evaluating Community-Level Strategies Focused on Population-Level Change Community is an N of 1 There are many external factors that influence outcomes Population-level outcome data requires different approaches to collect (no “captive” audience) Standards for implementation (especially intensity) less well known

Implications for Interpreting Findings from Community-Level Evaluations Cannot definitively attribute positive changes to your interventions But positive changes can suggest your interventions deserve some of the credit* Lack of positive changes does not mean your interventions were ineffective * Especially if evidence-based strategies were well-implemented, intervening variables also changed, and you have some reasonable comparison data.

Whether your evidenced-based strategies were appropriate and well implemented (hence the importance of process evaluation) Whether short and long term outcome indicators are moving in: the “right direction” or the “wrong direction” (This is also referred to as “program monitoring” or “performance monitoring) Essential Foci of Any Community- Level Program Evaluation

Logic Models  Simply a way of showing the connections between:  Strategies  Desired Outcomes  The mechanisms through which the strategies are designed to work (i.e., intervening variables)  May be expanded to also show the measures for each intervening variable and outcome  Provide a blueprint for outcome evaluation

Substance- related consequences and substance use Intermediate variables Programs/ policies/ practices Implementing the Strategic Prevention Framework Planning, Monitoring, Evaluation, and Replanning Outcomes-Based Prevention

Important Attributes of Logic Models  Can be depicted with a variety of formats  Show the connections between each strategy and each IV the strategy is expected to change  Shows the measure(s) to be used for each IV and outcome  Identifying the “contributing factors” for each IV will help define appropriate measures  Usually best to focus each model on a single ultimate outcome

One Example of a Logic Model Template

One Example of a Logic Model Template (Filled In)

Measuring Intervening Variables  Each intervening variable may have multiple possible measures (i.e., “contributing factors”)  Typically there is more than one way to measure an IV  Desirable attributes of measures:  Valid  Periodic collection (with consistency)  Available (free or for minimal cost)  Reflective of the target population

Data Sources for Intervening Variables  Established surveys (e.g., YRBS)  Other surveys (e.g., a local parent survey)  Archival data (e.g., compliance check results from Dept. of Liquor Control)  Public records (e.g., liquor outlet locations, enactment of policies, enforcement activities)  Observation (e.g., community scans)  Active surveillance (e.g., purchase surveys)  Focus groups  Interviews with key informants

Measuring Outcomes  Same criteria and sources as for IVs  In addition, keep in mind:  Outcomes based on small numbers of events are not very sensitive (e.g., fatalities)  Measures that may reflect the intensity of intervention efforts are not good measures (e.g., arrest rates, treatment admissions)  The further a measure is from the strategy in the logic model, the more external influences there are

Monitoring Implementation What did you do? How much did you do? And how well did you do it?

Why is it important to track and monitor implementation activities?  Document what was done so: You can determine how well it was done You and/or others can repeat what was done  The information can be used to adjust and improve implementation going forward (formative evaluation)  The information can be used to help understand outcome findings

Monitoring Helps Us Understand Why Strategies Succeed or Fail 25

It’s not a miracle! If you are using appropriate evidence-based strategies and implementing them well (following available implementation guidelines closely, completing all core activities with minimal deviations, enough dosage and intensity) You can expect that they will lead to the desired outcome!

Steps for effective program monitoring 1.Identify the core components/activities for your strategy 2. Develop an implementation plan that lists out the core activities and timeline for completion.

Steps for effective program monitoring contd. 3. As you implement the activities, document progress including anything that can be measured (number of meetings, number and types of media spots, how many people reached, etc.)

Steps for effective program monitoring contd. 4. Assess progress toward short-term outcomes and make adjustments as needed. Is this moving us toward our end goal?

RECAP OF KEY POINTS Environmental strategies focus on: Changing the context, not the individual Population-level outcomes ES require different (and creative) approaches for measuring process and outcomes Although outcome monitoring is important, conducting conclusive outcome evaluations is not the goal for community-level efforts Implementing evidence-based strategies with the highest possible fidelity should be a goal for every community Dramatic change usually takes time and perseverance (e.g., smoking rates in the U.S.)