Performance Indicators Table for Youth Substance Abuse & Use Prevention September 15, 2005 Ottawa, Ontario Wanda Jamieson & Tullio Caputo

Performance Indicator Overview What is a performance indicator? It is something that can be used to measure whether progress is being made against an anticipated output or outcome. Ideally, a performance indicator is measurable in a practical way. Performance measurement is ongoing - it is important to be able to report on results at various periods during the course of a program. Why identify performance indicators? It can help OAS Member States to measure actual results as compared to the anticipated results that are specified in the logic model. Individual programs and services may require more detailed indicators specific to their particular intervention.

Key Results of Literature Review  What we learned: Quasi-experimental designs are the gold standard for evaluation of prevention programs At a minimum, they use pre & post test measures and most use a comparison group They are challenging to design & implement; time & resource dependent; often hindered by lack of trained evaluators and by lack of support from program administrators & deliverers Program fidelity & adaptation is a key concern Data collection is challenging in many settings Most rely on quantitative methods but promising qualitative methods are being used (e.g. focus groups) Both process & outcome evaluations are important Few program evaluations have assessed intermediate and longer term program outcomes

Performance Indicator Table  Performance Indicators Are the measures that Member States can consider contributing to, to determine whether progress is being made towards a particular output, immediate or intermediate outcome  The PI Table indicates performance measure by: Outputs Immediate outcomes, and Intermediate outcomes.  Prevention Activity Indicates what type of prevention activity is involved  Data Sources/Methods Indicates the possible data sources that could use & helpful data methods  Comments and Considerations Includes information that Member States may wish to take into account during the discussion period  Discussion Points Provides space where Member States may write their own notes.

Output Performance Indicators  Output indicators measure the products & services of the activities including: the # and nature of programs and services developed or delivered the # and characteristics of the population reached the # and nature of research & evaluation undertaken  Output data sources and methods: Program or Service documentation - qualitatively analyzed to determine the nature of the program or service Program or service delivery statistics which can be quantified & analyzed to determine frequency & scope of delivery and population reached For research & evaluation: literature reviews, meta-analyses, best practice reviews  Considerations: Standard features of the “nature” of the program/service should be defined

Immediate Outcome Performance Indicators  Immediate Outcome indicators measure changes which can be determined upon completion of a program activity. They focus on changes in knowledge and awareness. At more intensive levels of prevention (e.g. selected or indicated) there may also be some consideration given to changes in patterns of use. Examples include: Changes in the level of knowledge and awareness about substance use/abuse & consequences Changes in attitudes towards substance use/abuse Changes in the level of knowledge and skills to make informed healthy decisions and about alternatives to substance use Changes in the age of first use of alcohol or any drug Changes in the level, nature and frequency of use Increases in the knowledge base and diffusion of promising or best practices  Data sources and methods: May include pre-post surveys (e.g. self-report surveys; pre-post focus groups) to obtain quantitative and qualitative data  Considerations: Requires baseline information to measure change Some measures (e.g. changes in use) may require tracking

Intermediate Outcome Performance Indicators  Intermediate Outcome indicators measure changes which can be determined from prevention activity more globally, over time. They may: Still focus on changes in knowledge and awareness and perceptions of substance use/abuse (these are dynamic factors that change with age and over time) Measure changes in substance abuse behaviour and patterns among youth (level, nature, frequency of use) Measure changes in healthy lifestyle behaviour and patterns among youth (e.g. drinking and driving) Measure changes in the adoption of promising/best practices in youth prevention programs and services  Data sources and methods: Pre-post surveys (e.g. self-report surveys; pre-post focus groups) to obtain quantitative and qualitative data; population-level surveys or statistics (e.g accident rates for youth where substance use was a factor) as well as stakeholder (opinion research)  Considerations: Requires baseline information to measure change. Attribution to specific prevention programs and services can be challenging.

Questions?