Washington State Prevention Summit Analyzing and Preparing Data for Outcome-Based Evaluation Using the Assigned Measures and the PBPS Outcomes Report October 20, 2006 Sarah Stachowiak Organizational Research Services

2 Purpose and Goals  Increased knowledge of Assigned Measures (AMs)  Increased skills in collecting participant data  Increased skills in interpreting PBPS Outcomes Report

3 DASA Required Measures  Pre-Post Survey Questions for All Youth Participants years old  PPG Items for Family, Community, School and Individual Domains  Questions on: Perceived Risk, Perceived Harm, Perceived “Wrongfulness” and 30-Day Use of Substances  15 Questions on PPG03 – Individual Domain Scale

4 DASA Assigned Measures – Development Process  Search of literature on Impacts and Effects of Different Best Practices  Search for common shorter term, more direct outcomes for youth and parents participating in different programs and practices  Definitions of Outcomes -> Measurable Indicators  Search for Valid and Reliable Measurement Scales

5 DASA Assigned Measures  Pre-Post Survey Questions for All Youth Participants years old or All Parents/Guardians  Set of Youth and Parent Outcomes that are aligned with different Best and Promising Practices (9 Youth Outcomes / 8 Parent Outcomes)  Scales with 5-8 questions for each of the Assigned Measures – drawn from existing tools or scales

6 Measurement Scales  Search Through Validated Instruments and Curriculum Surveys  Identified Survey Items Consistent with Chosen Indicators Linked to Youth and Parent Outcomes  5-10 Additional Survey Questions per Outcome  Data Collection Across Programs Addressing Outcome and Objectives

7 Parent Outcomes  Improved Family Cohesion  Improved Attitudes about Family Management Skills  Increased Use of Family Management Skills  Increased Family Involvement  Improved Family Communication  Reduced Family Conflict

8 Youth Outcomes  Improved Bonding  Less Favorable Attitudes  Increased Refusal/Resistance Skills  Improved Social Competence Skills  Improved Personal Competence  Reduced Anti-Social Behaviors  Improved Academic Performance

9 Benefits of Assigned Measures  More useful outcome data for County/Tribe and Provider purposes  Ability to look at common changes across different Best Practices and other Programs  More “realistic” questions for respondents  Now have parent outcome data!!

10 Collecting Participant Data  Participant ID Issues  Administering Surveys  Managing Data Collection

11 Assigning ID Numbers  Track participants over time  Administer a multiple tools (e.g., pre and post)  Confidentiality versus anonymity  Unique identifiers  Simple ID  Self-Generated ID  Local ID Field in PBPS

12 Self-Generated ID Numbers  What is the last letter of your first name?  What is the second letter of your last name?  What is the month of your birthday?  What is the first letter of your middle name?

13 Administering Surveys  Share the purpose and intent  Assure confidentiality  Make sure everyone understands the ID code directions  Consider type of administration (e.g., facilitator reads questions)

14 Managing Data Collection  Maintain a survey tracking system  Take steps to maximize response rate  Use “data windows”  Collect data when you have access to participants  Consider incentives

15 PBPS Outcome Report  Levels of Aggregation  Types of Data Presented  Service Characteristics  Pre-Post Changes

16 Levels of Aggregation

17 Descriptive Data  Frequencies: summaries of the number or percent of observations in each response category  Averages: mean of responses  Cross-tabulations: summaries of frequency distributions across different subgroups or levels of a second variable (not yet available)

18 T-TestsT-Tests  Test for statistically significant difference between mean values  Paired Samples – comparison of mean values on one variable over time for the same participants (e.g., Pre vs. Post)  Mean differences “not due to chance”  Standard convention p <.05 (probability that difference is due to chance is less than 5 percent)

19 Interpreting Quantitative Data Look at your data:  What patterns do you see in the rows and columns?  What findings are most interesting?  What client characteristics might explain these patterns?  What program strategies might explain these patterns?

20 Service Characteristics/Demographics  Survey Completion Rate  Average Attendance Rate  Frequencies for:  Gender  Race  Ethnicity  Age (not for parent programs)  Note: Data are dynamic; only relevant categories are shown  Note: Demographics for all participants, not those who had pre-post data

21 Question Detail  Scoring scale  # Pre Post  Pre and Post Results  Average scores  Statistical Significance  Better, Worse, No Change  % Change  State Comparison  Sub-Scales/Average of Questions  #/% Individuals whose scores were…

22 Interpretation Considerations  Sample size  Completion rate  Representativeness  Cross tabulations (available 2007)

23 Group Exercise  Interpreting Outcome Report Data

24 Reporting Findings Considerations:  What do the data say about the outcomes?  Who is your audience? What is your purpose?  How can you best communicate what the data say?  What are the implications of the findings for program development? For marketing?

25 Reporting Findings  Provide Context:  Outputs (e.g., dosage (frequency, quantity of intervention, number of participants)  Description of intervention  Background information that will help you interpret the data  Process information (e.g., fidelity)

26 ResourcesResources  Updated Evaluation Guidebook  Regional Prevention Managers

27 Final Thoughts  Goals of AMs and Outcome Report:  Learning!  Better decision-making  Stronger prevention planning and programming  Work in progress

28 Contact Information Sarah Stachowiak Organizational Research Services x10