1. University of Wisconsin POPULATION HEALTH INSTITUTE Translating Research into Policy and Practice Evaluating Wisconsin’s Safe and Supportive Schools Program D. Paul Moberg, PhD Daphne Kuo, PhD University of Wisconsin-Madison Population Health Institute December 7, 2012 1

2. University of Wisconsin POPULATION HEALTH INSTITUTE Translating Research into Policy and Practice Overview of S3 Evaluation Evaluation Goals Identify areas of relative safety and support (student perceptions) in each participating school Document improvement over time in seven areas of school safety and support, and in academic and behavioral outcomes Identify, code and describe intervention strategies (including curriculum, programs, and policies) Evaluate effectiveness of the program--explore the relationships between intervention strategies, student achievement, behaviors (suspension and expulsion rates), and school climate (including the seven dimensions of school safety scores). 2

3. University of Wisconsin POPULATION HEALTH INSTITUTE Translating Research into Policy and Practice Evaluation Approach Useful--Support program decision-making, improvement and accountability (advice?) Evaluation Design Change over time within S3 schools and in aggregate Comparison high schools (available indicators-- suspension, expulsion, graduation, achievement test scores; oYRBS where available) Complex statistical modeling—students nested in schools, controls for population characteristics; mediators and moderators; propensity scores 3

4. University of Wisconsin POPULATION HEALTH INSTITUTE Translating Research into Policy and Practice Research Design  2010 2011 2012 2013 2014  S3 schools (N=56+) O sch 0 T1T1 O1O1 T2T2 O2O2 T3T3 O3O3 T4T4 O4O4 Non-S3 schools O sch 0 O sch 1 O sch 2 O sch 3 O sch 4 4

5. University of Wisconsin POPULATION HEALTH INSTITUTE Translating Research into Policy and Practice Process Evaluation Process Measures Annual Reports from S3 schools Changes in school policy School programs implemented PBIS Implementation? Would like Input: What data are available on implementation with minimal burden to school staff ? Should annual report be supplemented with a survey? 5

6. University of Wisconsin POPULATION HEALTH INSTITUTE Translating Research into Policy and Practice Outcome Data Measures: DoE Required S3 Index—”School Safety Score” oYRBS—S3 Module (annual over 4 years) Student Achievement (test scores, graduation)* Behavioral Outcomes: (Suspensions, expulsions, behavioral incidents)* *WINSS trends over 6 years (2 before S3 baseline) measured at the school level 6

7. University of Wisconsin POPULATION HEALTH INSTITUTE Translating Research into Policy and Practice How do S3 Students compare to random sample of Wisconsin Students? Comparisons between S3 Schools and Wisconsin High Schools, Common Questions in OYRBS and YRBS 2011 Questions S3WI (the past 30 days) How many days did you not go to school because you felt you would be unsafe at school or on your way to or from school?8%3% (the past 30 days) How many days did you carry a weapon such as a gun, knife, or club on school property?6%3% (the past 12 months) How many times has someone threatened or injured you with a weapon such as a gun, knife, or club on school property?8%5% (the past 12 months) How many times were you in a physical fight on school property?17%9% (the past 12 months) Have you ever been bullied on school property?20%27% (the past 12 months) Have you ever been electronically bullied?15%17% (the past 30 days) How many days did you have at least one drink of alcohol?31%39% (the past 30 days) How many days did you have 5 or more drinks of alcohol in a row, that is, within a couple of hours?17%24% (the past 30 days) How many times did you use marijuana?23%22% (the past 12 months) Has anyone offered, sold, or given you an illegal drug on school property?23%21% * Frequencies for Wisconsin YRBS are from CDC online analysis of about 1400 9 th and 11 th grade students, does include some S3 schools. * Frequencies for S3 High schools are from Wisconsin online YRBS survey designed for S3 project. 7

8. University of Wisconsin POPULATION HEALTH INSTITUTE Translating Research into Policy and Practice Suspensions in S3 Schools Expulsion and Suspension Rates for Participating 66 Wisconsin High Schools MeanS.D.MedianMinimumMaximum Suspension (persons) 3 year average (08-11).251.203.162.018.729 2010-2011.221.172.164.008.678 Suspension (incidents) 3 year average (08-11).669.759.350.0203.370 2010-2011.516.511.315.0102.010 Expulsion 3 year average (08-11).007.004.000.029 2010-2011.006.008.004.000.030 * The 3 years average rates were calculated by the mean of explusion/suspension rates of year 2008-09, 2009-10, and 2010-11. 8

9. University of Wisconsin POPULATION HEALTH INSTITUTE Translating Research into Policy and Practice Suspension Rates in S3 Schools 9

10. University of Wisconsin POPULATION HEALTH INSTITUTE Translating Research into Policy and Practice oYRBS Data Factor Analysis  7 Indices  violence  bullying and harassment  alcohol and drug use  rule and discipline enforcement  general climate and support  general safety Caution—These are student self reported behaviors and perceptions 10

11. University of Wisconsin POPULATION HEALTH INSTITUTE Translating Research into Policy and Practice oYRBS S3 Factors (1) Violence 10During the past 30 days, on how many days did you not go to school because you felt you would be unsafe at school or on your way to or from school? 11During the past 30 days, on how many days did you carry a weapon such as a gun, knife, or club on school property? 12During the past 30 days, on how many days did you carry a gun on school property? 13During the past 12 months, how many times has someone threatened or injured you with a weapon such as a gun, knife, or club on school property? 14During the past 12 months, how many times were you in a physical fight on school property? 32During the past 30 days, how many times have you taken an over-the-counter drug to get high? Alcohol & drugs 31During the past 30 days, how many times did you use marijuana? 33During the past 12 months how many times have you attended school under the influence of alcohol or other illegal drugs, such as marijuana or cocaine 28Drank alcohol in past 30 days. This includes drinking beer, wine, wine coolers, and liquor such as rum, gin, vodka, or whiskey. For these questions, drinking alcohol does not include drinking a few sips of wine for religious purpose 29During the past 30 days, on how many days did you have 5 or more drinks of alcohol in a row, that is, within a couple of hours? 11

12. University of Wisconsin POPULATION HEALTH INSTITUTE Translating Research into Policy and Practice oYRBS S3 Factors (2) Bullying and harrassment 18(Have you been bullied....) Bullying is when one or more students tease, threaten, spread rumors about, hit, shove, or hurt another student over and over again. It is not bullying when two students of about the same strength or power argue. 19During the past 12 months, have you ever been electronically bullied?(Include through e-mail, chat rooms, instant messaging, websites, or texting.) 20During the past 12 months, have you bullied someone else on school property? 21Harassment is when one or more students bother another student with the goal to annoy, disturb, or upset that student. During the past 12 months, have you ever been harassed on school property? 22During the past 12 months, have you been harassed on school property because of your race or ethnic background? 23During the past 12 months, have you been harassed on school property because of your weight, size, or physical appearance? 25During the past 12 months, have you been harassed on school property because someone thought you were gay, lesbian, or bisexual? 24During the past 12 months, have you been harassed on school property because of your gender? General safety 17Do you agree or disagree that violence is a problem at your school? 26Do you agree or disagree that harassment and bullying by other students is a problem at your school? 27Is there gang activity in your school? 45I feel safe at my school 16When you are at school, how often do you feel safe from physical harm? 12

13. University of Wisconsin POPULATION HEALTH INSTITUTE Translating Research into Policy and Practice oYRBS S3 Factors (3) School climate 35 Do you agree or disagree that your teachers really care about you and give you a lot of encouragement? 36 Is there at least one teacher or other adult in this school that you can talk to if you have a problem? 37 Do you agree or disagree that you feel like you belong at this school? 38 In my school, students have lots of chances to help decide things like class activities and rules. 39 There are lots of chances for students in my school to talk with a teacher one-on-one. 40 Teachers ask me to work on special classroom projects. 41 There are lots of chances for students in my school to get involved in sports, clubs, and other activities outside of class. 42 There are lots of chances to be part of class discussions or activities. 43 My teacher(s) notices when I am doing a good job and lets me know about it. 44 The school lets my parents know when I have done something well. 46 My teachers praise me when I work hard in school. 53 Do you agree or disagree that this school has a friendly and welcoming atmosphere? 13

14. University of Wisconsin POPULATION HEALTH INSTITUTE Translating Research into Policy and Practice oYRBS S3 Factors (4) Rule Enforcement 54Do you agree or disagree that you understand the rules for student behavior and conduct at this school? 55Do you agree or disagree that this school has clear consequences for breaking the rules? 56Do you agree or disagree that staff at this school enforce the rules for student behavior and conduct? Importance of School Work 47How often do you feel that the school work you are assigned is meaningful and important? 48How interesting are most of your courses to you? 49How important do you think the things you are learning in school are going to be for your later life? 50Over the past school year how often did you enjoy being at school? 51Over the past school year how often did you hate being at school? 52Over the past school year how often did you try to do your best work in school? 14

15. University of Wisconsin POPULATION HEALTH INSTITUTE Translating Research into Policy and Practice Perceptions of Safe and Supportive School Environment 9th and 11th graders in 2012 Wisconsin S3 schools--oYRBS 15

16. University of Wisconsin POPULATION HEALTH INSTITUTE Translating Research into Policy and Practice Little difference by Gender and Grade 16

17. University of Wisconsin POPULATION HEALTH INSTITUTE Translating Research into Policy and Practice oYRBS Indices by Race/Ethnicity High school students' perceptions on safe and supportive school environment, 9th and 11th graders in 2012 Wisconsin S3 schools NH WhiteNH Black p1p1 Hispanic p1p1 NH Native Am. p1p1 Other NH groups p1p1 VariableNMeansdNMeansdNMeansdNMeansdNMeansd DPI items92180.460.1046220.480.10 *** 42430.490.11 *** 3200.500.11 *** 24020.480.11 *** School Climate90360.510.1344800.530.14 *** 41800.520.14 *** 3200.550.14 *** 23400.520.13 *** Violence (7 items)93000.190.0846750.210.09 *** 43370.210.12 *** 3290.230.13 *** 24310.200.12 *** Violence (6 items)93140.190.0746820.210.09 *** 43450.210.12 *** 3300.230.13 *** 24360.200.11 *** Bully/Harassment92590.490.1246680.470.10 *** 43150.490.12 3260.510.14 *** 24340.500.12 *** Alcohol/Drug Use93070.230.1546690.230.14 43420.250.17 *** 3280.290.19 *** 24380.230.15 Importance of School Work91860.550.1545280.540.14 *** 42660.550.15 3220.580.15 *** 23860.540.15 *** General Safety92530.570.1446470.620.15 *** 43030.600.15 *** 3290.600.15 *** 24140.610.14 *** Discipline/Rules91300.470.1545300.500.17 *** 42370.500.17 *** 3180.520.17 *** 23790.490.16 *** Total87020.430.0842210.440.08 *** 39570.440.09 *** 2970.460.10 *** 22300.440.09 *** Note: The p-values indicates the degree of statistical signficance in differences between the respective racial/ethnic minority students and the non-Hispanic white students. *** p<.001, ** p<.01, * p<.05. Note: "Other NH groups" included Asian American, Pacific Islanders, Mixed race, and other minority students. About 2000 students who did not identify race and ethnicity were excluded from this tables. 17

18. University of Wisconsin POPULATION HEALTH INSTITUTE Translating Research into Policy and Practice Academic Performance (n~21,000) 18

19. University of Wisconsin POPULATION HEALTH INSTITUTE Translating Research into Policy and Practice Academic Performance Relative to students of A’s 19

20. University of Wisconsin POPULATION HEALTH INSTITUTE Translating Research into Policy and Practice oYRBS Indices by Sexual Orientation High school students' perceptions on safe and supportive school environment, 9th and 11th graders in 2012 Wisconsin S3 schools Perceptions on safe and supportive school environment by sexual orientation HeterosexualLGB/Not SureNo answers VariableNMeansdNMeansdNMeansd DPI items177570.470.1022050.530.1411970.480.10 School Climate173610.510.1321550.560.1711820.520.14 Violence (7 items)179770.190.0822350.260.2412290.200.09 Violence (6 items)179990.190.0822430.250.2412340.200.09 Bully/Harassment179170.480.1122260.570.1812230.500.13 Alcohol/Drug Use179590.230.1422540.290.2712480.220.15 Importance of School Work176260.540.1521940.590.1612340.560.15 General Safety178480.590.1522340.640.1712290.600.14 Discipline/Rules175380.480.1621940.530.2112270.510.17 Total165810.430.0820230.490.1411140.440.09 Note: An ANOVA analysis suggested that the sexual orientation was significantly associated with experiences and perceptions on school environment. The LGB students and those with a uncertain orientation were most likely to have negative experiences and perceptions. Students who did not answer the question of sexual orientation were also more likely to report negative experiences and perceptions than those heterosexual students. The associations were significant at 0.001 level (p value) for all dimensions. 20

21. University of Wisconsin POPULATION HEALTH INSTITUTE Translating Research into Policy and Practice Student Perceptions by Sexual Orientation Relative to Heterosexual (=0.0) n~18,000 21

22. University of Wisconsin POPULATION HEALTH INSTITUTE Translating Research into Policy and Practice Violence Index by S3 School 22

23. University of Wisconsin POPULATION HEALTH INSTITUTE Translating Research into Policy and Practice Perceived School Safety by S3 School 23

24. University of Wisconsin POPULATION HEALTH INSTITUTE Translating Research into Policy and Practice Sample School Report (1) 24

25. University of Wisconsin POPULATION HEALTH INSTITUTE Translating Research into Policy and Practice Sample School Report (2) 25

26. University of Wisconsin POPULATION HEALTH INSTITUTE Translating Research into Policy and Practice What other analyses would help you? Contact us: Paul Moberg: dpmoberg@wisc.edudpmoberg@wisc.edu (608) 263-1304 Daphne Kuo: dkuo@ssc.wisc.edudkuo@ssc.wisc.edu (608) 261-1154 26