1. DEVELOPING AN EVALUATION SYSTEM FOR SWPBS Rob Horner and Bob Algozzine

2. Goals  Define core features of an evaluation plan for state or district SWPBS evaluation.  Define core features of an evaluation report for state or district SWPBS evaluators  Define current and future Evaluation Blueprint developments  Provide examples of effective state-level evaluation

3. Evaluating Implementation of SWPBS  Purposes  Evaluation to improve the fidelity of implementation Give teams regular information on how to improve  Evaluation to assess policy decisions Give state/ district decision-makers documentation of (a) fidelity (b) impact on student outcomes (c) sustainability

4. A Basic Model  Organize evaluation around decision-making questions.  What support was provided to whom for implementation of SWPBS?  Was SWPBS implemented at a level of fidelity that we can expect change in student outcomes?  Was SWPBS implementation related to improvement in student outcomes (reduction in problem behavior, increase in academic success)  Did SWPBS implementation improve the capacity of the district/state to sustain implementation and impact?

5. The Logic Model for Evaluation

6. Evaluation Questions  What support was provided to whom for implementation of SWPBS? CONTEXT and INPUT  Was SWPBS implemented at a level of fidelity that we can expect change in student outcomes? FIDELITY  Was SWPBS implementation related to improvement in student outcomes (reduction in problem behavior, increase in academic success)? IMPACT  Did SWPBS implementation improve the capacity of the district/state to sustain implementation and impact? SUSTAINABILITY

7. Context and Input  Demographic information about schools/districts  Enrollment, SES, Diversity, 504, IEP  Type and amount of Technical Assistance provided  Extent to which TA was perceived as helpful.

8. Fidelity  Fidelity = The extent to which the core features of SWPBS are being used.  Issues:  1. Implement with flexibility around form and format, but with consistency and precision around CORE Features.  2. Use Fidelity assessments as PART of implementation process as well as summative evaluation. Make the data useful for teams to develop their actions.

9. Research MeasuresAnnual Self- Assessment Measures Progress Monitoring Measures Universal Tier of SWPBS School-wide Evaluation Tool (SET) Self-Assessment Survey (SAS) Benchmarks of Quality (BoQ) Team Implementation Checklist (TIC) Secondary and Tertiary Tiers of SWPBS Individual Student School-wide Evaluation Tool (I- SSET) Benchmarks of Advanced Tiers (BAT) (To be developed) Overall Summary of Implementation Implementation Phases Inventory (IPI) Phases of Implementation (POI) The Tools: To Measure Fidelity

10. Year 1Year 2Year 3 PreSumFallWinSprSumFallWinSprSumFallWinSpr Universal SWPBS Progress Monitoring: TIC XXXXXX Annual Self- Assessment: BoQ, XXX Research Measure: SET XXX Self-Assessment Survey: SAS XXXX Secondary/ Tertiary Progress Monitoring: TBA XXXXXX Annual Self- Assessment: BAT XX Research Measure: I-SSET XX The Fidelity Tools: A Schedule

11. Using Fidelity Data  Forms on www.pbssurveys.orgwww.pbssurveys.org  New site Spring 2010: www.pbsassessment.orgwww.pbsassessment.org

12. Fidelity Tools: Uses

14. Commit Team Self-Assess Core Features Classroom Data Sec/Ter Mean Score Per Item Across 9 Middle Schools

15. What coaching advice do you have for this school? Commit Team Self-Assess Core Features Classroom Data Sec/Ter

16. What coaching advice do you have for this school? Commit Team Self-Assess Core Features Classroom Data Sec/Ter

17. What coaching advice do you have for this school? Commit Team Self-Assess Core Features Classroom Data Sec/Ter

18. What coaching advice do you have for this school? Commit Team Self-Assess Core Features Classroom Data Sec/Ter

19. What coaching advice do you have for this school?

20. What coaching advice do you have for a district leadership team with these BoQ data? School A B C D E F G

21. States Implementing SWPBS 10000+ schools in all 50 states States Number of Schools Illinois

22. Illinois Elementary Schools

23. Illinois Middle Schools

24. Illinois High Schools

25. Illinois K- (8/12) Schools

27. Impact: Is SWPBS benefiting Students?  Social Behavior  Attendance  Referrals to Special Education  Proportion of time in typical educational contexts  Office Discipline Referrals (ODR)… (e.g. SWIS)  Individual student points/ behavior records  Academic Behavior  Progress monitoring (oral reading fluency… DIBELS)  Universal Screening  Standardized Test Scores

28. Total Office Discipline Referrals Total Office Discipline Referrals as of January 10

29. Average Office Discipline Referrals per day per month as of January 10

30. SWIS summary 07-08 (Majors Only) 2,732 schools; 1,385,191 students; 1,244,026 ODRs Grade RangeNumber of Schools Mean Enrollment per school Mean ODRs per 100 per school day K-61,762444.34 (sd=.45) (1 /300 / day) 6-9482653.92 (sd=1.41) (1/ 100 / day) 9-121769141.05 (sd=.1.56) (1/105 / day) K-(8-12)3124011.00 (sd=1.85) (1/ 100 / day

31. Interpreting Office Referral Data: Is there a problem?  Absolute level (depending on size of school)  Middle, High Schools (> 1 per day per 100)  Elementary Schools (>1 per day per 300)  Trends  Peaks before breaks?  Gradual increasing trend across year?  Compare levels to last year  Improvement?

32. Application Activity: Absolute Value Is there a Problem? Middle School of 625 students? Compare with national average: 625/100 = 6.25 6.25 X.92 = 5.75 Office Discipline Referrals per School Day

33. Elementary School with 150 Students Compare with National Average 150 / 100 = 1.50 1.50 X.34 =.51

34. Cafeteria Class Commons Hall 12:00 Lang. Defiance Disruption Harrass Skip

35. SWPBS implementation in Oregon 714

36. ElementaryMiddle K (8-12)High Schools

37. National Means. 34.87 1.27 1.06 N = 343

42. Sustainability: Continuous Regeneration  Building capacity State District School  The critical role of coaching  The importance of regular improvement Stay focused, but make it easier to do each year.

43. Leadership Team Funding Visibility Political Support TrainingCoachingEvaluation Active Coordination Local School or District Teams/Demonstrations Behavioral Expertise Policy

44. Sustaining SWPBS in Districts  Build District Capacity for Sustained Effects  Policy statement focused on social behavior  Job Recruitment Content “knowledge and experience implementing school-wide positive behavior support systems”  Annual Evaluation Demonstrated effectiveness implementing school-wide pbs practices.  Fall Orientation Content For administrators, classified, certified staff  Board outcome measures  School Improvement Goal

45. Organization of an Evaluation Report  Purpose  Description of SWPBS Core features, Implementation Process  Context and Input What TA is being provided, to whom, at what intensity  Fidelity Is the TA resulting in implementation at criterion?  Impact Is the TA resulting in benefits for students Are the systems and outcomes sustainable  Cost  Recommendations

46. North Carolina: An Exemplar