DEVELOPING AN EVALUATION SYSTEM FOR SWPBS Rob Horner and Bob Algozzine

Goals  Define core features of an evaluation plan for state or district SWPBS evaluation.  Define core features of an evaluation report for state or district SWPBS evaluators  Define current and future Evaluation Blueprint developments  Provide examples of effective state-level evaluation

Evaluating Implementation of SWPBS  Purposes  Evaluation to improve the fidelity of implementation Give teams regular information on how to improve  Evaluation to assess policy decisions Give state/ district decision-makers documentation of (a) fidelity (b) impact on student outcomes (c) sustainability

A Basic Model  Organize evaluation around decision-making questions.  What support was provided to whom for implementation of SWPBS?  Was SWPBS implemented at a level of fidelity that we can expect change in student outcomes?  Was SWPBS implementation related to improvement in student outcomes (reduction in problem behavior, increase in academic success)  Did SWPBS implementation improve the capacity of the district/state to sustain implementation and impact?

The Logic Model for Evaluation

Evaluation Questions  What support was provided to whom for implementation of SWPBS? CONTEXT and INPUT  Was SWPBS implemented at a level of fidelity that we can expect change in student outcomes? FIDELITY  Was SWPBS implementation related to improvement in student outcomes (reduction in problem behavior, increase in academic success)? IMPACT  Did SWPBS implementation improve the capacity of the district/state to sustain implementation and impact? SUSTAINABILITY

Context and Input  Demographic information about schools/districts  Enrollment, SES, Diversity, 504, IEP  Type and amount of Technical Assistance provided  Extent to which TA was perceived as helpful.

Fidelity  Fidelity = The extent to which the core features of SWPBS are being used.  Issues:  1. Implement with flexibility around form and format, but with consistency and precision around CORE Features.  2. Use Fidelity assessments as PART of implementation process as well as summative evaluation. Make the data useful for teams to develop their actions.

Research MeasuresAnnual Self- Assessment Measures Progress Monitoring Measures Universal Tier of SWPBS School-wide Evaluation Tool (SET) Self-Assessment Survey (SAS) Benchmarks of Quality (BoQ) Team Implementation Checklist (TIC) Secondary and Tertiary Tiers of SWPBS Individual Student School-wide Evaluation Tool (I- SSET) Benchmarks of Advanced Tiers (BAT) (To be developed) Overall Summary of Implementation Implementation Phases Inventory (IPI) Phases of Implementation (POI) The Tools: To Measure Fidelity

Year 1Year 2Year 3 PreSumFallWinSprSumFallWinSprSumFallWinSpr Universal SWPBS Progress Monitoring: TIC XXXXXX Annual Self- Assessment: BoQ, XXX Research Measure: SET XXX Self-Assessment Survey: SAS XXXX Secondary/ Tertiary Progress Monitoring: TBA XXXXXX Annual Self- Assessment: BAT XX Research Measure: I-SSET XX The Fidelity Tools: A Schedule

Using Fidelity Data  Forms on  New site Spring 2010:

Fidelity Tools: Uses

Commit Team Self-Assess Core Features Classroom Data Sec/Ter Mean Score Per Item Across 9 Middle Schools

What coaching advice do you have for this school? Commit Team Self-Assess Core Features Classroom Data Sec/Ter

What coaching advice do you have for this school? Commit Team Self-Assess Core Features Classroom Data Sec/Ter

What coaching advice do you have for this school? Commit Team Self-Assess Core Features Classroom Data Sec/Ter

What coaching advice do you have for this school? Commit Team Self-Assess Core Features Classroom Data Sec/Ter

What coaching advice do you have for this school?

What coaching advice do you have for a district leadership team with these BoQ data? School A B C D E F G

States Implementing SWPBS schools in all 50 states States Number of Schools Illinois

Illinois Elementary Schools

Illinois Middle Schools

Illinois High Schools

Illinois K- (8/12) Schools

Impact: Is SWPBS benefiting Students?  Social Behavior  Attendance  Referrals to Special Education  Proportion of time in typical educational contexts  Office Discipline Referrals (ODR)… (e.g. SWIS)  Individual student points/ behavior records  Academic Behavior  Progress monitoring (oral reading fluency… DIBELS)  Universal Screening  Standardized Test Scores

Total Office Discipline Referrals Total Office Discipline Referrals as of January 10

Average Office Discipline Referrals per day per month as of January 10

SWIS summary (Majors Only) 2,732 schools; 1,385,191 students; 1,244,026 ODRs Grade RangeNumber of Schools Mean Enrollment per school Mean ODRs per 100 per school day K-61, (sd=.45) (1 /300 / day) (sd=1.41) (1/ 100 / day) (sd=.1.56) (1/105 / day) K-(8-12) (sd=1.85) (1/ 100 / day

Interpreting Office Referral Data: Is there a problem?  Absolute level (depending on size of school)  Middle, High Schools (> 1 per day per 100)  Elementary Schools (>1 per day per 300)  Trends  Peaks before breaks?  Gradual increasing trend across year?  Compare levels to last year  Improvement?

Application Activity: Absolute Value Is there a Problem? Middle School of 625 students? Compare with national average: 625/100 = X.92 = 5.75 Office Discipline Referrals per School Day

Elementary School with 150 Students Compare with National Average 150 / 100 = X.34 =.51

Cafeteria Class Commons Hall 12:00 Lang. Defiance Disruption Harrass Skip

SWPBS implementation in Oregon 714

ElementaryMiddle K (8-12)High Schools

National Means N = 343

Sustainability: Continuous Regeneration  Building capacity State District School  The critical role of coaching  The importance of regular improvement Stay focused, but make it easier to do each year.

Leadership Team Funding Visibility Political Support TrainingCoachingEvaluation Active Coordination Local School or District Teams/Demonstrations Behavioral Expertise Policy

Sustaining SWPBS in Districts  Build District Capacity for Sustained Effects  Policy statement focused on social behavior  Job Recruitment Content “knowledge and experience implementing school-wide positive behavior support systems”  Annual Evaluation Demonstrated effectiveness implementing school-wide pbs practices.  Fall Orientation Content For administrators, classified, certified staff  Board outcome measures  School Improvement Goal

Organization of an Evaluation Report  Purpose  Description of SWPBS Core features, Implementation Process  Context and Input What TA is being provided, to whom, at what intensity  Fidelity Is the TA resulting in implementation at criterion?  Impact Is the TA resulting in benefits for students Are the systems and outcomes sustainable  Cost  Recommendations

North Carolina: An Exemplar