Building Evaluation Capacity for States and Districts Susan Barrett Jerry Bloom Sheppard Pratt Health System www.pbis.org www.pbismaryland.org

Leadership Team Management Team PBS Systems Implementation Logic Marketing Visibility Funding Braiding Initiatives Political Support Leadership Team Management Team Active Coordination Training Coaching Evaluation Local School Teams/Demonstrations

Evaluation What are your questions? Do you have the tools to answer? Can you get the answer quickly? Easy, Efficient, Relevant Economic Benefits Behavior Achievement Regular Feedback to all Stakeholders- MD example

How many schools are implementing SW-PBS ? Number by cohort Number by local district Number by implementation phase Number of Active and Inactive by local district Number of schools reporting data (rate of return) Are schools implementing SW-PBS? State and District needs to determine process tool (i.e. team implementation checklist, implementation inventory, benchmarks of quality, EBS self assessment) Are schools in beginning or advances stages of implementation? Implementation Phase Inventory used to track level How do reported rates of problem behavior in local district or state compare to the national average? (SWIS) by Elementary, Middle, High, Alternative What is the percentage of students who receive 0 or 1 Office referrals/year? (SWIS) By Elementary, Middle, High, Alternative What are the problem behaviors? (SWIS) Elementary, Middle, High, Alternative Where are problem events occurring? (SWIS) What time do problem events occur? (SWIS) By Elementary, Middle , High, Alternative What are the discipline patterns by gender? (SWIS) What are the discipline patterns by ethnicity? (SWIS) What are the discipline patterns for students on an IEP? (SWIS)

Evaluation Template Fairfax County Example

Forms/Surveys and Checklists Tier I Surveying Staff- EBS Survey Fidelity Checks- Team Implementation Checklist, SET, BOQ, IPI Safety Surveys- Oregon Safety Survey Outcome Measures- -ODR, Suspension, Attendance, Special Ed, GPA, Honor Roll, AYP

Forms /Checklists to TRAINING MATERIALS EXAMPLE– MD Training Materials

Tools State Database- Data Entry, Report Generation Various Levels POC Coach State Team www.pbssurveys.org www.pbseval.org www.swis.org

Marketing and Visibility Who are your stakeholders? Do you have a spokesperson? Using the data to create newsletters, presentations, fact sheets, elevator business cards- important you can get access to what you need to make your case on the fly!! www.pbismaryland.org Maryland Newsletter Be Careful

Multiple levels of Visibility State and Local Level: Presentations, Trainings, Stakeholder meetings, Interagency efforts, (Transformation; Mental Health Integration; Wraparound) Multiple Media: Visual, Face to Face, Written, Website Multiple Audiences: School Administrations and Instructional Leaders; University staff; Legislators, Potential alternative funders; State and Local Political appointees; Juvenile Justice; Vendors in the System of Care; Parent and other advocacy organizations; Community Members

PBIS Maryland Partner Meeting Management Team meets with Assistant Superintendent of Student Services and Special Education

Jerry

Jerry

91% Rate of Return Susan

Data Source: Implementation Phases Inventory Evaluation Question: Are schools in beginning or advanced stages of implementation? Data Source: Implementation Phases Inventory Overall, the IPI data suggest a relatively advanced level of self-reported implementation among the schools in Maryland Susan Jerry, Can you include percentages? Thanks!

Evaluation Question: Are schools implementing School-wide PBIS Evaluation Question: Are schools implementing School-wide PBIS? Data Source: Team Implementation Checklist (Target Criterion = 80%) Based on the total average of the 327 checklists submitted, school teams report that 78% of items are in place. Susan Jerry, can you add trend line?

Evaluation Question: Are schools implementing SW-Positive Behavior Support? Data Source: SET (Objective Criterion = 80%) -All regions met objective criterion -A pre-post comparison regional average shows a 69% increase. Susan Schools were able to establish 80% of the school-wide critical features within the first year of implementation. 222 schools have at least two SET scores 182 SETs were completed in 2007

Catherine- Office Discipline Referrals are the “precursor” to suspension/expulsion. That is the data point that is analyzed most often in Universal PBIS—as the data can identify potential problem areas, students, teachers, times of day, etc. so that interventions can be put in place BEFORE it is a crisis. Since much of this work is about developing positive relationships, let me share one anecdote about an administrative intervention to decrease ODR’s: One of our local facilitators was a PPW at a middle school. She shared with us that she watched one of her middle school principals establish a new practice. Before a teacher could refer a student to the office (removal from class) they had to be able to articulate 5 things about the student that are “non-academic” in nature. ODR’s dropped significantly and the message was clear—know our students!

Evaluation Question: Do 80% of the students in MD Elementary PBIS schools using SWIS receive 0 or 1 office referrals? Data Source: SWIS PBIS Elementary schools in Maryland using SWIS report that 92% of their students receive 0 or 1 office referral. Susan Growing the Green

Evaluation Question: Where is the location of behavior problems in Middle Schools? Data Source: SWIS The majority of the problem behaviors reported occur in the classroom followed by hallways. Susan