Northern California PBIS Symposium November 18, 2013

Overview of available surveys/tools on the PBIS Assessment website Developing an Evaluation Action Plan Decisions on which tools to choose Decisions on timelines Assessment Example Plans Developing Evaluation Reports Questions/Comments

School-wide Positive Behavior Support: Implementers' Blueprint and Self-Assessment Sugai, G., Horner, R.H., Algozzine, R., et.al. (2010). Evaluation Blueprint for School-Wide Positive Behavior Interventions & Support Algozzine, R., Horner, R.H, Sugai, G., et.al. (2010). Fidelity Measures to Improve Implementation of Positive Behavioral Support Tobin, T. J., Vincent, C. G., Horner, R. H., Rossetto Dickey, C., & May, S. A. (2012).

“A hallmark of School-wide Positive Behavior Support (SWPBS) is a commitment to formal evaluation.” “Evaluation is the process of collecting and using information for decision making.

Evaluation is the process of gathering data to Answer questions Make decisions Define the core questions/decisions, before selecting measures. Provide an adequate context for interpreting the data. Effective evaluation is cyclical (repeated) Use the evaluation process for improvement

Core Questions Evaluation Plan Data Collection Evaluation Report

Context What are/were the goals and objectives for SWPBIS implementation? (state/district capacity; school adoption; student outcomes) Who delivered the training and technical assistance for SWPBIS implementation? Input What professional development events were part of SWPBIS implementation support? Was projected level of TA capacity provided (training/coaching)? Who received the professional development (schools/ cohorts)?

Fidelity To what extent was Tier I SWPBIS implemented with fidelity ? To what extent were Tier II and Tier III SWPBIS implemented with fidelity? To what extent is the Leadership Team establishing core functions?

Leadership Team Active Coordination Funding Visibility Political Support TrainingCoachingEvaluation Local School/District Teams/Demonstrations Behavioral Expertise Policy Sugai et al.,

Impact To what extent is SWPBIS associated with changes in student behavioral outcomes? To what extent is SWPBIS associated with changes in academic performance, and dropout rates? To what extent is district/state capacity established? (local training, coaching, evaluation, behavioral expertise) To what extent is leadership and policy structure established?

Replication, Sustainability, and Improvement To what extent do Fidelity and Impact outcomes sustain across time? To what extent does initial SWPBIS implementation affect Implementation with later cohorts? To what extent did SWPBIS implementation change educational/behavioral capacity/policy? To what extent did SWPBIS implementation affect systemic educational practice?

Fidelity To what extent was SWPBS implemented as designed? To what extent was SWPBS implemented with fidelity? “Are we doing what we said we would do?”

PBIS Applications.org PBIS Assessment PBIS Evaluation Help us answer the question: “Are we doing what we said we would do?”

Research Tool Annual Assessment Tool Progress Monitoring Tool Universal System School-wide Evaluation Tool (SET) Benchmarks of Quality (BoQ) Early Childhood BoQ (ECBoQ) PBIS Self-Assessment Survey (SAS) Team Implementation Checklist (TIC) Secondary & Tertiary Systems Benchmarks of Advanced Tiers (BAT) PBIS Self-Assessment Survey (SAS) Monitoring Advanced Tiers Tool (MATT) Outcome Measures: School Safety Survey

Research Tool Annual Assessment Tool Progress Monitoring Tool Universal System School-wide Evaluation Tool (SET) Benchmarks of Quality (BoQ) Early Childhood BoQ (ECBoQ) PBIS Self-Assessment Survey (SAS) Team Implementation Checklist (TIC) Secondary & Tertiary Systems Individual Student School- wide Evaluation Tool (ISSET) Benchmarks of Advanced Tiers (BAT) PBIS Self-Assessment Survey (SAS) Monitoring Advanced Tiers Tool (MATT) Outcome Measures: School Safety Survey

A Tour of at

Decisions: Evaluation Tools Which tools answer your district/schools’ evaluation questions? Evaluation Timeline When/how often will you administer the tools? Evaluation Reporting How will you report outcomes to schools and to the District? How often will you report outcomes to schools and to the District?

Decisions about Evaluation Tools Does your District/School need a research tool? (SET/ISSET) Which annual assessment tool(s) will you use? BoQ, SAS, SSS, BAT, ECBoQ Which progressing monitoring tool(s) will you use? TIC, MATT Does your District/School need to evaluate Tier I or Tier II & III implementation or all three?

Evaluation Questions What is our baseline, and what has been our progress through the year? Are we improving on our implementation of PBIS? How do our individual staff members rate our implementation? Are our self-assessments consistent with data from outside evaluators? Newly implementing, grant needing specific research data as well as fidelity of implementation data

MeasureYear OneYear TwoYear Three FallWinterSpringFallWinterSpringFallWinterSpring SETXXXX TICXXXXXXXXX Safety Survey XXXX SAS XXXX

Evaluation Questions In what specific PBIS areas are we strong, and in which areas do we need more training? Are we improving on our implementation of PBIS? How do our staff members actually rate our PBIS implementation? How is our implementation of Tiers II and III? Implementing PBIS for three years, have been using the TIC/SET, and have been consistently over 80%.

MeasureYear OneYear TwoYear Three FallWinterSpringFallWinterSpringFallWinterSpring BoQXXX SAS XXX Safety Survey XXX ATTXXXXXXXXX

Evaluation Questions Which schools are not meeting the SET benchmark? In what specific PBIS areas are we strong, and in which areas do we need more training? How can we tailor our training and evaluation to meet the needs of various schools? Are we improving on our implementation of PBIS? How do our individual staff members rate our implementation? Implementing for five years, some schools are doing very well, but others are not making progress.

MeasureYear OneYear TwoYear Three FallWinterSpringFallWinterSpringFallWinterSpring SAS SurveyX SET XXX Safety Survey XXX TICXXXXXXXX X Schools still not performing well

MeasureYear OneYear TwoYear Three FallWinterSpringFallWinterSpringFallWinterSpring SAS SurveyX BoQ XXX Safety Survey XXX MATTXXXXXXXXX Schools performing well

Evaluation Questions In what specific PBIS areas are we strong, and in which areas do we need more training? In what academic areas are we strong, and in which areas do we need more improvement? How can we tailor our training/evaluation to meet the needs of various schools? How can we schedule our evaluations so that we are not overwhelming our schools? The district wants to bring RtI Behavior and Academics together and create one evaluation plan.

Assessment Instrument Fall (August/Sept/Oct) Winter (Dec/Jan) Spring (April/May) Purpose Team Implementation Checklist (TIC) XXX Fidelity of Implementation Self-Assessment Survey X Fidelity of implementation School Safety Survey X Outcomes related to Protective and Risk Factors SET X Fidelity of Implementation AIMsWeb XXX Screening Progress Monitor SAT10 X Summative Evaluation DIBELS XXX Screening Progress monitoring Nieves Flores, 2012

Questions to Consider: Which surveys are your schools/districts currently using? Do they answer your evaluation questions: Are we implementing SWPBIS with fidelity? Is there more information that we need? Pbisapps.org > PBIS Assessment > Find Out More about Surveys Do your surveys assess the Tiers you are implementing? Do you need a research tool?

Are you “topping out” of the surveys you are currently taking? Do you need to customize your evaluation plan for different schools, or will one evaluation plan fit for all schools? Do you want to assess all tiers of PBIS implementation? Is your Evaluation Plan manageable for schools?

MeasureYear OneYear TwoYear Three FallWinterSpringFallWinterSpringFallWinterSpring

From the PBIS Evaluation Blueprint Context Input Impact on SWPBIS Fidelity Impact on Student Outcomes Implications for the Future

Define SWPBIS Define goals of the specific project Define stakeholders and evaluation questions

Who received what support and from whom? Leadership team Local Capacity Building Training, Coaching, Behavioral Expertise, and Evaluation School teams

Leadership Team SWPBIS District Leadership Team Self-Assessment Capacity Development Number of trainers/coaches available to support school teams/districts Behavioral expertise available to support Tier II and Tier III implementation Evaluation capacity School Teams Tier I Implementation (TIC, BoQ, SET, SAS) Collectively and/or by training cohort Tier II / Tier III Implementation (MATT, BAT, ISSET) Collectively and/or by training cohort

Behavioral Outcomes Office Discipline Referrals Suspension/Expulsion Attendance Drop Out/Graduation Disproportionality in ODR and suspension/expulsion Academic outcomes Curriculum Based Measures Oral Reading Fluency, Dibels, AimsWeb Standardized Measures State and National Math, reading, writing assessments

Advances to date Plans for next three years Areas in need of adaptation Narrowing of evaluation questions and concerns

State and District examples at PBIS.org Send in your district examples, so we can all learn!

Portal Support Section Search Function Community Forum Chat with us online Account Questions: Technical Assistance Questions: New phone number

When you leave the webinar, please complete the survey.