1. 1 Implementing a Three-Tiered State Evaluation Structure Bob Putnam The May Institute Karen Childs University of South Florida 2009 National PBIS Leadership Forum

2. Goals of Session Present the purpose of a state-wide evaluation Provide exemplars of the core components of three tier state-wide evaluations Provide an ongoing state-wide evaluation system

3. Purpose of Statewide SWPBS Evaluation Evaluation is the process of gathering information for decision-making. SWPBS decisions focus on levels of adoption, adaptation, and replication. –Efficiency –Effectiveness –Sustainability –Wide-spread adoptability 3

5. Core Indicators These indicators include considerations about the –context in which implementation of SWPBS is to occur –inputs available to guide and assist with implementation –fidelity with which core elements of SWPBS are put in place –impact of those core elements on the social and academic behavior of students. Assess the breadth of adoption, and sustainability of implementation. 5 Algozzine, Horner, Sugai, Barrett, Eber, Kincaid & Lewis, 2009

6. Context What are/were the goals and objectives for SWPBS implementation? Who provided support for SWPBS implementation? Who received support during SWPBS implementation? 6

7. Context Who received support during SWPBS implementation?

8. Input What professional development was part of SWPBS implementation support? Who participated in the professional development? What was the perceived value of the professional development? 8

9. Participating Schools 2004 Schools (21) 2005 Schools (31) 2006 Schools (50) 2000 Model Demonstration Schools (5) 2008 Schools (95) Who Is Receiving Training and Support?

10. Training/Technical Assistance More than three-fourths of the counties in the state have at least one school participating in the North Carolina Positive Behavior Support Initiative. Who Is Receiving Training and Support?

11. Tier 2 & Tier 3 Who Is Receiving Training and Support?

12. Tier 2 & Tier 3 Who Is Receiving Training and Support?

13. Tier 3 Who Is Receiving Training and Support?

14. Fidelity To what extent was SWPBS implemented as designed? To what extent was SWPBS implemented with fidelity? 14

15. School-Wide Evaluation Tool 35 Coaches trained as SET assessors 15 Contractual SET assessors 97 SETs completed 2004 154 SETs completed 2005 157 SETs completed 2006 104 schools have at least two SET scores 80% Total score is considered Maintenance Phase (IPI) All regions met 80% criterion across schools 69% increase after one year of implementation To What Extent Have Practices Changed?

16. SET Scores by Region To What Extent Have Practices Changed?

18. www.pbisillinois.org

19. Tier 3 To What Extent Have Practices Changed? www.pbisillinois.org

20. Tier 3 To What Extent Have Practices Changed? www.pbisillinois.org

21. Impact To what extent is SWPBS associated with changes in student outcomes? To what extent is SWPBS associated with changes in other areas of schooling? 21

22. What extent is SWPBS associated with changes in student behavior? www.pbisillinois.org

23. What extent is SWPBS associated with changes in student behavior? www.pbisillinois.org

24. Tier 2 & 3 What extent is SWPBS associated with changes in student outcomes? www.pbisillinois.org

25. Percent of Students at DIBELS Benchmark (Spring) and Major Discipline Referrals per 100 Students To What Extent has Behavior and Academic Performance Changed?

26. MiBLSi Schools and Reading MEAP: Average Total Office Discipline Referrals per 100 Students per Day 2004-2005 To What Extent has Academic Performance Changed?

27. MEAP- 4 th grade Reading Assessment 29 Elementary Schools In Michigan Schoolwide: Over 55% of major discipline referrals from classroom Schoolwide: Under 55% of major discipline referrals from classroom Probability of scoring below 75% proficiency on 4 th grade MEAP (Reading):.78 Probability of scoring above 75% proficiency on 4 th grade MEAP (Reading):.75 To What Extent has Academic Performance Changed?

28. North Carolina Positive Behavior Support Initiative [A]chievement causes [B]ehavior? [B]ehavior causes [A]chievement? [Context causes [A]chievement and [B]ehavior? To What Extent has Academic Performance Changed?

29. Replication, Sustainability, and Improvement To what extent did SWPBS implementation improved capacity for the state/region/district to replicate, sustain, and improve behavior and other outcomes? To what extent did SWPBS implementation change educational/behavioral policy? To what extent did SWPBS implementation affect systemic educational practice? 29

30. What is the breadth of adoption, and sustainability of implementation? www.pbisillinois.org

31. What is the breadth of adoption, and sustainability of implementation? www.marylandpbis.org

32. Conclusions Increase of states reporting on the implementation of SWPBS Information on implementation and outcomes of Tier 1 interventions more available than Tier 2 and 3 interventions Research is needed on the types of and presentations of data included in state evaluations that will have best outcome on funding, visibility, political support and policy.

33. For More Information Bob Putnam, Senior Vice President of School Consultation May Institute Phone: (781) 437-1207) Email: bputnam@mayisnstitute.org Website: http:/mayinstitute.org Karen Elfner Childs, Research & Evaluation Coordinator Florida’s PBS Project Phone: (813) 974-7358 Email: childs@fmhi.usf.edu Website: http://flpbs.fmhi.usf.eduhttp://flpbs.fmhi.usf.edu