1. Measuring Implementation: School-Wide Instructional Staff Perspective Amy Gaumer Erickson, Ph.D. University of Kansas Evaluator: Kansas & Missouri SPDGs SIG Evaluators Webinar March 17, 2011 Gaumer Erickson (2011)

2. What types of measures are implemented?  Self-Report  Interviews/Focus Groups  Observations  Retrospective/Reflective Learning  Goal Attainment  Ratings of Fidelity  Outcome Data Gaumer Erickson (2011)

3. What perspectives do we need to get a complete picture of implementation in schools?  Measures/observations completed by:  Implementation Coaches  School Leadership Teams  All School Instructional Staff  Students  Families/Communities Gaumer Erickson (2011)

4. Measures completed by Implementation Coaches  SW-PBS Benchmarks of Quality (BoQ)  MiBLSI: http://miblsi.cenmi.org/MiBLSiModel/Evalua tion/Measures/BenchmarksofQuality.aspx http://miblsi.cenmi.org/MiBLSiModel/Evalua tion/Measures/BenchmarksofQuality.aspx  SW-PBS School-wide Evaluation Tool (SET)  OSEP Technical Assistance Center on PBIS: http://www.pbis.org/evaluation/evaluation _tools.aspx http://www.pbis.org/evaluation/evaluation _tools.aspx Gaumer Erickson (2011)

5. Measures completed by School Leadership Teams  SW-PBS Team Implementation Checklist (TIC)  PBIS Surveys: http://www.pbssurveys.org/pages/Home.aspx http://www.pbssurveys.org/pages/Home.aspx  Effective Behavior Support Team Implementation Checklist (EBS-SAS)  PBIS Surveys: http://www.pbssurveys.org/pages/Home.aspxhttp://www.pbssurveys.org/pages/Home.aspx  Planning and Evaluation Tool for Effective Reading Supports – Revised (PET-R)  DIBELS Data System: http://dibels.uoregon.edu/resources/ http://dibels.uoregon.edu/resources/  Implementation Rubrics  Illinois PBIS Network: http://www.pbisillinois.org/http://www.pbisillinois.org/ Gaumer Erickson (2011)

6. Student Measures  DIBELS  DIBELS Data System: https://dibels.uoregon.edu/resources.php https://dibels.uoregon.edu/resources.php  AIMSweb  AIMSweb: http://www.aimsweb.com/http://www.aimsweb.com/  School-Wide Information System (SWIS)  SWIS: http://www.swis.org/http://www.swis.org/  State Performance Plan APR Data (Indicators 1, 2, 3, 4, 5, 13, 14)  Student Perceptions Gaumer Erickson (2011)

7. Family/Community Measures  Parent Satisfaction  Missouri Improvement Program Advanced Questionnaire http://dese.mo.gov/divimprove/sia/dar/advan ce_questionnaire_surveys.html http://dese.mo.gov/divimprove/sia/dar/advan ce_questionnaire_surveys.html  Family as a Teacher Inventory  Parent-Teacher Involvement Questionnaire  Fast Track Project http://www.fasttrackproject.org/data- instruments.php http://www.fasttrackproject.org/data- instruments.php  Interviews/Focus Groups Gaumer Erickson (2011)

8. Measures completed by All School Instructional Staff  Self-Assessment Survey (SAS)  PBIS Surveys: http://www.pbssurveys.org/pages/Home.as px http://www.pbssurveys.org/pages/Home.as px  Standards Assessment Inventory  National Staff Development Council http://www.learningforward.org/standards/ sai.cfm http://www.learningforward.org/standards/ sai.cfm  Classroom/Teacher Observations Gaumer Erickson (2011)

9. Which perspective do you think your grants measure most effectively?  A: Implementation Coaches  B: School Leadership Teams  C: All School Instructional Staff  D: Stakeholders: Student/Parents/Communities Gaumer Erickson (2011)

10. School Staff Survey Design  Identified essential features of school reform initiatives (e.g., RTI, SW-PBS, PLCs, High Schools that Work, Reading First)  Analyzed other measures completed by school instructional staff (e.g. PBIS Self-Assessment Survey, RTI Implementation Tool, National Staff Development Council Standards Assessment Inventory, Missouri School Improvement Process Faculty Advanced Questionnaire)  Wrote over 100 items and then narrowed it down to 33 through pilot testing with schools and analysis by a state evaluation team  Implemented survey in 14 schools Gaumer Erickson (2011)

11. School Staff Survey Reliability  Overall Alpha: 0.98  Three Factors  School Implementation (17 items): 0.95  Classroom Implementation (10 items): 0.85  Individual Student Implementation (5 items): 0.90  Four Groups  Teachers (N=294): 0.96  Administrators (N=8): 0.90  Other Certified Staff (N=20): 0.97  Noncertified Staff (N=6): 0.85 Gaumer Erickson (2011)

12. School Implementation 1) I can summarize the school's shared vision/mission. 2) I have a clear understanding of the phrase: “tiered levels of academic & behavior support.” 3) I feel that my administrators are committed to implementing tiered levels of academic & behavior supports. 4) I receive coaching/mentoring to implement evidence-based instructional practices. 5) I think my school does a good job of addressing the academic & behavior needs of students at tier 1 (universal). 6) I think my school does a good job of addressing the academic & behavior needs of students at tier 2 (small group). 7) I think my school does a good job of addressing the academic & behavior needs of students at tier 3 (intensive). 8) I regularly see students move between tiers of support as their academic needs change. 9) I regularly see students move between tiers of support as their behavior needs change. 10) I am involved in meetings where data results are discussed & problem solving occurs. 11) I receive school-wide academic & behavior data in usable & understandable formats. 12) I participate in professional development where I learn how to monitor students' progress & use progress monitoring data. 13) I have the time necessary to analyze student data & problem solve with my colleagues. 14) I think my school is a good place to work. 15) I think my school has an effective process in place to identify available resources (e.g., materials, technology, people). 16) I evaluate the effectiveness of core instruction based on progress monitoring data. 17) I think that the current behavior/academic programs in my school are improving education for students in my school. Gaumer Erickson (2011)

13. Classroom Implementation 1) I use assessment data at least three times a year to monitor students' progress. 2) I adapt the environment, curriculum, & instruction based on each student’s academic & behavior data. 3) I participate in professional development where I learn ways to improve my instructional practices. 4) I receive coaching/mentoring to help me implement tiered levels of academic & behavior support. 5) I have the technology & resources that I need to provide effective instruction. 6) I am able to meet the students' diverse needs. 7) I consider my students' background when I teach and/or interact with students. 8) I regularly communicate with families regarding student academic & behavior goals/progress. 9) I make informed decisions based on feedback from families. 10) I collaborate with my colleagues on a regular basis. Gaumer Erickson (2011)

14. Individual Student Implementation 1) I am involved in action planning tiered supports with the other staff & administrators at my school. 2) When I am concerned about a student’s academic success, I collaborate with a team to identify intervention. 3) I feel that the team that addresses academic needs provides valuable feedback & makes informed decisions. 4) When I am concerned about a student’s behavior success, I collaborate with a team to identify intervention. 5) I feel that the team that addresses behavioral needs provides valuable feedback & makes informed decisions. 6) I think my school does a good job in including parents as team members in data-based decision-making. Gaumer Erickson (2011)

15. Reporting Results  School Summaries (to support data-based decision-making and action planning)  School Comparisons Across Years  Dissemination & Evaluation of Program Effectiveness  SPDG APR Gaumer Erickson (2011)

20. SPDG APR Reporting Percent of school personnel involved in the MIM who report the use of data-driven decision making; [correlated with SPP Indicator 3; Program Performance Measure 1.2]. To identify the level of data-driven decision making, school staff were asked to rank the statement, I adapt the curriculum, instruction, & environment based on each student’s academic & behavior data, on the MIM School Staff Survey. The results show that 79% of MIM school staff report using data to make adaptations for students. In addition, school staff was asked to rank the following statements related to data-driven decision making:  I use assessment data at least three times a year to monitor students' progress. Out of 429 school staff who responded, 351 (82%) ranked this statement as a 4 or 5.  I am involved in meetings where data results are discussed & problem solving occurs. Out of 444 school staff who responded, 302 (68%) ranked this statement as a 4 or 5.  My students move between tiers of support as their academic needs change. Out of 424 school staff who responded, 247 (58%) ranked this statement as a 4 or 5.  My students move between tiers of support as their behavior needs change. Out of 410 school staff who responded, 204 (50%) ranked this statement as a 4 or 5. Gaumer Erickson (2011)

21. Discussion  What other methods/tools are you using to obtain data from all school instructional staff?  Is a measure like the School Staff Survey something that would be helpful on your projects? Gaumer Erickson (2011)

22. For More Information: Amy Gaumer Erickson, Ph.D. University of Kansas aerickson@ku.edu 785-864-0517 Gaumer Erickson (2011)