1. Evaluating the Performance of Specialized Professionals in a MTSS AMM - September 11, 2012

2. Evaluating Specialized Personnel in a Multi-tiered System of Supports (MTSS) Student Support Services Project, USF Bureau of Exceptional Education and Student Services (BEESS) Bureau of Educator Recruitment, Development, and Retention

3. Overview Florida’s New Evaluation System Student Services Role in a Multi-tiered Support System Overview of Student Services Personnel Evaluation Model (SSPEM) Update on Development of ESE Evaluation Model for Specialized Personnel

4. Link to Resources http://sss.usf.edu/resources/format/presentations/2012/a mm.html http://sss.usf.edu/resources/format/presentations/2012/a mm.html

5. Florida’s New Evaluation System Eileen McDaniel, Chief, Bureau of Educator Recruitment, Development, and Retention 5

6. Purpose for Personnel Evaluations As set forth in the Student Success Act and Race to the Top, teacher evaluations are: Designed to support effective instruction and student learning growth. Results used when developing district and school level improvement plans. Results used to identify professional development and other human capital decisions for instructional personnel and school administrators.

7. Purpose for Personnel Evaluations Evaluations must differentiate among 4 levels of performance: – Highly effective – Effective – Needs improvement, or for instructional personnel in first 3 years of employment, Developing – Unsatisfactory State Board of Education must establish student growth standards for each performance level (no date required). Commissioner must consult with experts, instructional personnel, school administrators and education stakeholders in developing the criteria for the performance levels.

8. Purpose for Personnel Evaluations To support those objectives, the law also sets forth that teacher evaluations are to be based on sound educational principles and contemporary research in effective practices in three major areas: 1.The performance of students 2.Instructional practice 3.Professional and job responsibilities 8

9. Two Major Components of the Evaluation System 9 Instructional Practice measured by the District’s Instructional Practice Framework

10. Instructional Practice Section 1012.34, F. S., requires that instructional practice evaluate the following: For Classroom teachers, excluding substitutes: Florida Educator Accomplished Practices (FEAPs) For Instructional personnel, not classroom teachers: FEAPs May include specific job expectations related to student support Instructional Framework goal: An expectation that all teachers can increase their expertise from year to year which produces gains in student achievement from year to year with a powerful cumulative effect 10

11. Two Major Components of the Evaluation System 11 Performance of Students is focused primarily on student learning growth Instructional Practice 50% Performance of Students 50%

12. Performance of Students Performance of Students. At least 50% of a performance evaluation must be based upon data and indicators of student learning growth assessed annually and measured by statewide assessments or, for subjects and grade levels not measured by statewide assessments, by district assessments as provided in s. 1008.22(8), F.S. -Section 1012.34(3)(a)1., Florida Statutes 12

13. 13 Performance of Students For subjects and grades not assessed by statewide assessments: By 2014-15, districts shall measure growth using equally appropriate formulas. DOE shall provide models. Allows district to request through evaluation system review process to: Use student achievement, rather than growth, or combination of growth and achievement for classroom teachers where achievement is more appropriate; For courses measured by district assessments, include growth on FCAT Reading and/or Mathematics as part of a teacher’s growth measure, with a rationale. In this instance, growth on district assessment must receive the greater weight.

14. 14 Performance of Students Section 1012.34(7)(e) For classroom teachers of courses for which there are no appropriate assessments under s. 1008.22(8), F.S., and the district has not adopted growth measures: Student growth must be measured by using results of assigned students on statewide assessments, OR If the teacher’s assigned students do not take statewide assessments, by established learning targets approved by principal that support the school improvement plan. The superintendent may assign instructional personnel in an instructional team the growth of the team’s students on statewide assessments. These provisions expire July 1, 2015.

15. Performance of Students The performance of students represents 50% of a teacher’s evaluation, with performance based on student learning growth – Growth data for 3 years of students assigned to the teacher. – If less than 3 years of data are available, years for which data are available must be used, and percentage of evaluation based on growth may be reduced to not less than 40%. To meet the above requirement, the development of a fair and transparent measure of student growth is essential.

16. Florida’s Value Added Model Overview of the Model to Measure Student Learning Growth on FCAT as developed by the Student Growth Implementation Committee 16

17. The Measure: Value-Added Analysis A value-added model measures the impact of a teacher on student learning, by accounting for other factors that may impact the learning process. These models do not: Evaluate teachers based on a single year of student performance or proficiency (status model). Evaluate teachers based on simple comparison of growth from one year to the next (simple growth). 17

18. Value-Added Example 18 The difference between the predicted performance and the actual performance represents the value-added by the teacher’s instruction. The predicted performance represents the level of performance the student is expected to demonstrate after statistically accounting for factors through a value-added model.

19. Advantages of Value-Added Models Teacher teach classes of students who enter with different levels of proficiency and possibly different student characteristics. Value-added models “level the playing field” by accounting for differences in the proficiency and characteristics of students assigned to teachers. Value-added models are designed to mitigate the influence of differences among the entering classes so that schools and teachers do not have advantages or disadvantages simply as a result of the students who attend a school or are assigned to a class.

20. Florida’s Value-Added Model Developed by Florida Educators Florida’s VAM model begins by establishing expected growth for each student: Based on historical data each year. Represents the typical growth seen among students who have earned similar test scores the past two years, and share the other characteristics identified by the committee.

21. Factors Identified by the SGIC to “Level the Playing Field” To isolate the impact of the teacher on student learning growth, the model developed by the SGIC and approved by the Commissioner accounts for: Student Characteristics Classroom Characteristics School Characteristics 21

22. Factors Identified by the SGIC to “Level the Playing Field” Student Characteristics Up to two prior years of achievement scores (the strongest predictor of student growth) The number of subject-relevant courses in which the student is enrolled Students with Disabilities (SWD) status English Language Learner (ELL) status Gifted status Attendance Mobility (number of transitions) Difference from modal age in grade (as an indicator of retention) Classroom Characteristics Class size Homogeneity of students’ entering test scores in the class School Component

23. Factors Identified by the SGIC to “Level the Playing Field” The model recognizes that there is an independent factor related to the school that impacts student learning – a school component. The model recognizes that there is an independent factor related to the school that impacts student learning – a school component. Statistically is simply the factors already controlled for in the model measured at the school level by grade and subject. May represent the impact of the school’s leadership, the culture of the school, or the environment of the school on student learning. Acts as another covariate, just like all other factors. 23

24. What does a teacher’s value-added score represent? An estimate of a teacher’s impact on student learning, after accounting for other factors that may impact learning. An estimate of a teacher’s impact on student learning, after accounting for other factors that may impact learning. A score of “0” indicates that students performed no better or worse than expected based on the factors in the model. A positive score indicates that students performed better than expected. A negative score indicates that students performed worse than expected. Individual teacher scores are expressed in terms of Developmental Scale Score (DSS) points. 24

25. What does a teacher’s value-added score represent? If a teacher’s score is “20,” for example, what does a score of 20 points mean? It means that students, on average, performed 20 DSS points higher than typical. To account for differences in the FCAT vertical scale across grade levels, subject areas, and years, a teacher’s value- added scores are aggregated into one score, and then transformed into a proportion of an “average year’s growth.” Proportion of an average year’s growth provides more context and helps describe the magnitude of the gain. 25

26. What does a teacher’s value-added score represent? Thus, if the average amount of growth in a given grade, subject, and year is 40 scale score points, transforming a score of 20 points into a proportion yields a score of 0.50 (20 divided by 40). Now, one can interpret the raw value-added score of 20 to say that on average students performed 50% higher than an average year’s growth. These analyses using historical data (and those provided to school districts) use this metric of a proportion of an “average year’s growth.” 26

27. Additional Data Provided through the VAM – Percent Meeting Expectations In addition to the value-added score, the model also yields information on the number and percent of students that met their statistical performance expectations. Though these data do not provide information on how far students improved or declined, it does provide information on the quantity of students who met their expectations. These data are used in analyzing the disaggregated performance of student subgroups. 27

28. Impact Analyses for the Preliminary 2011-12 VAM Scores Analyses were run to determine the relationship of the teacher VAM scores with various classroom characteristics, including the percentage of students in a teacher’s classroom who were disabled. In all cases, the correlations are negligible indicating no advantages or disadvantages in terms of the VAM score for teachers based on the group of students they serve. 28

29. Correlation of Teacher VAM Score and Percent Students with Disabilities 29

30. Observed Correlations with Teacher VAM Score Teacher VAM (Reading) Teacher VAM (Math) Percent SWD0.009-0.03 Percent ELL0.060.004 Percent ED-0.08 Percent Gifted0.010.003 Percent Non-White0.03-0.04 Mean Prior-0.08-0.03 30

31. Rule Development Activities Department is currently in rule development on proposed rule 6A-5.0411, Calculations of Student Learning Growth Using Statewide Assessment Data for Use in School Personnel Evaluations Six workshops were held across the state to gather input May 21, Naples May 22, Coconut Creek (Broward) May 23, Orlando June 5, St. Augustine June 6, Pensacola June 7, Conference Call 31

32. Rule Development Activities This proposed rule focuses on the statutory requirement in Section 1012.34(7) and (8), Florida Statutes, requiring the State Board of Education to adopt rules which establish: Each formula for measuring student growth that is approved by the Commissioner. Specific, discrete standards for each performance level to ensure clear and sufficient differentiation in the performance levels and to provide consistency in meaning across school districts. The measurement of student learning growth and associated implementation procedures. 32

33. Rule Development Activities 33 Seeking Feedback on: What standard should be used to evaluate and classify teachers and administrators based on VAM data? What levels of standard error should be applied in determining performance categories? What circumstances must be satisfied in order for the specific requirements in s. 1012.34(8), F.S., to apply in affecting a summative rating when the standard for learning growth is not achieved?

34. Rule Development Activities - Timeline Currently – Continue to gather input on proposed rule To provide public input, go to https://app1.fldoe.org/rules/default.aspx and submit comment under 6A-5.0411 https://app1.fldoe.org/rules/default.aspx September/October 2012 – Publish proposed rule text No earlier than December 2012 – State Board action on the rule (subject to change) Implement state standard for VAM results with the 2013-14 teacher evaluation results; Districts will continue to set their own standards for 2012-13 34

35. Student Services Role in a Multi-tiered Support System George Batsche, Project Director, Student Support Services Project, USF

36. MTSS A Multi-Tiered System of Supports (MTSS) is a term used to describe an evidence-based model of schooling that uses data-based problem-solving to integrate academic and behavioral instruction and intervention. The integrated instruction and intervention is delivered to students in varying intensities (multiple tiers) based on student need. “Need-driven” decision-making seeks to ensure that district resources reach the appropriate students (schools) at the appropriate levels to accelerate the performance of ALL students to achieve and/or exceed proficiency. “Need-driven” decision-making seeks to ensure that district resources reach the appropriate students (schools) at the appropriate levels to accelerate the performance of ALL students to achieve and/or exceed proficiency.

37. Why Organize an Evaluation System Around an MTSS Model? Research supports that an integrated (academic/behavior/social emotional) service delivery system has greater impact on student performance than separate systems Services and personnel in schools already are organized by levels of intensity of service delivery Tier 1—What everybody gets—typically general education teacher led Tier 2—What “some” get—typically more intensive, smaller groups Tier 3—What “few” get—typically most intensive, specialized

38. Why Organize an Evaluation System Around an MTSS Model? Existing and proposed statutes, regulations and practices support a multi-tiered system IDEIANCLB Learn Act Achievement Through Prevention (PBIS) Act (SB 541) Evaluation systems require clear responsibility for levels of service delivery and “stakeholders” who are one focus of the evaluation process

39. Why Organize an Evaluation System Around an MTSS Model? Instructional support staff of all types typically provide instruction/intervention at all levels (Tiers 1,2 and 3) in a school and/or district School-based research that identifies evidence-based practices is conducted at levels aligned with the Tiers School-wide (e.g., PBIS, Crisis Prevention) Classroom level (e.g., Group Procedures, Instructional Strategies Group level (e.g., academic instruction, social skills training Very Small Group/Individual (e.g., therapeutic, intense psychological skills training, academic skills)

40. Highly Effective Practices: Research High quality academic instruction (e.g., content matched to student success level, frequent opportunity to respond, frequent feedback) by itself can reduce problem behavior (Filter & Horner, 2009; Preciado, Horner, Scott, & Baker, 2009, Sanford, 2006) Implementation of school-wide positive behavior support leads to increased academic engaged time and enhanced academic outcomes (Algozzine & Algozzine, 2007; Horner et al., 2009; Lassen, Steele, & Sailor, 2006) “Viewed as outcomes, achievement and behavior are related; viewed as causes of the other, achievement and behavior are unrelated. (Algozzine, et al., 2011) Children who fall behind academically will be more likely to find academic work aversive and also find escape- maintained problem behaviors reinforcing (McIntosh, 2008; McIntosh, Sadler, & Brown, 2010) 40

41. School-wide Behavior & Reading Support The integration/combination of the two: Are critical for school success Utilize the three-tiered prevention model Incorporate a team approach at school level, grade level, and individual level Share the critical feature of data-based decision making Produce larger gains in literacy skills than the reading-only model (Stewart, Benner, Martella, & Marchand-Martella, 2007)

42. Student Services Role in an MTSS System Academic Performance of students (educator appraisal factor) is influenced significantly by social, emotional and behavior factors—the professional practices of student services personnel Combining evidence-based instructional strategies with evidence-based strategies to enhance student engagement results in the most dramatic student gains (LESSON STUDY) Enhancing student engagement (at all levels) is a primary role of students services personnel

43. Student Services Role in an MTSS System The continued viability and importance of student services personnel is influenced strongly by the impact of their practices on student performance-particularly academic performance Services provided by student services personnel have a strong, evidence-based relationship with student academic performance A blueprint for a clear, explicit relationship between the provision of evidence-based student services practices and positive student outcomes is critical in the context of school accountability Student services personnel must PLAN in such a way as to demonstrate ACCOUTABILITY and COMMUNICATE those outcomes.

44. Overview of SSPEM Developing a State Model for Student Support Services Personnel Evaluations Gria Davison and David Wheeler, Student Support Services Project, USF/BEESS

45. 45

46. 46

47. Student Support Services

48. Fundamental Principles Fundamental Purpose: Improve academic & behavioral outcomes for students Reflect a Multi-tiered System of Support (MTSS) framework. Align with evidence and research-based practices and professional standards linked to positive student outcomes. Integrate common practice standards across student services professions. Support professional growth and continuous improvement.

49. Fundamental Principles (cont.) Offer a state-approved evaluation framework that is dynamic (flexible & fluid) and complies with the Student Success Act for districts to adopt, adapt, or use as a guide.

50. Focus on “practices” component Crosswalk Professional Practice Standards with FEAPs, Professional Competencies, & Teacher/Principal models Identify Domains of practice; Practices; Indicators for each practice (levels of performance/proficiency) Research/evidence supporting practice Develop an evaluation rubric Vet model rubric with key stakeholders Developing the Model 50

51. Relevance of Professional Standards Purpose Establishes foundation Authenticates Lends credibility and agreement Integrates Research linked to positive student outcomes Evidence-based strategies Best practices

52. Bureau of Educator Recruitment & Student Support Services (BEESS) Task: Develop Process and Model Core Development Workgroup (District Coordinators from each of the Student Service disciplines) Task: Develop Draft Model Focus Group (Student Services Directors; Coordinators from Student Service disciplines; Administrators; Other stakeholders) Task: Feedback on Draft Model Contributors/Partners 52

53. Conceptual Model Domains (5 Domains) – broad categories used to organize professional practices and help structure the evaluation. Practices (25 Practices) – standards of practice within a a domain related to a specific area of professional skill. Indicators – continuum of descriptive statements that assist in differentiating levels of performance for each practice (Highly Effective, Effective, Emerging, Ineffective). 53

54. Domains of Practice Data-based Decision Making and Evaluation Instruction/Intervention Planning & Design Instruction/Intervention Delivery & Facilitation Learning Environment Professional Learning, Responsibility, & Ethics

55. Scoring Rubric for Indicators Highly Effective – practice has broader, systemic impact (school-wide/district-wide) OR facilitates effective practice of others through mentoring and/or training Effective – demonstrates the essential elements of the practice competently and independently with individual students and groups Emerging – developing practice competence but requires additional supervision, support or training Ineffective – does not demonstrate practice or demonstrates practice poorly

56. 56 Evaluation Rubric

57. 57 Evaluation Rubric

58. 58 Evaluation Rubric

59. 59 Evaluation Rubric

60. 60 Evaluation Rubric

61. Resources/Tools in Guide Research Support for Model (Appendix B) Crosswalks Professional Practice Standards (Table 2) FEAPs, Marzano, & Danielson (Table 3) Methods and Sources of Evidence for Evaluating Professional Practice (Table 1) Scoring Protocols Evaluation Rubric Scoring Protocol (Form 1) Student Growth Protocol Summative Evaluation (Form 3)

62. 62

63. 63

64. 64

65. 65

66. Recommendations for District Use The Evaluation Cycle Process SSPEM and the District Framework Student Growth Component

67. SSPEM is designed to Establish practices and expectations that are linked to student outcomes (academic & behavioral) and based on research. Develop evaluation procedures that align with professional standards and accomplished educator practices (FEAPs). Provide feedback to the professional that recognizes effective performance, identifies areas for improvement, and directs professional growth activities. Provide support to supervisees/practitioners not meeting performance expectations. 67

68. 68 Evaluation Cycle Process

69. Student Growth Component Student learning growth component must account for 50% of the evaluation (modified if less than three years of data). Must be based on students assigned to the professional. Measurable student outcomes – up to 20% of the student learning growth component may be based on measurable student outcomes specific to the position.

70. 70

71. ESE Evaluation Model for Non-classroom Personnel Bettye Hyle, Student Support Services Project, USF

72. Areas Addressed PT–Physical Therapist OT–Occupational Therapist SLP–Speech Language Pathologist PD–Professional Development Behavior Specialist Staffing Specialist Itinerant Teachers–Serving students with visual and/or hearing impairments.

73. Timeline July 2012 Identify Participants Internal Core Group Draft Domains of ESE Evaluation Model August 29, 2012 Conduct Core Group Meeting Preparation for Focus Work Group Meeting September 11, 2012 Presentation at AMM October 3, 2012 Focus Workgroup Meeting Finalize Review Rubric Draft Student Growth and Guide Submit Final Draft–ESE Personnel Eval. Model & Guide Vetting/Approval

74. FEAP and ESE Comparison Chart

76. ESE Domain/Practices Chart