1. www.engageNY.org 1 New York State Growth Model for Educator Evaluation 2011-12 June 2012 PRESENTATION as of 6/14/12

2. www.engageNY.org 2 2 Today’s Agenda  Background  The What, Why, and How of Growth Models and Measures  Using Growth Measures for Educator Evaluation  What Data Will Be Available and When?

3. www.engageNY.org 3 Background

4. www.engageNY.org 4 4 Evaluating Educator Effectiveness Student growth on state assessments (state- provided) Student learning objectives Growth 20% Student growth or achievement Options selected through collective bargaining Locally Selected Measures 20% Rubrics Sources of evidence: observations, visits, surveys etc. Other Measures 60%

5. www.engageNY.org 5 The What, Why, and How of Growth Models and Measures

6. www.engageNY.org 6 6 By the End of This Section….  You should be able to: –Explain why the state is measuring student growth and not achievement –Describe how the state is measuring growth compared to similar students –Define a student growth percentile and mean growth percentile

7. www.engageNY.org 7 7 Prior Year Performance for Students in Two Teachers’ Classrooms ─ Proficiency

8. www.engageNY.org 8 8 Current Year Performance of Same Students ─ Proficiency

9. www.engageNY.org 9 9 Prior and Current Year Performance for Ms. Smith’s Students Ms. Smith’s Class Prior ScoreCurrent Score Student A450510 Student B470500 Student C480525 Student D500550 Student E600650

10. www.engageNY.org 10 www.engageNY.org 10 ELA Scale Score 20112012 Student A 450 High SGPs Low SGPs Student A’s Current Year Performance Compared to “Similar” Students If we compare student A’s current score to other students who had the same prior score (450), we can measure her growth relative to other students. We describe her growth as a “student growth percentile” (SGP). Student A’s SGP is the result of a statistical model and in this example is 45, meaning she performed better in the current year than 45 percent of similar students.

11. www.engageNY.org 11 www.engageNY.org 11 Comparing Performance of “Similar” Students Prior Year Score Current Year Score Given any prior score, we see a range of current year scores, which give us SGPs of 1 to 99.

12. www.engageNY.org 12 www.engageNY.org 12 SGPs for Ms. Smith’s Students Ms. Smith’s Class Prior Score Current Score SGP Student A450510 45 Student B470500 40 Student C480525 70 Student D500550 60 Student E600650 40

13. www.engageNY.org 13 www.engageNY.org 13 Student Growth Percentiles: True or False? 1.A student with an SGP of 50 performed better than 50 percent of similar students. 2.A student with an SGP of 80 must be proficient. 3.A student with an SGP of 20 grew less than a student with an SGP of 60. 4.The highest SGP that a student can receive is 99. 5.A student with an SGP of 80 grew twice as much as a student with an SGP of 40. 1.True 2.False 3.False 4.True 5.False

14. www.engageNY.org 14 www.engageNY.org 14 From Student Growth to Teachers and Principals Ms. Smith’s Class SGP Student A45 Student B40 Student C70 Student D60 Student E40 To measure teacher performance, we find the mean growth percentile (MGP) for her students. To find an educator’s mean growth percentile, take the average of SGPs in the classroom. In this case: Step 1: 45+40+70+60+40=255 Step 2. 255/5=51 Ms. Smith’s mean growth percentile (MGP) is 51, meaning on average her students performed better than 51 percent of similar students. A principal’s performance is measured by finding the mean growth percentile for all students in the school.

15. www.engageNY.org 15 www.engageNY.org 15 Which Students Count in a Teacher or Principal’s MGP for 2011-12? Student has valid test scores for at least 2011-12 and 2010-11 Student scores do not count for 2011-12 Yes Student meets continuous enrollment standard for 2011-12 No Student growth is attributed to the teacher and the school Yes No Expected for 2012-13: students weighted by duration of instructional linkage

16. www.engageNY.org 16 www.engageNY.org 16 From Student Growth to Teachers and Principals  In order for an educator to receive a growth score, he or she must have a minimum sample size of 16 student scores in ELA or mathematics across all grades he or she teaches. Examples: –A teacher has a self-contained classroom with 8 students who take the 4 th grade ELA and math assessments; this teacher would then have 16 student scores contributing to his or her growth score. –A teacher has a class with 12 students who are in varied grades (4 th, 5 th, 6 th ) who take the ELA and math assessments for their respective enrolled grade level; this teacher would then have 24 student scores contributing to his or her growth score.  If an educator does not have 16 student scores, they will not receive a growth score from the State and will not receive information in the reporting system at the educator level. –Educators likely to have fewer than 16 scores should use SLOs.

17. www.engageNY.org 17 www.engageNY.org 17 MGPs and Statistical Confidence 87 Confidence Range Upper Limit Lower Limit MGP NYSED will provide a 95 percent confidence range, meaning we can be 95 percent confident that an educator’s “true” MGP lies within that range. Upper and lower limits of MGPs will also be provided. An educator’s confidence range depends on a number of factors, including: number of student scores included in their MGP and the variability of student performance in their classroom.

18. www.engageNY.org 18 www.engageNY.org 18 Pause and Reflect: Mean Growth Percentiles  We talked about: –How to find a mean growth percentile –How to interpret a mean growth percentile –What students are counted in a mean growth percentile –How many student scores are needed to provide an MGP –How a measure of statistical confidence (upper and lower limits of a 95% confidence range) will be provided with mean growth percentiles and why

19. www.engageNY.org 19 www.engageNY.org 19 Expanding the Definition of “Similar” Students  So far we have been talking about “similar” students as those with the same prior year assessment score  We will now add two additional features to the conversation:  Two additional years of prior assessment scores –Remember—a student MUST have current year and prior year assessment score to be included  Student-level factors – Economic disadvantage – Students with disabilities (SWDs) – English language learners (ELLs)

20. www.engageNY.org 20 www.engageNY.org 20 Adjustments for Three Student-Level Factors in Measuring Student Growth Student performance Teacher Instruction Other factors (12-13) Economic disadvantage Language proficiency Disability

21. www.engageNY.org 21 www.engageNY.org 21 ELA Scale Score 20112012 Student A 450 High SGPs Low SGPs Student A’s Current Year Performance Compared to “Similar” Students If we compare student A’s current score to other students who had the same prior score (450), we can measure her growth relative to other students. We describe her growth as a “student growth percentile (SGP”). Student A’s SGP is the result of a statistical model and in this example is 45, meaning she performed better in the current year than 45 percent of similar students.

22. www.engageNY.org 22 www.engageNY.org 22 ELA Scale Score 20112012 Student A 450 High SGPs Low SGPs Expanding the Definition of “Similar” Students to Include Economically Disadvantaged—An Example Now if student A is economically disadvantaged, we compare student A’s current score to other students who had the same prior score (450) AND who are also economically disadvantaged. In this new comparison group, we see that student A now has an SGP of 48.

23. www.engageNY.org 23 www.engageNY.org 23 Further Information on Including Student Characteristics in the Growth Model  The following slides were developed using sample data from 2010-2011. –The “combined” MGPs on the charts have been calculated at the educator level (combining all grades and subjects). –Not all districts provided data linked to teachers for grades 4-8 ELA/Math in 2010-11.

24. www.engageNY.org 24 www.engageNY.org 24 Teacher MGPs after Accounting for Economic Disadvantage Taking student-level characteristics into account helps ensure educators with many students with those characteristics have a fair chance to achieve high or low MGPs. For example, note that for teachers with any percent of economically disadvantaged students, teacher MGPs range from 1 to 99. NOTE: Beta results using available 2010-2011 data.

25. www.engageNY.org 25 www.engageNY.org 25 Teacher MGPs after Accounting for SWD NOTE: Beta results using available 2010-2011 data.

26. www.engageNY.org 26 www.engageNY.org 26 Teacher MGPs after Accounting for ELL Percent of ELL Students in Class NOTE: Beta results using available 2010-2011 data.

27. www.engageNY.org 27 www.engageNY.org 27 “Similar” Students: A Summary “Similar” Student Characteristics Unadjusted Mean Growth Percentiles Adjusted Mean Growth Percentiles Up to Three Years of Prior Achievement English Language Learner (ELL) Status Students with Disabilities (SWD) Status Economic Disadvantage Reported to Educators Used for Evaluation

28. www.engageNY.org 28 www.engageNY.org 28 One Last Feature of the Growth Model…. All tests contain measurement error, with greater uncertainty for highest and lowest-achieving students The New York growth model accounts for measurement error in computing student growth percentiles.

29. www.engageNY.org 29 www.engageNY.org 29 State Growth Model Summary Regulations allow Prior years of student test results Three student- level variables: SWD, ELL, Econ Disadvantage Measurement error correction Model includes Up to three years, as available All three Measurement error correction Growth model for 2011-12 only for grades 4-8 ELA/Math for teachers and principals

30. www.engageNY.org 30 www.engageNY.org 30 By the End of This Section….  You should be able to: –Explain why the state is measuring student growth and not achievement  Allow teachers to achieve high ratings regardless of incoming levels of achievement of their students –Describe how the state is measuring growth compared to similar students  Similar students: Up to three years of the same prior achievement, three student-level characteristics (economic disadvantage, SWD, and ELL status) –Define a student growth percentile and mean growth percentile  An SGP (student growth percentile) is a measure of a student’s growth relative to similar students  An MGP (mean growth percentile) is the average of the student growth percentiles attributed to a given educator

31. www.engageNY.org 31 Using Growth Measures for Educator Evaluation

32. www.engageNY.org 32 www.engageNY.org 32 By the End of This Section….  You should be able to: –Explain how growth ratings and scores will be obtained, using illustrative data

33. www.engageNY.org 33 www.engageNY.org 33 Growth Ratings and Score Ranges Growth RatingDescriptionGrowth Score Range (2011-12) Highly Effective Well-above state average for similar students 18-20 EffectiveResults meet state average for similar students 9-17 DevelopingBelow state average for similar students 3-8 IneffectiveWell-below state average for similar students 0-2

34. www.engageNY.org 34 www.engageNY.org 34 Distribution of 10-11 Teacher-Level MGPs MGP 1 50 99 Number of Teachers NOTE: Beta results using available 2010-2011 data. For illustrative purposes only Distribution of Mean Student Growth Percentiles (Teacher Level) Percent of MGPs

35. www.engageNY.org 35 www.engageNY.org 35 MGPs and Statistical Confidence 87 Confidence Range Upper Limit Lower Limit MGP NYSED will provide a 95 percent confidence range, meaning we can be 95 percent confident that an educator’s “true” MGP lies within that range. Upper and lower limits of MGPs will also be provided. An educator’s confidence range depends on a number of factors, including: number of student scores included in their MGP and the variability of student performance in their classroom.

36. www.engageNY.org 36 www.engageNY.org 36 Illustrating Possible Growth Ratings MGP 1 MGP 99 Well Below Average Average Well Above Average MGP 50 MGP

37. www.engageNY.org 37 www.engageNY.org 37 Illustrating Possible Growth Ratings MGP 1 MGP 99 Well Below Average Average Well Above Average MGP 50 MGP

38. www.engageNY.org 38 www.engageNY.org 38 Yes No Yes From MGPs to Growth Ratings Mean Growth Percentile >= Well Above Average Lower Limit > 50 Highly Effective: Results are well- above state average for similar students Mean Growth Percentile <= Well Below Average Upper Limit < 50 Ineffective: Results are well- below state average for similar students Developing: Results are below state average for similar students No Effective: Results equal state average for similar students Mean Growth Percentile Confidence RangeGrowth Rating Mean Growth percentile = Average Yes Any Yes

39. www.engageNY.org 39 www.engageNY.org 39 Illustrating Possible Growth Ratings Ineffective Developing Highly Effective Effective MGP 1 MGP 99 Well Below Average Average Well Above Average MGP 50 MGP Developing

40. www.engageNY.org 40 www.engageNY.org 40 Illustrating Possible Growth Ratings Effective MGP 1 MGP 99 Well Below Average Average Well Above Average MGP 50 MGP

41. www.engageNY.org 41 www.engageNY.org 41 By the End of This Section….  You should be able to: –Explain how growth ratings and scores are obtained  Step 1. Determine if an educator’s MGP is well above or well below the state average  Step 2. Determine if an educator’s MGP is above or below the state average

42. www.engageNY.org 42 What Data Will Be Available and When

43. www.engageNY.org 43 www.engageNY.org 43 Data – What to Expect When Growth scores provided to districts Mid-July Test scores finalized and teacher linkage data final submission Early fall Mid-August Online reporting system available

44. www.engageNY.org 44 www.engageNY.org 44 Data – What to Expect in August Data Elements (for teachers and schools)  Unadjusted mean growth percentiles (Unadjusted MGPs)  Adjusted mean growth percentiles (Adjusted MGPs and upper and lower limits based on confidence range for these adjusted MGPs)  Percent of students above the median  Number of student scores included  Growth rating (HEDI)  Growth score (0-20) Breakdowns (by teacher and school)  MGPs by subject, grade, and overall (not HEDI)  Overall MGPs for subgroups – ELL, SWD, Economic Disadvantage, High- and Low-Achieving –Subgroup scores will not be included on reports if there are fewer than 16 student scores

45. www.engageNY.org 45 www.engageNY.org 45 One Teacher’s Information - August Number of Student Scores Percent of Students Above the State Median Unadjusted MGP Adjusted MGP Growth Rating Growth Score Lower LimitUpper Limit Jane Smith 5660 70 75 65 85 Highly Effective 18 Number of Student Scores Percent of Students Above the State Median Unadjusted MGP Adjusted MGP Lower LimitUpper Limit Jane Smith Math Math Grade 4 287075 786588 ELA ELA Grade 4 2850656855 79 Students with disabilities 4* **** English language learners 0 * * * * * Economically disadvantaged 2* **** Low achieving (Level 1) 4* **** High achieving (Level 4) 4* ****

46. www.engageNY.org 46 www.engageNY.org 46 Number of Students Percent of Students Above the Median Unadjusted MGP Adjusted MGP Lower LimitUpper Limit Jane Smith Math Math Grade 4 287075 786588 ELA ELA Grade 4 2850656855 79 Students with disabilities 4* **** English language learners 0* ** * * Economically disadvantaged2* **** Low achieving 4* **** High achieving 4* **** Adjusted MGP: 75 Number of Students Percent of Students Above the Median Unadjusted MGP Adjusted MGP Growth Rating Growth Score Lower LimitUpper Limit Jane Smith 5660 70 75 65 85 Highly Effective 18 2 SWD students, 0 ELL students, 1 econ disadvantaged student, 2 high- and 2 low-achieving students Math number of student scores: 28 ELA number of student scores: 28 Total number of student scores: 56 Upper and lower limits of adjusted MGP: 65 and 85 Growth rating of highly effective and growth score of 18 60 percent of students above the State Median Unadjusted MGP: 70 One Teacher’s Information - August Unadjusted and adjusted MGPs by subject No scores reported here since fewer than 16 student scores in a sub-group

47. www.engageNY.org 47 www.engageNY.org 47 Scavenger Hunt and Quiz 1.What is Ms. Smith’s overall adjusted MGP? 2.What are the upper and lower confidence limits for Ms. Smith’s overall MGP and what do they represent? 3.How many scores are included from Ms. Smith’s class for ELA? 4.What is the adjusted MGP for Ms. Smith’s class in ELA? 5.How do Ms. Smith’s high- achieving students compare to her low-achieving students in terms of growth? 1.75 2.65 and 85; represent 95 percent confidence range around Ms. Smith’s overall MGP score 3.28 4.68 5.No scores reported for high- and low-achieving groups since too few student scores (less than 16 for reporting purposes) are included

48. www.engageNY.org 48 www.engageNY.org 48 Definitions  SGP (student growth percentile): the result of a statistical model that calculates each student’s change in achievement between two or more points in time on a State assessment or other comparable measure and compares each student’s performance to that of similarly achieving students  Similar students: students with the same prior test scores, ELL, SWD, and economic disadvantage status  ELLs: English language learners  SWD: students with disabilities  Economic disadvantage: a student who participates in, or whose family participates in, economic assistance programs such as the Free- or Reduced-price Lunch Programs (FRPL), Social Security Insurance (SSI), Food Stamps, Foster Care and others

49. www.engageNY.org 49 www.engageNY.org 49 Definitions  High-achieving, low-achieving: defined by the performance of students based on prior year State assessment scores (i.e., Level 1 = low-achieving, Level 4 = high-achieving)  MGP (mean growth percentile): the average of the student growth percentiles attributed to a given educator  “Unadjusted” MGP: an MGP based on SGPs for which ELL, SWD, and economic disadvantage status have NOT been accounted  “Adjusted” MGP: an MGP based on SGPs for which ELL, SWD, and economic disadvantage status have been accounted  Growth rating: HEDI rating based on growth  Growth score: growth subcomponent points from 0-20

50. www.engageNY.org 50 www.engageNY.org 50 Definitions  Measurement error: uncertainty in test scores due to sampling of content and other factors  Standard error: a measure of the statistical uncertainty surrounding a score  Upper/lower limit: highest and lowest possible MGP taking statistical confidence into account  Confidence range: range of MGPs within which we have a given level of statistical confidence that the true MGP falls (95 percent statistical confidence level used for state growth measure)

51. www.engageNY.org 51 www.engageNY.org 51