1. DRE Agenda Student Learning Growth – Teacher VAM – School Growth PYG Area Scorecards. PYG, and other Performance Indicators

2. DRE Florida Department of Education’s “Value-Added Model (VAM)” Marc Baron, Chief, Performance Accountability Mark Howard, Director, Research, Evaluation, and Assessment

3. DRE FLDOE Value-Added Model Established by Student Success Act (SB 736) Evaluation Criteria for All Instructional Personnel and School Administrators –P–Professional Practice –S–Student Learning Growth FCAT Classroom Teachers: 40%-50% Non FCAT Classroom Teachers: 30%-40% Non Classroom Teachers: 20%-30% Principals: 50% JTEC & Principal Evaluation Committee

4. SDPBC Must Classify Instructional Personnel and School Administrators into these Four Categories – Highly Effective – Effective – Needs Improvement – Unsatisfactory

5. DRE VAM & Student Learning Growth Contribution to a change in a student’s achievement Calculated from a measure of student learning growth

6. DRE The difference between Current test score and Predicted test score What is the Predicted Student Score? What is the Student Learning Growth Score?

7. DRE It is score you would predict a student to achieve based on the student’s performance on prior tests and other information available on the student What is the Predicted Student Score?

8. DRE FLDOE Value-Added Model Variables determining student learning growth Two or more years of prior achievement scores Gifted status Student Attendance (Days) Mobility (number of transitions) Difference from modal age in grade (indicator of retention) Class size The number of subject-relevant courses in which the student is enrolled Homogeneity of entering test scores in the class

9. DRE FLDOE Value-Added Model Variables determining student learning growth Students with Disabilities (SWD) status – Language impaired – Hearing impaired – Visually impaired – Emotional/behavioral – Specific learning disability – Dual sensory impaired – Autism spectrum disorder – Traumatic brain injury – Other health impaired – Intellectual disability English Language Learner (ELL) status – LY

10. DRE FLDOE Value-Added Model Variables not determining student learning growth Gender Race Ethnicity Socioeconomic status

11. DRE FLDOE Value-Added Model Variables determining student learning growth TERMS District Data Base of All Student Records

12. DRE What is the Predicted Student Score? CURRENT TEST PRIOR TEST …100 200 300 400 500 600 700 800 900 1000… (FY10 FCAT) (FY11 FCAT)

13. DRE What is the Predicted Student Score? CURRENT TEST PRIOR TEST 900 800 700 (FY10 FCAT) (FY11 FCAT) …100 200 300 400 500 600 700 800 900 1000…

14. DRE What is the Predicted Student Score? CURRENT TEST PRIOR TEST 900 800 700 600 500 400 300 (FY10 FCAT) W (FY11 FCAT) …100 200 300 400 500 600 700 800 900 1000…

15. DRE What is the Predicted Student Score? CURRENT TEST PRIOR TEST 900 800 700 600 500 400 300 PREDICTED SCORE 600 Answer: Average of current scores of similar students (FY10 FCAT) (FY11 FCAT) …100 200 300 400 500 600 700 800 900 1000…

16. DRE Student Learning Growth is the Amount Above or Below Predicted Score CURRENT TEST PRIOR TEST 900 800 700 600 500 400 300 PREDICTED SCORE 600 Above Predicted Score Below Predicted Score ACTUAL SCORE (FY10 FCAT) (FY11 FCAT) …100 200 300 400 500 600 700 800 900 1000…

17. CURRENT TEST PRIOR TEST CURRENT TEST PRIOR TEST 10 20 30 40 50 60 70 80 90 100… Based on Prior Achievement OnlyBased on Prior Achievement & ELL What is the Predicted Student Score? Based on Prior Achievement 400 50 600 800 70 90 Each prior year test score has a Predicted current year test score. (FY10 FCAT) (FY11 FCAT) …100 200 300 400 500 600 700 800 900 1000…

18. FLDOE Value-Added Model Variables determining student learning growth Two or more years of prior achievement scores Attendance Mobility (number of transitions) Difference from modal age in grade (indicator of retention) Class size The number of subject-relevant courses in which the student is enrolled Homogeneity of entering test scores in the class LY Gifted status Language impaired Hearing impaired Visually impaired Emotional/behavioral Specific learning disability Dual sensory impaired Autism spectrum disorder Traumatic brain injury Other health impaired Intellectual disability

19. Reading Grade 7 Indicators Ranked By Statistical Significance IndicatorT-Statistic Achievement: Prior Year75.667 Achievement: Two Prior Years38.167 Enrolled in 2 or more courses18.354 Difference from Modal Age15.977 Attendance13.571 Number of students in Class 18.439 Homogeneity of Class 2 Prior Year Test Scores7.182 Language Impaired5.362 Specific Learning Disability5.013 Homogeneity of Class 1 Prior Year Test Scores4.750 ELL Indicator4.673 Homogeneity of Class 3 Prior Year Test Scores3.188 Number of students in Class 52.385 Enrolled in 2 or more class periods2.289 Mobility: Number of School Transfers2.271 Intellectual Disability2.141

20. Illustrative Growth Points All other variables being equal VariablesGrowth Points* Typical100 ELL100 more Low Attendance100 points less *adapted from AIR materials presented at August 2, 2011,meeting These are illustrative values, not actual values

21. CURRENT TEST PRIOR TEST CURRENT TEST PRIOR TEST 10 20 30 40 50 60 70 80 90 100… Based on Prior Achievement OnlyBased on Prior Achievement & ELL What is the Predicted Student Score? 400 50 600 800 70 90 (FY10 FCAT) (FY11 FCAT) …100 200 300 400 500 600 700 800 900 1000…

22. CURRENT TEST PRIOR TEST CURRENT TEST PRIOR TEST Based on Prior Achievement OnlyBased on Prior Achievement & ELL What is the Predicted Student Score? 400 500 600 800 700 900 (FY10 FCAT) (FY11 FCAT) (FY10 FCAT) (FY11 FCAT) …100 200 300 400 500 600 700 800 900 1000…

23. CURRENT TEST PRIOR TEST CURRENT TEST PRIOR TEST Based on Prior Achievement OnlyBased on Prior Achievement & Low Attendance What is the Predicted Student Score? 400 300 600 800 500 700 (FY10 FCAT) (FY11 FCAT) (FY10 FCAT) (FY11 FCAT) …100 200 300 400 500 600 700 800 900 1000…

24. How to calculate the student learning growth score? CURRENT TEST PRIOR TEST DRE The diagonal line is the predicted growth observed among similar students. (FY10 FCAT) (FY11 FCAT) …100 200 300 400 500 600 700 800 900 1000…

25. How to calculate the student learning growth score? CURRENT TEST PRIOR TEST StudentPYCYPredGrowth A200300400-100 B305545 10 C40 50-10 D507060 10 DRE (FY10 FCAT) (FY11 FCAT) FY10 FY11 FY11 FY11 FCAT FCAT …100 200 300 400 500 600 700 800 900 1000…

26. How to calculate the student learning growth score? CURRENT TEST PRIOR TEST StudentPYCYPredGrowth A200300400-100 B C40 50-10 D507060 10 DRE (FY10 FCAT) (FY11 FCAT) FY10 FY11 FY11 FY11 FCAT FCAT 300 550450 100 W …100 200 300 400 500 600 700 800 900 1000…

27. How to calculate the student learning growth score? CURRENT TEST PRIOR TEST StudentPYCYPredGrowth A200300400-100 B300550450 100 C400 500-100 D507060 10 DRE (FY10 FCAT) (FY11 FCAT) FY10 FY11 FY11 FY11 FCAT FCAT ? ? ? ? Y …100 200 300 400 500 600 700 800 900 1000…

28. How to calculate the student learning growth score? CURRENT TEST PRIOR TEST StudentPYCYPredGrowth A203040-10 B305545 10 C400 500-100 D500700600 100 DRE (FY10 FCAT) (FY11 FCAT) FY10 FY11 FY11 FY11 FCAT FCAT …100 200 300 400 500 600 700 800 900 1000…

29. DRE How do you calculate the teacher growth score? 1.Identify the students assigned to the teacher 2.Determine the learning growth of each of the assigned students 3.Calculate the average learning growth of the students TERMS

30. CURRENT TEST PRIOR TEST DRE StudentPYCYPredGrowth A200300400-100 B300550450 100 C400 500-100 D500700600 100 Average0 Typical Teacher Growth Score (FY10 FCAT) (FY11 FCAT) FY10 FY11 FY11 FY11 FCAT FCAT …100 200 300 400 500 600 700 800 900 1000…

31. Teacher Growth Score CURRENT TEST PRIOR TEST Above DRE StudentPYCYPredGrowth E200300400- 100 F300550400 100 G400750500250 H500700600 100 Average80.75 (FY10 FCAT) (FY11 FCAT) FY10 FY11 FY11 FY11 FCAT FCAT …100 200 300 400 500 600 700 800 900 1000…

32. CURRENT TEST PRIOR TEST DRE StudentPYCYPredGrowth I200300400-100 J300250450- 200 K400 500-100 L500700600 100 Average- 70.5 Below Teacher Growth Score (FY10 FCAT) (FY11 FCAT) FY10 FY11 FY11 FY11 FCAT FCAT …100 200 300 400 500 600 700 800 900 1000…

33. DRE Teacher Growth Teacher Growth School Growth Avg Growth = (A+B+C+D)/4

34. DRE Teacher Effect Teacher Effect School Growth

35. Calculating Teacher Value-Added Score Part 1 = DRE School Growth Score Teacher Growth Score - () Teacher Effect

36. DRE How do you calculate the school learning growth score? 1.Identify the students assigned to the school 2.Determine the learning growth of each of the assigned students 3.Calculate the average learning growth of the students

37. School Growth Score CURRENT TEST PRIOR TEST DRE StdtPYCYPredGrowth A200300400-100 B300550450 100 C400 500-100 D500700600 100 E200300400- 100 F300550450 100 G400750500250 H500700600 100 I200300400-100 J300250450- 200 K400 500-100 L500700600 100 Avg 4.1 (FY10 FCAT) (FY11 FCAT) FY10 FY11 FY11 FY11 FCAT FCAT …100 200 300 400 500 600 700 800 900 1000…

38. Calculating Teacher Value-Added Score Part 1: AKA Teacher Effect TypicalAboveBelow Teacher Growth Score 080.75-70.5 School Growth Score 4.1 Teacher Effect (Tch growth score – Sch growth score) -4.176.65-74.6 DRE Tch effect = (Tch growth score – Sch growth score )

39. What is a Teacher Value-Added Score? and How is it calculated? DRE

40. FLDOE Teacher Value-Added Score Conceptual.5 + ( X School Growth Score ) = DRE School Growth Score Teacher Growth Score - () Teacher Effect Tch vas = (Tch growth score – Sch growth score ) + (.5 x Sch growth score )

41. Teacher Value-Added Score TypicalAboveBelow Teacher Growth Score 080.75-70.5 School Growth Score 4.1 Teacher Effect Tch growth score – Sch growth score -4.176.65-74.6.5 x Sch growth score 2.05 Tch vas -2.0578.7-72.55 DRE Tch vas = (Tch growth score – Sch growth score ) + (.5 x Sch growth score )

42. State Average Student Learning Growth -20 -15 -10 -5 0 5 10 15 20 VAS = 15 VAS = 20 VAS = 25 Teacher Growth = 20Teacher Growth = 20 The greater a teacher’s growth score is from the state’s average and the school’s growth score, the higher a teacher’s VAS Tch vas = (Tch growth - Sch growth ) + (.5 x Sch growth )

43. DRE State Average Student Learning Growth -20 -15 -10 -5 0 5 10 15 20 VAS = 0 VAS = -20 VAS = 20 School Growth = 0School Growth = 0 Tch vas = (Tch growth - Sch growth ) + (.5 x Sch growth ) At a school with a growth score of 0, the teacher’s VAS equals the teacher’s growth

44. DRE State Average Student Learning Growth -20 -15 -10 -5 0 5 10 15 20 VAS = 15 School Growth = 10School Growth = 10 Tch vas = (Tch growth - Sch growth ) + (.5 x Sch growth ) VAS = 5 VAS = 0 VAS = -10 VAS = -5 At a high VAS school, a teacher’s growth must exceed half the school’s growth to have a positive VAS What is the significance of the green line?

45. DRE State Average Student Learning Growth -20 -15 -10 -5 0 5 10 15 20 VAS = -15 School Growth = -10School Growth = -10 Tch vas = (Tch growth - Sch growth ) + (.5 x Sch growth ) VAS = -5 VAS =0 VAS = 10 VAS = 5 At a low VAS school, a teacher’s growth must exceed half the school’s growth to have a positive VAS

46. DRE A Real Picture is Worth a Thousand Words

47. DRE Teacher VAM

48. DRE SDPBC Teacher VAM (Grade 5 Subject: Reading) Mean = 3.3 N = 561 Class >= 10

49. DRE State Teacher VAM (Grade 5 Subject: Reading) Mean = 0.1 N = 12182

50. Teacher VAM Distribution for Reading

51. DRE School Growth

52. DRE SDPBC School Growth Grade 5 Subject: Reading Mean = 4.0 N = 120

53. DRE State School Growth Grade 5 Subject: Reading Mean = 0.02 N = 2082

54. School Component Distribution for Reading

55. DRE How precise are VAM measures?

56. DRE Standard Error of Teacher VAMs The standard error gives the uncertainty (error band) surrounding a teacher’s VAM score It can be used to prevent categorizing teachers when that categorization would be uncertain

57. DRE Standard Error of Teacher VAMs Teacher Score Plus one SE Minus one SE

58. DRE Using SE to Categorize Teachers Teacher 2 Teacher 1 Teacher 3 Teacher 4 State Average Score

59. DRE Teacher VAMs Similar to State VAM # of Stdts Sig from State State Mean No sig higher & No sig diff No sig higher & No sig diff No sig higher & No sig diff

60. DRE Some teacher VAMs Higher than State VAM # of Stdts Sig from State State Mean 1 sig higher & No sig diff 2 sig higher & No sig diff 4 sig higher & sig differences

61. DRE Some teacher VAMs Lower than State VAM # of Stdts Sig from State State Mean 1 sig lower No sig diff 2 sig lower & sig differences 3 sig lower & No sig diff

62. DRE Some Teacher VAMs Lower/Higher than State VAM # of Stdts Sig from State State Mean 1 sig higher 2 sig lower & sig differences 1 sig higher 1 sig lower & sig differences

63. DRE Summary: VAM and School Learning Growth State Mandated Student Learning Growth – Difference between current score and predicted score – Based on prior achievement and selected student characteristics Teacher VAM includes both teacher effect and school learning growth School component is school learning growth Consider standard error when using Teacher VAM and school learning growth

64. SDPBC Must Classify Instructional Personnel/School Administrators into these Four Categories – Highly Effective – Effective – Needs Improvement – Unsatisfactory

65. DRE Proportion of a Year’s Growth & School Growth Score

66. CURRENT TEST PRIOR TEST DRE (FY10 FCAT) (FY11 FCAT) …100 200 300 400 500 600 700 800 900 1000… FY10 FY11 FY11 FCAT FCAT -FY10 LESSLESS MOREMORE One Year’s Worth of Growth StdPYCYCY- PY Year's Worth of Growth PYG A200300100 150 0.67 E200300100 150 0.67 I200300100 150 0.67 B300550250 125 2.00 F300550250 125 2.00 J300250-50 125 -0.40 C400 0 100 0.00 G400750350 100 3.50 K400 0 100 0.00 D500700200 80 2.50 H500700200 80 2.50 L500700200802.50 Avg 1.38 PYG 150 125 100 80

67. DRE A Real Picture is Worth a Thousand Words

68. SDPBC Mean

70. 11 10 09 08 07

72. DRE Area Scorecard and PYG To review area and school scorecards, go to the EDW.

73. DRE Summary: PYG and Scorecard PYG – Difference between current score and predicted score – Highly correlated with school learning growth – Use PYG reports and scorecard to see how you would have done?