1. 2007 Accountability Annual Meeting Evaluation and Reporting Office Florida Department of Education September 6, 2007

2. 1 Agenda Welcome and Introductions Responsibilities for Evaluation and Reporting 2006-07 School Grade Changes, Results, and Outlook 2006-07 AYP Results, Benchmarks, and Safe Harbor Impact of Growth Model AYP in 2007 and Beyond SV23 Matching/Updating Process New Website Existing Website Improvements School Grade Issues for 2007, Going Forward School Improvement Ratings for Alternative Schools Rule Development Workshop

3. 2 Evaluation and Reporting Staff Juan C. Copa, Bureau Chief Accountability Programs – Ed Croft, Director State Accountability Programs – Tracy Selman, Program Director Analysts – Sue Klos; Gambhir Shrestha, Ph. D. Federal Accountability Programs – Reneé Bruno, Program Director Analyst – Kiersten Farmer Support Staff – Toye Coxe, Staff Assistant Phone – 850-245-0411 Email – evalnrpt@fldoe.org

4. 3 Evaluation and Reporting Staff (cont.) Research and Evaluation – Marcus Mauldin, Ph. D, Director Analysts – Martha Miller, Ph. D; Jennifer Blalock Support Staff – Tria Parsons, Administrative Secretary Phone – 850-245-0429 Email – evalnrpt@fldoe.org

5. 4 School Grades and AYP School Grades Under A+ Plan School grades were first issued in 1999. Since 1999 there have been many changes to the calculation of school grades. We continue to look for ways to improve the calculation. Adequate Yearly Progress (AYP) The federal accountability indicator in the No Child Left Behind Act of 2001. AYP has been calculated since 2003. The growth model was approved in 2007.

6. 5 Schools in Need of Improvement Required by No Child Left Behind (NCLB). Evaluation and Reporting works with the Division of K-12 Public Schools to determine Schools in Need of Improvement. Only Title I schools subject to sanctions.

7. 6 Alternative Schools Accountability Part of the A++ Legislation passed during the 2006 Legislative Session (s. 1008.341, F.S.). Provides Alternative Schools the option of receiving a traditional school grade or a school improvement rating. If an alternative school chooses to receive a school improvement rating, the performance of the alternative school’s students will be included in the calculation of the rating and the school grade of the students’ home school.

8. 7 School Improvement Ratings for Alternative Schools – 2006 and 2007 Alternative Schools were identified based on their Primary Service Type as reported on the Master School Identification (MSID) file and district input and adjustment. Those schools were given the option of receiving a “Points Only” designation or a school grade. “Points Only” refers to the calculation of a school grade without the assignment of a letter grade.

9. 8 School Improvement Ratings for Alternative Schools – Activities (2006-07) Collected information to identify students who are statutorily to be excluded from the calculation Collected information on a student’s zoned school and district (element description adjusted for 2007-2008 data collection) Held rule development workshops on March 20 and August 30 in Tallahassee Workshop Scheduled for this afternoon (3:30- 5:30pm)

10. 9 School Improvement Ratings for Alternative Schools – Timeline April 2007 – Provided districts the opportunity to verify the list of alternative schools in their district (as identified in 2005-06) May 2007 – Provided alternative schools the option of a “Points Only” designation or a school grade August - September 2007 – Conduct rule development workshops on the calculation of school improvement ratings October/November 2007 – Final passage of rule before State Board of Education June 2008 – Release school grades, school improvement ratings for alternative schools, and AYP

11. 10 Summary of Issues Raised at Workshops Major concerns raised Potential exclusion of choice option schools from the definition of “alternative school” Some charters serve at-risk populations and have students referred to the school Crediting of students back to the “home school” What limitations? Only referred students? Can middle school student performance be credited back to an elementary school? Lack of a full-year enrollment requirement for the school improvement rating Statute specifies Survey 2 and/or Survey 3

12. 11 Calculation Issues to be Determined Crediting of students back to the “home school” Limit to only those students referred? Limit to only those students whose “home schools” reflect the same grade configuration as the alternative school? Magnitude of Learning Gains needed to define the categories of improvement – Improving, Maintaining, Declining Improvement of at least 1%, 5%, 10% over the previous performance? Examine other performance measures (e.g., QA for DJJ schools) and determine their compliance with the statutory requirements. Cell Size Unlike school grades, statute outlines that students enrolled in either membership count (fall and/or spring) are to be included. At least 30 students in either count? More than 10 students?

13. 12 Voluntary Pre-Kindergarten (VPK) Accountability Evaluation and Reporting worked with the Department’s Office of Early Learning to calculate Readiness Rates for VPK Providers Readiness Rates can be found at https://vpk.fldoe.org/ https://vpk.fldoe.org/

14. 13 Voluntary Pre-Kindergarten (VPK) Accountability Sample School Information

15. 14 Evaluation and Reporting developed Value Tables that districts may use to measure improved student achievement for teachers in performance pay calculations. Value Tables were provided to measure performance in Reading and Math in Grades 4-10. FCAT NRT (Grades 4 and 5) A total of 19 Value Tables were made available to districts (grade-level and school type-level). Value Table Points were also attached to student records and made available through the INDV file. Merit Award Program (MAP)

16. 15 Evaluation and Reporting is working with the Division of K-12 Public Schools to develop an accountability calculation for effectiveness of SES providers. Supplemental Educational Services Accountability

17. 16 Research, Analysis, and Evaluation. Evaluation and Reporting is tasked with a number of analyses, focusing mainly on examining the impact of programs and policies on student achievement. Research, Analysis and Evaluation

18. 17 SAT/ACT/PSAT/Plan, Teacher Projections and Projected High School Graduates SAT/ ACT/ PSAT/ PLAN and Teacher Projections Multiple results and trends are published on our website. Teacher Projections Multiple trends and statistical reports pertaining to teacher data are on our website. New hires Critical teacher shortage areas Florida teacher retention Supply of New Teachers Projected High School Graduates

19. 18 2006-07 School Grade Changes (Recap) Writing Remained at 3.5, no increase to 4.0. Essay only. Science Added as a seventh component. Math, Lowest 25 Percent Added as an eighth component. Revised School Grading Scale 800 point scale changed from current 600 point scale. Retakes of Grade 11 and 12 High Schools earned ten bonus points when half of all 11 th and 12 th graders retaking the FCAT met the graduation requirements in reading and math.

20. 19 History of School Grades

21. 20 Raising Standards and Higher Accountability Lead to Increased Student Performance 78 40 64 35 49 78 21 83 0 20 40 60 80 100 120 140 199920002001200220032004200520062007 Lowest Performing Schools Red bars indicate years when accountability requirements were increased. The count of "F" schools drops the year after each major increase in requirements.

22. 21 School Grades 2007: What is Measured Additional requirements applied AFTER a school’s points are calculated: Adequate Progress for Lowest 25%—required to earn grade based on calculated points. If a school does not meet this requirement, the school’s grade is lowered one letter grade. Participation Requirement (Percent Tested)—required to earn grade based on calculated points. Schools must test at least 95% of their students to earn an “A”, at least 90% to earn any other grade.

23. 22 Performance Components 2007

24. 23 Students Included in Performance Components Students who are enrolled all year (i.e., enrolled in the fall and spring terms through the testing period) and who are standard curriculum students* * Including speech-impaired, gifted, hospital homebound, and English language learners in an ESOL program two or more years.

25. 24 Performance Component Criterion: Are students achieving at or above grade level?

26. 25 Meeting Performance Criteria: Grade level performance for math, reading, and science = FCAT achievement level 3; Grade level performance for writing (essay) = FCAT score of 3.5; One point is awarded for each percent of students scoring at or above grade level.

27. 26 Performance Components Points 2007 READINGMATHWRITINGSCIENCE 100 possible pts. Schools receive one point for each percent of students meeting the performance criteria.

28. 27 Overall Learning Gains Components READINGMATHWRITINGSCIENCE Performance (new in 2007) Learning Gains Learning Gains of Lowest 25% (new in 2007)

29. 28 Students Included in Learning Gains Components All students (including students with disabilities and ELL students) who are enrolled all year (i.e., enrolled in the fall and spring terms through the testing period) and who have both current and prior-year FCAT scores.

30. 29 Learning Gains Criterion: Are students making at least one year’s worth of progress in a year’s time? Schools receive one point for each percent of students making learning gains.

31. 30 Meeting the Learning Gains Criterion: a.Improving by one or more FCAT achievement levels – e.g., from 1-2, 2-3, 3-4, 4-5; b. Maintaining FCAT achievement levels 3, 4, or 5; c.For non-retained students at achievement levels 1 or 2: showing more than one year’s growth on FCAT developmental scale scores (DSS).

32. 31 Students Included in Learning Gains for the Lowest Performing 25% Of students included in the learning gains calculation, the count of the lowest 25% is based on prior-year FCAT developmental scale scores.

33. 32 Learning Gains for the Lowest 25%: Who Is Included The count of the lowest performing 25% is restricted to students at FCAT achievement levels 1, 2, and 3.

34. 33 2007 School Grades Points Components, Total READINGMATHWRITINGSCIENCE Performance 100 possible pts. Performance 100 possible pts. Performance 100 possible pts. Performance 100 possible pts. Learning Gains 100 possible pts. Learning Gains 100 possible pts. Learning Gains of Lowest 25% 100 possible pts. Learning Gains of Lowest 25% 100 possible pts. PLUS 11 th and 12 th grade retakes for possible bonus points (10)

35. 34 Percent Tested (Participation Requirement) Adequate Progress of Lowest Performing Students Additional Criteria

36. 35 What percentage of eligible students who should have been tested on the FCAT were actually tested? Percent Tested Grade A schools - 95% or more tested All other grades - at least 90% tested Grade I schools - fewer than 90% tested

37. 36 Adequate Progress for the Lowest 25 % of Students Criterion: At least half (50% or more) of the lowest performing students must show learning gains in reading and math. Penalty for missing adequate progress = drop one letter grade.

38. 37 Impact of Adequate Progress Requirement In 2007, a total of 136 Schools dropped a letter grade due to the Adequate Progress requirement: 66 schools dropped from an “A” to a “B” 22 schools dropped from a “B” to a “C” 48 schools dropped from a “C” to a “D” Impact by School Type 28 elementary schools dropped a letter grade (2% of all elementary schools) 2 middle schools dropped a letter grade (less than 1% of all middle schools) 101 High schools dropped a letter grade (26% of all high schools 5 combination schools dropped a letter grade

39. 38 School Grades - 1999-2007

40. 39 A/B Schools Compared to D/F Schools - 1999-2007

41. 40 School Grades by Type 2007

42. 41 School Grades: 2007 Compared to 2006

43. 42 School Recognition: Schools Maintaining an “A” or Improving Grades

44. 43 2007 Grades for 2006 “F” Schools

45. 44 22 High Schools That Earned the Bonus Points Improved a Letter Grade

46. 45 2007-08 Outlook Adequate Progress of Low 25% Writing Plus Science and Writing (Use of District Averages in the Calculations) School Grading Scale Rating System for Alternative Schools

47. 46 No Child Left Behind Adequate Yearly Progress 2007 vs. 2006

48. 47 2007 School Grades Compared to AYP

49. 48 Subgroups Classifications AYP calculations are based upon nine subgroups. 1. Total School 2. White (W) 3. Black (B) 4. Hispanic (H) 5. Asian (A) 6. Native American (I) 7. Economically Disadvantaged Students 8. English Language Learners (ELL) 9. Students With Disabilities (SWD)

50. 49 39 Components of AYP Participation Rate in Reading for the 9 subgroups Participation Rate in Math for the 9 subgroups Reading Proficiency of 9 subgroups Math Proficiency of 9 subgroups Change in School Writing Proficiency Change in School Graduation Rate School Grade

51. 50 2006-07 State Targets for AYP  95% Participation Rate in Reading  95% Participation Rate in Math  51% of Students Proficient in Reading  56% of Students Proficient in Math  1% Improvement in Writing Proficiency or 90% at 3 and above  1% Improvement in Graduation Rate or graduation rate of 85% or higher  School Grade not a D or F

52. 51 AYP Benchmarks ReadingMathematics 2001-023138 2002-033138 2003-043138 2004-053744 2005-064450 2006-075156 2007-085862 2008-096568 2009-107274 2010-117980 2011-1286 2012-1393 2013-14100

53. 52 Determine if Reading/Math Proficiency Targets were Met Three Ways to Compute: First Method: Did the school and ALL subgroups meet the reading/math proficiency target? If Yes: Proficiency Targets are met If Not: Check Second Method Second Method: Did the school and/or ALL subgroups that did not meet reading/math proficiency targets meet Safe Harbor requirements? If Yes: Reading/Math proficiency targets are Met If Not: Check Third Method Third Method: Did the school and/or ALL subgroups that did not meet Safe Harbor requirements meet Growth Model “on track to be proficient” requirements? If Yes: Reading/Math proficiency targets are Met If Not: Reading/Math proficiency targets are Not Met

54. 53 When can a School Use Safe Harbor? The Safe Harbor provision can be applied only if a school has met all four of the requirements below: At least 95% of students in total and in each subgroup have participated in testing. The school has met writing criteria. The school has met graduation rate criteria. The school grade is not D or F.

55. 54 What are Requirements for Safe Harbor? Safe Harbor requires that for each subgroup being evaluated: The percent of non-proficient students decrease by at least 10% from the preceding year Graduation Rate Criteria is met Writing Criteria is met

56. 55 Safe Harbor: 10% Reduction in Non-Proficient Students

57. 56 When can a School Use the Growth Model provision? The Growth Model provision can be applied only if a school has met all four of the requirements below: At least 95% of students in total and in each subgroup have participated in testing. The school has met the writing criteria. The school has met the graduation rate criteria. The school grade is not D or F.

58. 57 What are Requirements for the Growth Model? The Growth Model requires that for each subgroup being evaluated: The percent of students “on track to be proficient” in three years or less is at least 51% in reading and 56% in math and The group has met the writing criterion (the increase in writing proficiency is at least 1% or the school has a writing proficiency rate of 90% or better) and The group has met the graduation rate criterion (the increase in graduation rate is at least 1% or the school has a graduation rate of 85% or better).

59. 58 AYP Growth Model Grade3456 Student’s Actual Reading Developmental Score 1001132514501635 Required DSS Score for Proficiency1198145615101622 Cut score needed to be “on track to be proficient” NA33% of 621 66% of 621 100% of 621 Is student “on track to be proficient”NoYes Year In State Tested Grade Decrease From Baseline Assessment In Performance Discrepancy 133% of original gap 266% of original gap 3Student must be proficient

60. 59 Impact of the Growth Model 675 schools met all proficiency benchmarks. 114 schools met requirements using Safe Harbor. 185 schools used the Growth Model or a combination of the Safe Harbor and the Growth Model. 113 schools needed the additional help of the mathematical adjustment to meet AYP.

61. 60 SV2/3 Match Process Two Critical Processes Performed by local MIS in Conjunction with EIAS Matching Process Data Update Process EIAS matches records by district, school and SID/AID Must be done through NWRDC – not a web process

62. 61 Purpose of Matching For AYP and School Grading Process: Identify students meeting the requirement of being in school all year. In Survey 2 and Survey 3. Identify students needing updates to data critical to the school grades and AYP calculations. Race. Grade Level. SWD Code. ELL Code and Entry Date. FRL Code. Withdrawals.

63. 62 The Five Files on NWRDC F70550 Unmatched Problem Records Students whose Prior School Status (PSS) shows same school, same district. Students for whom the demographic record is missing a matching PSS record. Students whose PSS withdrawal code is other than W01, W02, or ZZZ. Students who did not match between Svy2 and Svy3. Students with duplicated alias numbers. F70551 Unmatched Expected Records Most likely these students are new to the state between surveys. Withdrawal code of ZZZ and Entry/re-entry date after Svy2.

64. 63 The Five Files on NWRDC (cont.) F70549 Deleted Survey 3 Records and F70586 Deleted Survey 2 Records McKay, Home Ed, & Private School students. Be sure each student is identified correctly. F70548 Matched Records

65. 64 2007-08 Timeline Survey 3, 2007-08. Survey Week: February 4-8, 2008. State Processing: February 11 – March 7, 2008. Correct errors and submit corrections by 4:00 p.m. EST on Friday, March 7, 2008.

66. 65 Need for Improvement Missing and Incorrect Data Race, Gender, SWD, ELL, etc. not always updated. ELL dates often missing. Incorrect Grade Levels. Students enrolled in a school where they do not receive the majority of their instruction.

67. 66 To Remember Submit all four records, as appropriate: Student Demographic, Exceptional Student (SWD) English Language Learner (ELL) Student Course Schedule Students must have a course record to be included. Data should reflect Friday of FTE Week (February 8).

68. 67 Tentative Timeline for Corrections Processes School Types – January/February 2008 Prior Year Data – February  April 2008 Student Data Updates – March 13  April 2008 Retakes – March 13  April 2008 Assessment Corrections – June 2008 Appeals – Mid July to Mid August 2008

69. 68 Aids for Gathering Prior Year Data Student Locator Use CICS system at NWRDC Locate student IDs and prior FL public schools attended FASTER method of sending records electronically. Keep Reading and Math Litho Codes locally Send to new school when student moves Required data elements as of 1/1/07

70. 69 Prior Year Data Search Engine Allows a search of FCAT assessments. Search for Litho Codes. Search by last year’s District, School, Last Name, First Name, Grade level. An expanded Litho Code was necessary in 2007 to assure each record was unique. 8 digit Litho Code + 2 digit grade + 1 digit Retake indicator—R=Retake; 0=Not a Retake. Example: 12345678040 or 3456789810R.

71. 70 Overall Improvements The site will be made more secure. Pages will be made more aesthetically pleasing. Brief descriptions will be available (mouse over?) for clarification. Links will be larger, logically arranged, and consistent among pages and applications. Links for downloading views into Excel. Links will be added where necessary to enhance navigation.

72. 71 More Improvements Error messages will be more meaningful and consistent among applications. A “View All Errors” page will be added to appropriate applications. Page Titles and messages will be clear and appropriately placed. Highlighting will be made more visible. Instructions will be more user friendly. Views will contain records unique to that view where appropriate.

73. 72 Page Specific Improvements Login Page. DOE Contact information will be put on this page. Upload Page A link to download the formats in Excel. District and School Home Pages School page will have a note to call district if a reset is needed. Student Edit page The back button may be disabled. There will be a reset button that will reset all data entry cells to blank.

74. 73 Application Specific Improvements School Types Add a way of indicating new schools. Prior Year Data Corrections School dropdowns will be populated appropriately. Student Data Updates The “met graduation requirements” flag will be recoded. Withdrawal flag will be recoded. Assessment Data Corrections A “view all records” page will be added. Appeals Data will be entered via the web rather than uploaded.

75. 74 Issues for 2007, Going Forward Adequate Progress of Low 25% Writing Plus Science and Writing (Use of District Averages in the Calculations) School Grading Scale

76. 75 Contact Information Evaluation and Reporting Office Florida Department of Education 325 West Gaines Street, Room 445 Tallahassee, FL 32399 Website: http://www.fldoe.org/evaluation/ http://www.fldoe.org/evaluation/ Email: EVALNRPT@fldoe.org