1. 1 Reading your Accountability Reports U-PASS AYP AMAO Judy Park, Assessment & Accountability Director Russell Klein, Results Coordinator Utah State Office of Education August 2006

2. 2 Reading your Accountability Reports U-PASS, AMAO, AYP August 18 – Sevier D.O., 180 E 600 N, Richfield August 22 – Granite D.O., 2500 S. State St., Salt Lake August 23 – Weber D.O., 5320 S. Adams Ave. Pkwy, Ogden August 25 – Nebo Learning Center, 570 S. Main St., Springville

3. 3 U-PASS Accountability Plan Utah Performance Assessment System for Students U-PASS Accountability Plan

4. 4 U-PASS ●State Accountability ●Required by State Law ●School level report ●Based on multiple assessments ●Includes attendance & graduation rate ●2006 – schools without a 12 th grade only ●2007 – all schools

5. 5 U-PASS Accountability Plan ●Student Participation on Assessments ●Status Scores  Percentage of students proficient on the collection of assessments and other indicators ●Progress Scores  Longitudinal measure based on comparing the achievement levels of the same student from one year to the next

6. 6 Achieved State Level of Performance Total School &Subgroup Low progress Medium progress High progress Acceptable status Unacceptable status Needs Assistance

7. 7 U-PASS Accountability Plan ●Student Participation  95% ●Acceptable Status Score  80% ●Progress Score  Low 0 – 189  Medium 190 - 214  High 215 - 400

8. 8 Progress Value Table Year 1 Level Year 2 Level 1a1b2a2b34 1a0225350375 1b0125225350375 2a050150225325350 2b0075175275325 3000100200275 40000125225 196.0196.0

9. 9 Schools will be identified as: ●Achieved State Level of Performance  Participation is 95% and  Total School AND Subgroup Status is acceptable (80%) or progress is medium (190) or high ●Needs assistance  Participation is less than 95% or  Total School or Subgroup Status is not acceptable and progress is low

10. 10 Schools without a 12 th grade Status and Progress Score ●Language Arts (35%)  ELA CRT (35%) Or  ELA CRT (30%) & DWA 5% ●Math (35%)  Math CRT ●Science (20%)  Science CRT ●Attendance (10%)

11. 11 U-PASS Accountability Plan ●Aggregate Subgroup Accountability  Every student who belongs to a subgroup other than white  Individual student proficiencies are added together and divided by the number of students to determine the subgroup proficiency level ●Any individual subgroup not meeting proficiency will appear on the front page of the report

12. 12 U-PASS Accountability Plan ●Subgroup Status Accountability.  Allows greater accountability.  Students in a subgroup with less than 10 are now included in the aggregate calculation.  Students are more fairly represented.  Each student only counts once, regardless of the number of subgroups in which the student qualifies.  Increased reliability.  We are evaluating a larger group.  More students and more schools are represented.  Increased validity.  We do not overweight small populations.  This lifts the burden of one subgroup being the sole determinant of unacceptable status of the school.

13. 13 The 2006 U-PASS Report ●Web-based report format ●An increased amount of information available on each school ●Allows for “drill-down” information

14. 14

15. 15 1

16. 16 2

17. 17 3

18. 18 Must achieve both criteria Final Result

19. 19 Subgroups Needing Assistance

20. 20 Status “Drill Down”

21. 21 Click on Grade Name for Instant Graph

22. 22 Second Level “Drill Down” Click on Grade Score for Further Drill Down

23. 23 Third Level “Drill Down” Boxes expand on ‘mouseover’

24. 24 Progress “Drill Down” coming in October

25. 25 Additional Reports

26. 26 U-PASS ●Additional Information:  NRT  ACT/SAT  AP  Concurrent Enrollment  School Summary Information  Student Summary Information  Percent of student reading on grade level  (reported for grades 1-10)  Dropout Rate (disaggregated by ’08)  Disciplinary Action  Course Taking Patterns and Trends

27. 27

28. 28

29. 29

30. 30 AYP Reports

31. 31 Schools with a 12 th Grade ●Assessment Drill Downs ●Additional Information

32. 32 Questions?

33. 33 AYP School Reports For 2006

34. 34 AYP ●Federal Accountability ●Required by Federal NCLB Title I Law ●School, District & State level Reports ●Based on Language Arts CRT & Math CRT ●Includes attendance & graduation rate

35. 35 AYP 2006 Level 1 ●Three parts: 1.Participation in testing 2.Academic achievement 3.School attendance / graduation ●All-or-nothing concept

36. 36 1 2 3

37. 37 AYP 2006 Level 1 ●Did the school make AYP? ●Math vs. Language Arts? ●AMO (Annual Measurable Objectives) same as last year ●Consequences only for Title I schools

38. 38 1 2 33 4

39. 39

40. 40 AYP 2006 Level 1 ●WHICH STUDENTS ARE IN THE AYP SCHOOL REPORTS?

41. 41 Students in AYP & U-PASS School Reports ● Two large data collections  1) End-of-Year Clearinghouse  2) Core CRT Files ● These Two Files Need To Match To Be On AYP & U-PASS School Reports! ● 6 Fields for Matching: District#, School#, Last Name, First Name, Date of Birth, Gender ● As of Spring ’06 – SSID!

42. 42 Full Academic Year Used for Core CRT tests to be included - Full Academic year 2006- 1. Student who was in membership on Sept. 15th 2. Student membership >=160 3. Student membership = School Membership + Special Ed Membership (Self-Contained)+ YIC membership YIC membership

43. 43 Math Core CRTs Math CRTs Grade 3 – 8 Mathematics: 64% Proficient is the Status Goal (AMO) for 2006 Grades 10-12 Algebra & Geometry: Applied Math 1 & 2 – not included 47% Proficient is the Status Goal – ONLY students in Grades 10,11,12 who took the Core CRTs in either Algebra or Geometry. Schools with a 12 th grade class get this AMO/Goal.

44. 44 Language Arts Core CRTs Language Arts Grades 3 – 8 Language Arts: 71% Proficient. This is the Status Goal (AMO) for 2006. This is the Status Goal (AMO) for 2006. Grade 10 Language Arts: 70% Proficient. No other language arts Core CRTs are being No other language arts Core CRTs are being used for AYP for schools with a 12 th grade class! used for AYP for schools with a 12 th grade class!

45. 45 AYP 2006 Level 1 ●Questions?

46. 46 AYP 2006 Level 2 ●Status vs. Safe Harbor (Improvement) ●Safe Harbor  10% reduction in % of students not proficient

47. 47 AYP 2006 Safe Harbor (Improvement) ProficientNot Proficient Status

48. 48 AYP 2006 Safe Harbor (Improvement) Did not make AYPMade AYP ProficientNot Proficient Status

49. 49 AYP 2006 Safe Harbor (Improvement) Shows Improvement Does not show improvement Improvement

50. 50 AYP 2006 Safe Harbor (Improvement) Shows ImprovementMade AYP Does not show improvement Did not make AYP Improvement

51. 51 AYP 2006 Safe Harbor (Improvement) ProficientNot Proficient Status Does not show improvement Shows Improvement Good for Status Good for Improvement Improvement

52. 52 AYP 2006 Safe Harbor (Improvement) ProficientNot Proficient Status Does not show improvement Shows Improvement Improvement Made AYP

53. 53 Status

54. 54 Safe Harbor

55. 55 AYP 2006 Level 2 ●Safe Harbor  Additional Indicator is only used when Safe Harbor calculation is needed

56. 56 AYP 2005 Level 2 ●Confidence Interval on Status  99% ●Confidence Interval on Safe Harbor  75% ●Participation & Additional Indicator  N/A

57. 57 Confidence Interval ●Voting – exit polls

58. 58 Confidence Interval on Status

59. 59 Confidence Interval on Safe Harbor

60. 60 AYP 2006 Level 2 ●N size – 10 vs. 40 ●Small N = Automatic YES

61. 61 AYP 2006 Level 2 ●N=40 participation, attendance, and graduation ●N=10 academic achievement

62. 62

63. 63

64. 64

65. 65 AYP 2006 ●Graduation rate – 85.7%  One year delayed (have until Sep 1)  05 data from warehouse  04, 03 data from USOE statistical files ●Quality of data has improved ●40 ways to make (or not make) AYP

66. 66

67. 67 40 ways to make (or not make) AYP Times 10

68. 68 “Elementary”

69. 69 “Elementary”

70. 70 “District”

71. 71 Available on the secure FTP site

72. 72 AYP 2006 Level 2 ●Questions?

73. 73 AYP 2006 Level 3 ●Definition of “safe harbor” ●Definition of “participation” ●Definition “confidence intervals” ●Definition of “graduation rate” ●UAA

74. 74 AYP 2006 Level 3 ●Questions?

75. 75 AYP 2006 Did your school “make it”? ●Participation – DID YOU MEET IT? ●Participation – Is the N size less than 40? ●Did they have 95% participation? ●Did they average 95% participation over the last three (or two) years?

76. 76 AYP 2006 Did your school “make it”? ●Status – Is the subgroup size less than 10? ●Status – Is the % proficient enough? ●Status – If the % proficient is not enough – does the confidence interval change it?

77. 77 AYP 2006 Did your school “make it”? ●Safe Harbor – Is there a 10% reduction in students not proficient? ●Safe Harbor – Is the % reduction of students not proficient within the confidence interval?

78. 78 AYP 2006 Did your school “make it”? ●Attendance/Graduation Rate – Is the subgroup less than 40? ●Attendance/Graduation Rate – Did the school meet 93% attendance rate or 85.7 graduation rate ●Attendance/Graduation Rate – has there been an increase from the previous year?

79. 79 Level Three Clarifications

80. 80 AYP 2006 Level 3 ●Participation Category – Students enrolled on test date. ●Participation within tests – Students you’ve had for the full academic year. (subset of “category” set) ●Participation – grad/attend. Is ALL students…

81. 81 95% Participation For 2004 - 2006, Missing tests will be handled by the testing process. Same as 2003 with the exception that participation percentage can be averaged over the previous two (or three) years, if necessary. If this does not help, the current year participation percentage will be used

82. 82 LEP Subgroup Definition 2004 - 2006, LEP students will count in the LEP subgroup for two years past their exit of the ELL program. This is determined by the lep_exit_date in the student_lep table of the warehouse.

83. 83 Additional Indicator AYP Attendance 93% The goal for attendance will be set at 93%, consistent with the additional indicator goal for elementary and middle schools. Schools must meet this goal or improve from the previous year. Improvement is defined as any increase in this area. The attendance rate is the sum of days in attendance divided by the sum of days in membership for all students in the unit of accountability who were in membership for at least eleven (11) days and whose membership is equal to or greater than attendance (180 days). The attendance rate is the sum of days in attendance divided by the sum of days in membership for all students in the unit of accountability who were in membership for at least eleven (11) days and whose membership is equal to or greater than attendance (180 days). Membership includes self- contained SCRAM and YIC systems.

84. 84 Additional Indicator AYP Graduation Rate 85.7% Grad Rate is Yes, if Grad Rate is >= 85.7 The cohort graduation rates by school according to the ED approved algorithm in our AYP plan. Here is the algorithm: =IF(G2 0.857,"pass",IF(L2>W2,"pass","fail"))) In English: If the cohort [G] contains less then 40 students, there are too few students to make a reliable decision about the graduation rate for the unit of analysis (school or district). = NA If the cohort is equal to or greater than 40 students and the 2002 graduation rate [L] is greater than (the constant) 85.7%, the unit passes. = Yes If the 2002 graduation rate is less than 85.7% but greater than the 2001 graduation rate [W], the unit passes. = Yes In all other cases, the unit fails. = No

85. 85 UAA Testing Special Ed Alternate Assessment ●Grade level is derived based on birthdate. ●Core Test Id and grade level will be used to determine which Actual Test ID to assign a student to. LA and grade level will correspond to an Actual Language (Reading) class for that student. MA and grade level will correspond to a math course. ●UAA students whose birthdate is in 9-12th grades will be added to GE0812R Geometry. They are part of High School Math AYP! ●UAA in 7-8th grade will be added to 7th grade math.

86. 86 Out-of-Level Testing ●Not allowed  No for participation  No for academic achievement ●Math Levels:  6 th – Math 6; 7 th – Math 7; 8 th – Pre-Alg. Anything lower for the grades is Out-of-Level and considered Not Proficient.

87. 87 District AYP Reports ●Previous District AYP Report  Grades 3 – 8 only

88. 88 District AYP Reports ●2006 District AYP Report  Grades 3 – 8  Grades 10 – 12 Business Rule An LEA will receive a report based on both 3- 8th and 10-12th grade AMO's. An LEA must fail at both the 3-8 AND the 10-12 grade levels to fail AYP. (Either subject)

89. 89 District AYP Reports ●2006 District AYP Report  Grades 3 – 8  Grades 10 – 12 ●Program Improvement Business Rule An LEA is identified for improvement only when it misses AYP in the same subject and in all grade spans for two consecutive years, or the other academic indicator in all grade spans for two consecutive years. (It is possible for an LEA to not pass AYP and yet not be put into 'Improvement' because 'Improvement' considers 'subject' in addition to grade levels whereas 'not making AYP' does not.

90. 90 AMAO Reports

91. 91 AMAO ●Federal Accountability ●Required by NCLB Title III Law ●District & State Level Reports ●Based on  IPT progress  IPT proficiency  AYP for ELL

92. 92

93. 93

94. 94 ●Enrollment Information ●AMAO #1 – Annual increases in the number or percentage of children making progress in learning English ●AMAO #2 – Annual increases in the number or percentage of children attaining English proficiency ●AMAO #3 – AYP for ELL

95. 95 ELL Enrollment ●Grade Span ●Language Proficiency Category ●2005 ELL Enrollment ●2006 ELL Enrollment

96. 96 ELL Enrollment Grade Span The grade span used for AMAO targets is grades K-6 and 7-12.

97. 97 ELL Enrollment Language Proficiency Category ●A - Non-English proficient in speaking, reading and writing ●B - Limited English proficient (student is non in at least one of the language modalities of speaking, reading and writing or at least limited in two of the language modalities of speaking, reading and writing) ●C - Fluent in two of the language modalities and limited in one

98. 98 ELL Enrollment Language Proficiency Category ●D - Monitored student for minimum of two years for English proficiency (student is fluent in all language modalities). ●E - Exited, Former LEP student who is fully proficient in English speaking, reading and writing and has been exited from an alternative language services program and fully assimilated into the mainstream. ●O - Identified as LEP but opted out of services by parent.

99. 99 ELL Enrollment 2005 ELL Enrollment The number of ELL students enrolled in 2005 for the full academic year. For E students, only those students who have exited within the last 3 years are included.

100. 100 ELL Enrollment 2006 ELL Enrollment The number of ELL students enrolled in 2006 for the full academic year. For E students, only those students who have exited within the last 3 years are included.

101. 101 AMAO Requirement #1 ●Grade Span ●% of 2005 students making progress learning English in 2006 (AMAO 80%) ●District made the AMAO targets for making progress learning English

102. 102 AMAO Requirement #1 Grade Span The grade span used for AMAO targets is grades K-6 and 7-12.

103. 103 AMAO Requirement #1 % of 2005 Students making progress learning English in 2006 The percentage of ELL students who made progress in one or more domains (speaking, reading, writing). Progress is defined as moving from non speaking to limited or limited to fluent.

104. 104 AMAO Requirement #1 District made the AMAO targets for making progress learning English This column is Yes if the district made the AMAO target of 80% of students showing progress in one or more domains (speaking, reading, writing).

105. 105 AMAO Requirement #2 ●% of 2005 students Attaining English Proficiency in 2006  K-6, 16.2%  7-12, 21.6% ●District made the AMAO target for Attaining English Proficiency

106. 106 AMAO Requirement #2 % of 2005 students Attaining English Proficiency in 2006 (K-6, 16.2%) (7-12, 21.6%) Percentage of students enrolled for both the 2005 and 2006 full academic year, with a language proficiency of D or E in 2006.

107. 107 AMAO Requirement #2 District made the AMAO target for Attaining English Proficiency The column is Yes if the district made the AMAO target. For grade ranges K-6, 16.2% of the students are at the D or E level. For grade ranges 7-12, 21.6% of the students are at the D or E level.

108. 108 AMAO Requirement #3 ●Grade Span ●% Participation of ELL for AYP Language Arts ●District made AYP for the ELL subgroup in Language Arts ●% participation of ELL for AYP Math ●District made AYP for the ELL subgroup in Math

109. 109 AMAO Requirement #3 Grade Span The grade span used for AYP targets is grades 3-8 and grades 10-12. % Participation of ELL for AYP Language Arts The percentage of ELL students who participated in the language arts CRT.

110. 110 AMAO Requirement #3 District made AYP for the ELL subgroup in Language Arts The ELL subgroup attained 95% participation and for the 3-8 grade span attained 71% of the students proficient on the language arts CRT; and for the 10-12 grade span attained 70% of the students proficient on the language arts CRT.

111. 111 AMAO Requirement #3 % Participation of ELL for AYP Math The percentage of ELL students who participated in the Math CRT.

112. 112 AMAO Requirement #3 District Made AYP for the ELL subgroup in Math The ELL subgroup attained 95% participation and for the 3-8 grade span attained 64% of the students proficient on the Math CRT; and for the 10-12 grade span attained 47% of the students proficient on the Math CRT.

113. 113 District Made AMAOs ●The District is “Yes” if there is a “Yes” in every column. One “No” in any column will result in a “No” for District Made AMAOs.

114. 114 Appeals 1.An error in the aggregating and merging of data and/or a calculation error, or other errors that directly affects school or district status. 2.Extreme circumstances that affected data and/or test administration that had a significant impact on the school (e.g., flood, fire, natural disaster).

115. 115 Appeal Process ●September 11  Deadline for District Superintendent to submit a written appeal request for U- PASS, school AYP, District AYP, or District AMAO stating the reason for the appeal An E-mail response from USOE will confirm receipt of the appeal. Data and documentation that directly influences the appeal needs to be stored at the district.

116. 116 ●September 12-15  USOE reviews appeals and makes changes to the U-PASS, AYP & AMAO school or district designation. ●September 16  Districts release U-PASS, school AYP, district AYP & district AMAO to the public. ●September 17  U-PASS, AYP & AMAO reports posted on the web. Reports for granted appeals, changed from a No to a Yes (by appeal).