1. + Utah Comprehensive Accountability System (UCAS) 1 Hal Sanderson, Ph.D. Research and Assessment August 21, 2013 2013

2. + UCAS performance data is embargoed. September 3 rd is the public release. 2

3. + Utah Comprehensive Accountability System (UCAS) Incorporates both student achievement and growth toward improvement in a composite score for each school (range 0 to 600). This is largely based on the Core CRTs. UCAS provides summary data for the entire school, as well as disaggregated results by ethnicity, and for economically disadvantaged, English language learners, and students with disabilities. UCAS achievement and growth results are for students who are in school for a full academic year (160 days in membership). 3

4. + Key Features of UCAS Each school can earn up to 300 points in Achievement and 300 points in Growth. Achievement (Percent of students proficient on CRTs in Math, Language Arts, Science and DWA). This is similar to previous accountability reports. Growth (Year-to-year scale score gain compared to like scoring “academic” peers across the state). This is a very different approach to calculating growth/progress. Subgroup(s). Identifies below proficient students as a single subgroup. Defined as all students who scored below proficiency (level 1 or 2) on the previous year’s CRT 4

5. + Key Features of UCAS continued Participation. A school must meet the 95% participation rate for the whole school and non- proficient subgroup of 40 students or more in each content area High School (Graduation Rate = College Readiness) AMO Information for schools is reported. 5

6. + ELEMENTARY/MIDDLE SCHOOL UCAS REPORT 6 Total Points School Info. Achievemen t Points Growth Points Report Info

7. + HIGH SCHOOL UCAS REPORT 7 Total Points School Info. Achievement Points Growth Points Report Info Grad Rate Points

8. + Comparison to AYP/U-PASS Achievement UCASAYPU-PASS Achievement For Students who are Full Academic Year (160 days) Percent of Students Proficient CRTs Included: Language Arts Yes MathematicsYes ScienceYesNoYes DWAYesNoYes Participation (n => 40) Yes 8 Achievement – is calculated by dividing the number of students scoring proficient or above (Levels 3 and 4) in each content area using the CRT/DWA/UAA tests by the number of students who were enrolled at the same school for the full academic year and took each test.

9. + Comparison to AYP/U-PASS Growth UCASAYPU-PASS Growth/Progres s For Students who are Full Academic Year (160 days) CRT scale score gain compared to like scoring peers Safe Harbor (gain in percent of student proficient) Progress Score (gain in U-PASS proficiency) The Growth or Progress Question? Student: How does my scale score gain compare to students who had my same prior year(s) score? What is my percentile rank? School: Is a greater percent of students proficient than the previous year? Student: Has my proficiency level increased from the prior year (e.g., Level 1b to Level 2b)? 9 UCAS Student Growth Percentile (SGP) – For students taking the CRTs, growth is determined by comparing the performance of a student with all other students in the state with the same past performance (1-3 years of CRT scores). Next, determine how performance in the current year compares with that of the student’s peer group to produce a growth percentile. Range from 0 to 100.

10. + 2013 UCAS Assessments 10 CRT LANGUAGE ARTS – Elementary Grades 3 – 6; Secondary Grades 7 - 11 CRT MATH - Elementary Grades 3 – 6; Math 7 CRT, Math 8 CRT, Algebra 1, Geometry, Algebra 2 CRT SCIENCE – Elementary Grades 4 – 6; Secondary 7 th Integrated Science, 8 th Integrated Science, Earth Systems Science, Biology, Chemistry, Physics Direct Writing Assessment – Grades 5 and 8

11. + Point Structure for Elementary and Middle Schools Overall School 600 Total Points Growth 300 total points All Students 200 total points Below Proficient Students 100 total points Achievement 300 total points Percent at or above proficient 300 points 11 Schools without a 12 th grade includes students in grades 3-8

12. Overall School 600 total points Growth 300 total points All students 200 total points Below Proficient Students 100 total points Achievement 300 total points Percent at or above proficient 150 total points Readiness Graduation rate 150 Total points Point Structure for High Schools Schools with a 12 th grade includes students in grades 9-12 12

13. + Subgroups 13 Identifies below proficient students as a single subgroup Below Proficient Subgroup = All students who scored below proficiency (level 1 or 2) on the previous year’s CRT Below Proficient Subgroup is determined independently for each content area (ELA, Math, Science) Ensures all students who are below standard, regardless of group, are the focus for improvement Below proficient subgroup is double weighted in the growth calculation to increase focus on those most at risk Complete disaggregated data for all 10 subgroups will be included in UCAS report including gap analysis

14. + Participation Requirement 14 A school must meet the 95% participation rate for the whole school and non-proficient subgroups of 40 students or more in each content area Participation is calculated for the whole school and the non-proficient subgroup Schools not meeting the participation requirement will receive a UCAS total score of 0

15. + Sample Elementary Achievement Calculation 15 Note: Schools without DWA, content areas are weighted equally (1/3 each) SubjectPercent Proficient Points Possible (Weighted) Achievement Points Earned ELA80%x86 28.57% of 300 =69 Math70%x86 28.57% of 300 =60 Science60%x86 28.57% of 300 =51 DWA80%x43 14.29% of 300 =34 Total300=214

16. + High Schools College & Career Readiness College and Career Readiness accounts for 150 of the 300 points for high schools in the achievement component. The readiness component is the federal graduation rate calculation as approved by USED. All graduation reporting includes this rate. For purposes of calculating UCAS, the graduation rate is calculated by multiplying the graduation rate by 150 (e.g..70 x 150 = 105). 16

17. + Sample High School Achievement Calculation 17 Subject Percent Proficient Points Possible Achievemen t Points Earned ELA80%x50 33% of 150 =40 Math70%x50 33% of 150 =35 Science60%x50 33% of 150 =30 Readiness/ Grad Rate (4 year federal) 80%x 150 100% of 150 =120 Total300=225

18. + Additional Details Student Growth Percentile 18 SGP

19. Student Growth Percentile Student growth is determined by comparing the performance of a student with all other students in the state with the same past scale score performance (1-3 years of CRTs). How does your gain compare to that of your like scoring peers? What is my percentile rank based on gain? The Student Growth Percentile is also known as the “Colorado Growth Model”. It is used for accountability in the following states: Colorado, Nevada, Indiana, Massachusetts, New Hampshire, Maine, Arkansas, Rhode Island, Virginia, West Virginia, New York, Hawaii, Idaho, Georgia, Wyoming and Utah. 19

20. + Normative 20 How does it work? Think of a group of students, where each student has two test scores – one for 2009 and one for 2010. We could show the distribution of these scores at the same time as pictured.

21. + Normative 21 We could ‘slice’ through the picture to show the 2010 distribution for just one 2009 score. This is called a conditional distribution. The red shaded curve shows the conditional distribution in 2010 for all students who scored 166 in 2009.

22. + Normative 22 Assume we are interested in just one score, 170, in 2010. We could ask, what percentage of students who scored 166 in 2009 scored at or below a 170 in 2010? In this case, that turns out to be 75%. In other words, a score of 170 is at the 75 th percentile. SGP = 75

23. Why use SGP? Determines growth based on multiple years of data for each student Honors variable amounts of growth (including small changes) Does not replicate proficiency Recognizes growth for students who are achieving at low and high rates Growth percentiles are calculated for every student, but can be aggregated to the classroom, subgroup, school, district, and state 23

24. + Additional Details Calculating Growth 24

25. New UCAS Growth Calculation More finite point calculation vs. old table (2012) below. 25 ALL STUDENT Growth TABLE Median Growth Score Points Median Growth Score Points 70 or higher20050125 6919649121 6819348118 6718947114 6618546110 6518145106 6417844103 631744399 621704295 611664191 601634088 591593984 581553880 571513776 561483673 551443569 541403465 531363361 521333258 511293154 30 or Lower50 Median SGP Achieved All Students (Maximum 200 points) 0-3450 35-49100 50-59150 60 and above200 OLD Growth TABLE

26. New UCAS Below Proficient Student Growth Calculation More finite point calculation vs. old table (2012) below. 26 BELOW PROFICIENT STUDENT Growth TABLE Median SGP Achieved Below Proficient Students (Maximum 100 points) 0-34 25 35-49 50 50-59 75 60 and above 100 OLD Growth TABLE Median Growth Score Points Median Growth Score Points 70 or higher1005063 69984961 68964859 67944757 66934655 65914553 64894451 63874349 62854248 61834146 60814044 59793942 58783840 57763738 567436 55723534 54703433 53683331 52663229 51643127 30 or Lower25

27. MGPPoints ELAAll Students Group56150 Below Proficient Group3550 MathAll Students Group45100 Below Proficient Group5575 ScienceAll Students Group50150 Below Proficient Group4050 Group ELA Points Math Points Science Points Point Total (mean) All Students150100150133 Below Proficient Students50755058 Growth Calculation Example calculation Total Growth Points School Total Growth Points 133 + 58 = 191 ( 27 )

28. + NEW Calculations Annual Measureable Objectives 28

29. Annual Measureable Objectives Federal Requirement to establish and report AMOs Utah’s Minimum Compliance Plan AMOs are not used in any UCAS calculation AMO trajectory will reduce in half the percent of non- proficient over six years AMOs will be established separately for each subgroup at each school UCAS reporting will list the AMO and performance of each school subgroup AMO reporting page will be a drill down page in the UCAS report AMOs will be used in identifying and exiting Focus schools 29

30. + Establishing AMOs AMOs will be based on the percent of students achieving proficiency on the state’s Criterion- Referenced Tests (CRTs) separately in English language arts and mathematics. ELA: CRT results in grades 3-8 and 10 are used to determine the percent of students proficient Mathematics: results are based on CRTs in grades 3-6 and in the course appropriate CRT thereafter which includes math 7, algebra, or geometry for grades 7 and 8. High schools will be determined by calculating the percent of 10 th grade students who scored proficient on the Algebra I CRT in 10 th grade year or a prior year Results from the Utah Alternative Assessment (UAA) are included for students with significant cognitive disabilities approved to participate in this assessment 30

31. + AMO Sample Calculation for a School Subgroup with ELA Proficient = 82% 100% – 82% = 18% ½ of 18 is 9 9 / 6 years = 1.5 per year Year one 82.0 + 1.5 = 83.5 Year two 83.5 + 1.5 = 85.0 Year three 85.0 + 1.5 = 86.5 Year four 86.5 + 1.5 = 88.0 Year five 88.0 + 1.5 = 89.5 Year six 89.5 + 1.5 = 91.0 (half way to 100 percent) 31 AMO Sample Calculation

32. + Sample AMO Trajectories for a School AllAsianAfrican Amer. Amer. Indian HispanicPacific Islander ED LEP SWD 2011818264606373705154 2012838467636675735558 2013848570676978755962 2014868773707280786366 2015878876737582806769 2016899079777884837173 201791 82808287857677 Goal:91 82808287857677 32