Assessment and Accountability Update Chun-Wu Li, Ph.D. Administrator II, Assessment and Accountability Assessment Accountability Continuous Improvement Unit Division of Educational Services December 8, 2017

Assessment Update

English Language Proficiency Assessments for California (ELPAC) (See Handout “The ELPAC and CSA-On the Road to Completion but in Different Gears”)

New Practice Test for ELPAC

New Resources http://www.cde.ca.gov/ta/tg/ep/documents/howtotestel.pdf http://www.cde.ca.gov/ta/tg/ca/documents/elpacassessmenrfs.pdf

New Resources (cont.)

New Resources (cont.) http://www.cde.ca.gov/ta/tg/ep/elpacparentguide.asp

California Assessment of Student Performance and Progress (CAASPP)

Accessibility and Accommodations Educational Testing Service (ETS) presented a Webcast on November 15, 2017 that includes information about the CAASPP accessibility and accommodations to prepare LEAs for effective implementation of the 2017–18 student accessibility resources. Topics include 2017–18 student accessibility highlights, updated Matrix One, 2017–18 CAASPP Individual Student Assessment Accessibility Profile (ISAAP) Tool, and the Unlisted Resources approval and implementation process. The archived webinar can be located at Accessibility and Accommodations.

Interim Assessment User Guide The CAASPP 2017–18 Interim Assessment User Guide is now available on the Interim Assessment Administration Resources page of the CAASPP Portal Web site. The guide provides instructions for LEA staff using the interim assessment online systems, including:  Interim Assessment Viewing System Interim Assessment Test Delivery System Interim Assessment Hand Scoring System Interim Assessment Reporting System

Interim Assessment Reporting System Video A demonstration video for the new Interim Assessment Reporting System is now available on the Interim Assessment Administration Resources page of the CAASPP Portal Web site. This 10-minute reporting system video provides a system demonstration of features including:   how to create and manage student groups, where to view aggregate student results, how to see detailed item information, and how to access student responses to test items—which are just a few of the capabilities the new system provides.

Digital Library demonstration Video Located on the CAASPP Portal Web site, there is the Smarter Balanced Digital Library demonstration video that is available for Digital Library users. This four-minute video guides users through the features of the system’s new search interface. It illustrates how users can optimize the functionality of their searches and ensure that they have access to the high-quality instructional and professional resources they need to strengthen their formative assessment practices.

California Alternate Assessments (CAAs) The 2017–18 One Percent Threshold Justification Form is now available for LEAs to complete that think they will exceed one percent. The form must be submitted by December 15, 2017. Now available on the CAASPP Reporting Web site are the results from 2016–17 One Percent Threshold Justification Form. This Excel file lists the LEAs that reported they may have exceeded the 1% threshold on alternate assessments for 2016−17.  On November 1, the CAA Test Examiner Tutorial video is now available. The tutorial is divided into sections: CAAs for ELA and mathematics and CAA for Science.

California Alternate Assessments Science The 2017–18 embedded performance tasks (PTs) for the year two pilot are now available on TOMS. Six PTs have been posted, three PTs for grade five, three PTs for grade eight, and three for high school.  Beginning December 2, 2017, LEAs will be able to assign the CAA for Science to all grade twelve students who are eligible for the CAA for Science. In addition, LEAs will be able to assign students who are in grade ten or eleven to take the CAA for Science.  Draft blueprint, performance level descriptors, and scoring structure CAA for Science will be presented to the State Board of Education (SBE) for approval in January 2018.

California Science Test (CAST) The draft CAST blueprint, draft achievement level descriptors, and score reporting structure were presented to the SBE for approval on November 8, 2017.  All grade twelve students will be assigned the CAST for spring, 2018. LEAs also have the option of assigning the CAST to grade ten or eleven students beginning on December 2, 2017.

Accountability Update (Additional Handout “California’s New Accountability System and School Dashboard”)

SBE Action on Academic Indicator Approved: New Change cut scores for English language arts/literacy (ELA) and mathematics New High and Medium Status cut scores for mathematics New color designation for five-by-five grid California Department of Education

Background on Academic Indicator Based on Smarter Balanced Summative Assessments results for ELA and mathematics Fall 2017 Dashboard uses data from spring 2017 administration for grades 3–8 Status and Change calculated using “Distance from Level 3” (DF3) California Department of Education

California Department of Education Distance from Level 3 Distance between: Student’s score on Smarter Balanced Summative Assessments, and Lowest possible scale score* for Standard Met Achievement Level (Level 3) *Scale scores vary by content area and grade level. SBE changes do not affect how DF3 is calculated. As a reminder, DF3 is the difference between a student’s scale score on the SBSA and the lowest possible score – for that grade level and content area) – for Level 3 (which is the Standard Met Achievement Level) California Department of Education

DF3 Example: Grade Five Mathematics Grade five student received a score of 2,505. Here’s an example. In grade 5 mathematics, the scale score for the Standard Met Achievement Level from 2,525 and 2,578. So 2,525 is the lowest possible score for Level 3. It represents the threshold for Level 3. Every student score will be compared to that fixed number. Therefore, a score of 2,505 is 23 points below the Standard 3 threshold. The DF3 is – 23. When calculating Status for the Academic Indicator, each student’s DF3 is calculated separately, and then all distances are combined to determine an average at the LEA, school, and student group levels. Student scored 2,505. This is 23 points below the lowest possible score for Level 3. (2,528 – 2,505 = 23 points) California Department of Education

Volatility in Scale Scores Performance standards based on two years of data (2015 and 2016) Distributions positively skewed: Over 80 percent of LEAs had positive change. 2017 Change distribution more symmetrical: 45 percent of LEAs had a positive change In the next several slides, we are going to look at some of the issues that arose with the original cut scores and color designations and how the SBE actions attempt to remedy these issues. When the standards were set for the Academic Indicator, we only had two years of Smarter Balanced assessment data available: for 2015 and 2016. As it turned out, the distributions were positively skewed, with over 80 percent of districts showing a positive Change. However, when we looked at the results for the 2017 assessments, we saw a more symmetrical distribution: Only 45% of districts saw a positive Change. California Department of Education

Produces Dramatic Downward Swings High performing schools must sustain continued improvement that may be untenable. Some 200 schools at “High” or “Very High” Status level dropped two performance levels in 2017 (from Blue to Yellow), even as most of their students continued to meet or exceed state standards. Example: School with a DF3 of +116.3 in 2016 and +99.3 in 2017 would move from a Blue to Yellow performance level. California Department of Education

Produces Dramatic Upward Swings Schools with Very Low Status that improve by just 15 points from prior year could jump two performance levels (i.e., Red to Yellow). Applying current cut scores and methodology to 2017 DF3 would result in 1,117 schools (15 percent) moving up or down at least two performance levels in ELA. California Department of Education

New Change Cut Scores for Both English language arts and Mathematics Increased Significantly By 15 points or more Increased by 3 to less than 15 points Maintained Declined by less than 3 points or Increased by less than 3 points Declined By 3 to 15 points Declined Significantly by more than 15 points California Department of Education

Rationale: Revisions to Change Cut Scores Normal distribution supports symmetrical cut scores 2016 cut scores (“Maintained” level): ˂ -1 to ˂ +7 (ELA) ˂ -1 to ˂ +5 (mathematics) Revised Change cut scores: ˂ -3 to ˂ +3 (both ELA and mathematics) Based on 2017 data, the distribution for Change is a normal bell curve, so cut scores should be more symmetrical to align with those established for the other indicators. Current cut scores for “Maintained” level range from less than -1 to less than +7 (for ELA) and less than -1 to less than +5 (for mathematics) The revised Change cut score for “Maintained” ranges from less than negative three (-3) to less than positive three (+3) California Department of Education

Revisions to Mathematics Cut Scores Status Cut Scores Very High 35 or higher High Zero to 34.9 points (revised) Medium -25 points below to less than Zero (revised) Low -25.1 points to -95 points Very Low -95 points or lower California Department of Education

Rationale: Revisions to Status Cut Scores for Mathematics Ensure that LEAs and schools cannot receive a “High” Status unless they have a positive DF3: Addresses issues raised by advocacy groups after release of Spring 2017 Dashboard Addresses issues raised by advocacy groups after release of Spring 2017 Dashboard California Department of Education

New ELA Five-by-Five Colored Table Level Change: Declined Significantly by more than 15 points Change: Declined By 3 to 15 points Change: Maintained Declined by less than 3 points or Increased by less than 3 points Change: Increased by 3 to less than 15 points Change: Increased Significantly By 15 points or more Status: Very High 45 points or higher Green* Green Blue Status: High 10 to 44.9 points Status: Medium -5 points to +9.9 points Yellow* Yellow Status: Low -5.1 to -70 points Orange* Orange Status: Very Low -70.1 points or lower Red * Change in color designation California Department of Education

New Mathematics Five-by-Five Colored Table Level Change: Declined Significantly by more than 15 points Change: Declined By 3 to 15 points Change: Maintained Declined by less than 3 points or Increased by less than 3 points Change: Increased by 3 to less than 15 points Change: Increased Significantly By 15 points or more Status: Very High 35 points or higher Green* Green Blue Status: High zero to 34.9 points Status: Medium -25 points to less than zero Yellow* Yellow Status: Low -25.1 to -95 points Orange* Orange Status: Very Low -95 points or lower Red * Change in color designation California Department of Education

Rationale: Revisions to Color Designation LEAs and schools that maintain their Status can only be assigned one of two colors: School with Very High Status can only move from Blue to Green if it declines. School will Very Low Status can only move from Red to Orange if it increases.   Revised color designation limits each row to two colors, ensuring that LEAs and schools that maintain their Status will always be assigned one of two colors, regardless of their Change. California Department of Education

Feedback on Academic Indicator What information should be included in a one- page flyer on the Academic Indicator: Should contain graphics Should be clear for general public California Department of Education

SBE Action on Chronic Absenteeism Indicator Include information in Fall 2017 Dashboard that redirects users to Chronic Absenteeism reports on DataQuest. Direct CDE staff to develop recommendation for: March 2018 SBE meeting: Proposed Status cut scores to be used to update Fall 2017 Dashboard September or November 2018 SBE meeting: Proposed Change cut scores California Department of Education

Feedback on College/Career Indicator (CCI) On Equity page of the Dashboard, CCI row reflects “N/A” Causing confusion and generating many questions. Is there more explanatory language that we can use? (Example: Status Only)  California Department of Education

This the final step in a series of actions by the SBE related the development of the new accountability system. The system is designed for continuous improvement in mind. In March we will begin the process again, reviewing indicators for 2017-18 and making recommendations for changes to the indicators. California Department of Education

New Communication Materials California Department of Education

Flyers on the Dashboard Sacramento County Office of Education is developing a series of one- and two-page flyers on the Dashboard: Five-by-Five Colored Grids Academic Indicator CCI Dashboard Alternative School Status (DASS) English Learner Subgroups Distribute the Five-by-Five Colored Grids and DASS fliers and ask for feedback on the DASS California Department of Education

Dashboard Release Schedule California Department of Education

Release Schedule for the Fall 2017 Dashboard Early November 2017 LEA private preview began Rolling weekly release of indicators Release of Academic Indicator: November 16 Week of November 27, 2017 Public launch of the Dashboard California Department of Education

Assessment and Accountability Related Items

2017-2018 RCAN Dates (Revised) 2017-2018 RCAN Dates (Regular Meeting: 9:00 a.m.-12:00 p.m.; Mini Workshop: 12:30 p.m.-2:30 p.m.) Friday, August 18, 2017 Friday, October 20, 2017 Friday, December 8, 2017 Friday, February 2, 2018 (cancelled due to the conflict with CAASPP Institute) Thursday, March 22, 2018 Friday, May 18, 2018

Training Materials May-June, 2017: 2017 CAASPP Post-Test Workshop Part 1: Accessing Score, Reports, Resources and Tools Part 2: Analyzing Summative Assessment Results to Inform Teaching and Learning September 2017: Preparing Technology for Online Testing October 3, 2017: 2017 Assessment and Accountability Information Meeting October 13, 2017: Formative Assessment Training (Part 1) October 17, 2017: CALPADS Information Meeting October 17, 2017: ELPAC Test Operations Management System (TOMS) Overview Webcast October 23, 2017: New CAASPP Coordinator’s Workshop November 13, 2017: Formative Assessment Training (Part 2) November 15, 2017: Accessibility and Accommodations December 11-12, 2017: CAASPP Institute Training Materials

Upcoming Conferences/Meetings/Trainings December 11-12, 2017: COE CAASPP Institute TOT, RCOE-Riverside. December 14, 2017: ELPAC Administration and Scoring Training, RCOE-Indio. December 19, 2017 or January 23, 2018: ELPAC Administration and Scoring Training, RCOE-Riverside. February 1-2, 2018: District CAASPP Institute, Riverside Convention Center. February 6-7, 2018: NGSS: Exploring Performance Assessments in Science (EPAS) Workshop for Secondary School Educators (Grades 6-12) Leading by SCALE, UC-Riverside Extension. February 20, 2018: CAASPP Pretest Workshop, UC-Riverside. April 9-10, 2018: Building Educator Assessment Literacy (BEAL) Mathematics Training Leading by SCALE, RCOE-Riverside.