1. Assessment and Accountability Update
Chun-Wu Li, Ph.D.
Administrator II, Assessment and Accountability
Assessment Accountability Continuous Improvement Unit
Division of Educational Services
December 8, 2017

2. Assessment Update

3. English Language Proficiency Assessments for California (ELPAC)
(See Handout “The ELPAC and CSA-On the Road to Completion but in Different Gears”)

4. New Practice Test for ELPAC

5. New Resources http://www.cde.ca.gov/ta/tg/ep/documents/howtotestel.pdf

6. New Resources (cont.)

7. New Resources (cont.)

8. California Assessment of Student Performance and Progress
(CAASPP)

9. Accessibility and Accommodations
Educational Testing Service (ETS) presented a Webcast on November 15, 2017 that includes information about the CAASPP accessibility and accommodations to prepare LEAs for effective implementation of the 2017–18 student accessibility resources.
Topics include 2017–18 student accessibility highlights, updated Matrix One, 2017–18 CAASPP Individual Student Assessment Accessibility Profile (ISAAP) Tool, and the Unlisted Resources approval and implementation process.
The archived webinar can be located at Accessibility and Accommodations.

10. Interim Assessment User Guide
The CAASPP 2017–18 Interim Assessment User Guide is now available on the Interim Assessment Administration Resources page of the CAASPP Portal Web site. The guide provides instructions for LEA staff using the interim assessment online systems, including: 
Interim Assessment Viewing System
Interim Assessment Test Delivery System
Interim Assessment Hand Scoring System
Interim Assessment Reporting System

11. Interim Assessment Reporting System Video
A demonstration video for the new Interim Assessment Reporting System is now available on the Interim Assessment Administration Resources page of the CAASPP Portal Web site. This 10-minute reporting system video provides a system demonstration of features including:  
how to create and manage student groups,
where to view aggregate student results,
how to see detailed item information, and
how to access student responses to test items—which are just a few of the capabilities the new system provides.

12. Digital Library demonstration Video
Located on the CAASPP Portal Web site, there is the Smarter Balanced Digital Library demonstration video that is available for Digital Library users. This four-minute video guides users through the features of the system’s new search interface. It illustrates how users can optimize the functionality of their searches and ensure that they have access to the high-quality instructional and professional resources they need to strengthen their formative assessment practices.

13. California Alternate Assessments (CAAs)
The 2017–18 One Percent Threshold Justification Form is now available for LEAs to complete that think they will exceed one percent. The form must be submitted by December 15, Now available on the CAASPP Reporting Web site are the results from 2016–17 One Percent Threshold Justification Form. This Excel file lists the LEAs that reported they may have exceeded the 1% threshold on alternate assessments for 2016−17.
 On November 1, the CAA Test Examiner Tutorial video is now available. The tutorial is divided into sections: CAAs for ELA and mathematics and CAA for Science.

14. California Alternate Assessments Science
The 2017–18 embedded performance tasks (PTs) for the year two pilot are now available on TOMS. Six PTs have been posted, three PTs for grade five, three PTs for grade eight, and three for high school. 
Beginning December 2, 2017, LEAs will be able to assign the CAA for Science to all grade twelve students who are eligible for the CAA for Science. In addition, LEAs will be able to assign students who are in grade ten or eleven to take the CAA for Science. 
Draft blueprint, performance level descriptors, and scoring structure CAA for Science will be presented to the State Board of Education (SBE) for approval in January 2018.

15. California Science Test (CAST)
The draft CAST blueprint, draft achievement level descriptors, and score reporting structure were presented to the SBE for approval on November 8, 2017.
 All grade twelve students will be assigned the CAST for spring, LEAs also have the option of assigning the CAST to grade ten or eleven students beginning on December 2, 2017.

16. Accountability Update
(Additional Handout “California’s New Accountability
System and School Dashboard”)

17. SBE Action on Academic Indicator
Approved:
New Change cut scores for English language arts/literacy (ELA) and mathematics
New High and Medium Status cut scores for mathematics
New color designation for five-by-five grid
California Department of Education

18. Background on Academic Indicator
Based on Smarter Balanced Summative Assessments results for ELA and mathematics
Fall 2017 Dashboard uses data from spring administration for grades 3–8
Status and Change calculated using “Distance from Level 3” (DF3)
California Department of Education

19. California Department of Education
Distance from Level 3
Distance between:
Student’s score on Smarter Balanced Summative Assessments, and
Lowest possible scale score* for Standard Met Achievement Level (Level 3)
*Scale scores vary by content area and grade level.
SBE changes do not affect how DF3 is calculated.
As a reminder, DF3 is the difference between a student’s scale score on the SBSA and the lowest possible score – for that grade level and content area) – for Level 3 (which is the Standard Met Achievement Level)
California Department of Education

20. DF3 Example: Grade Five Mathematics
Grade five student received a score of 2,505.
Here’s an example.
In grade 5 mathematics, the scale score for the Standard Met Achievement Level from 2,525 and 2,578.
So 2,525 is the lowest possible score for Level 3. It represents the threshold for Level 3. Every student score will be compared to that fixed number.
Therefore, a score of 2,505 is 23 points below the Standard 3 threshold. The DF3 is – 23.
When calculating Status for the Academic Indicator, each student’s DF3 is calculated separately, and then all distances are combined to determine an average at the LEA, school, and student group levels.
Student scored 2,505. This is 23 points below the lowest possible score for Level 3.
(2,528 – 2,505 = 23 points)
California Department of Education

21. Volatility in Scale Scores
Performance standards based on two years of data (2015 and 2016)
Distributions positively skewed: Over 80 percent of LEAs had positive change.
2017 Change distribution more symmetrical: 45 percent of LEAs had a positive change
In the next several slides, we are going to look at some of the issues that arose with the original cut scores and color designations and how the SBE actions attempt to remedy these issues.
When the standards were set for the Academic Indicator, we only had two years of Smarter Balanced assessment data available: for 2015 and 2016.
As it turned out, the distributions were positively skewed, with over 80 percent of districts showing a positive Change.
However, when we looked at the results for the 2017 assessments, we saw a more symmetrical distribution: Only 45% of districts saw a positive Change.
California Department of Education

22. Produces Dramatic Downward Swings
High performing schools must sustain continued improvement that may be untenable.
Some 200 schools at “High” or “Very High” Status level dropped two performance levels in 2017 (from Blue to Yellow), even as most of their students continued to meet or exceed state standards.
Example: School with a DF3 of in 2016 and in 2017 would move from a Blue to Yellow performance level.
California Department of Education

23. Produces Dramatic Upward Swings
Schools with Very Low Status that improve by just 15 points from prior year could jump two performance levels (i.e., Red to Yellow).
Applying current cut scores and methodology to 2017 DF3 would result in 1,117 schools (15 percent) moving up or down at least two performance levels in ELA.
California Department of Education

24. New Change Cut Scores for Both English language arts and Mathematics
Increased Significantly
By 15 points or more
Increased
by 3 to less than 15 points
Maintained
Declined by less than 3 points or Increased by less than 3 points
Declined
By 3 to 15 points
Declined Significantly
by more than 15 points
California Department of Education

25. Rationale: Revisions to Change Cut Scores
Normal distribution supports symmetrical cut scores
2016 cut scores (“Maintained” level):
˂ -1 to ˂ +7 (ELA)
˂ -1 to ˂ +5 (mathematics)
Revised Change cut scores:
˂ -3 to ˂ +3 (both ELA and mathematics)
Based on 2017 data, the distribution for Change is a normal bell curve, so cut scores should be more symmetrical to align with those established for the other indicators.
Current cut scores for “Maintained” level range from less than -1 to less than +7 (for ELA) and less than -1 to less than +5 (for mathematics)
The revised Change cut score for “Maintained” ranges from less than negative three (-3) to less than positive three (+3)
California Department of Education

26. Revisions to Mathematics Cut Scores
Status Cut Scores
Very High
35 or higher
High
Zero to 34.9 points (revised)
Medium
-25 points below to less than Zero (revised)
Low
-25.1 points to -95 points
Very Low -95 points or lower
California Department of Education

27. Rationale: Revisions to Status Cut Scores for Mathematics
Ensure that LEAs and schools cannot receive a “High” Status unless they have a positive DF3:
Addresses issues raised by advocacy groups after release of Spring 2017 Dashboard
Addresses issues raised by advocacy groups after release of Spring 2017 Dashboard
California Department of Education

28. New ELA Five-by-Five Colored Table
Level
Change: Declined Significantly
by more than 15 points
Change: Declined
By 3 to 15 points
Change: Maintained
Declined by less than 3 points or Increased by less than 3 points
Change: Increased
by 3 to less than 15 points
Change: Increased Significantly
By 15 points or more
Status: Very High
45 points or higher
Green*
Green
Blue
Status: High 10 to 44.9 points
Status: Medium
-5 points to +9.9 points
Yellow*
Yellow
Status: Low
-5.1 to -70 points
Orange*
Orange
Status: Very Low points or lower
Red
* Change in color designation
California Department of Education

29. New Mathematics Five-by-Five Colored Table
Level
Change: Declined Significantly
by more than 15 points
Change: Declined
By 3 to 15 points
Change: Maintained
Declined by less than 3 points or Increased by less than 3 points
Change: Increased
by 3 to less than 15 points
Change: Increased Significantly
By 15 points or more
Status: Very High
35 points or higher
Green*
Green
Blue
Status: High zero to 34.9 points
Status: Medium
-25 points to less than zero
Yellow*
Yellow
Status: Low
-25.1 to -95 points
Orange*
Orange
Status: Very Low -95 points or lower
Red
* Change in color designation
California Department of Education

30. Rationale: Revisions to Color Designation
LEAs and schools that maintain their Status can only be assigned one of two colors:
School with Very High Status can only move from Blue to Green if it declines.
School will Very Low Status can only move from Red to Orange if it increases.  
Revised color designation limits each row to two colors, ensuring that LEAs and schools that maintain their Status will always be assigned one of two colors, regardless of their Change.
California Department of Education

31. Feedback on Academic Indicator
What information should be included in a one- page flyer on the Academic Indicator:
Should contain graphics
Should be clear for general public
California Department of Education

32. SBE Action on Chronic Absenteeism Indicator
Include information in Fall 2017 Dashboard that redirects users to Chronic Absenteeism reports on DataQuest.
Direct CDE staff to develop recommendation for:
March 2018 SBE meeting: Proposed Status cut scores to be used to update Fall 2017 Dashboard
September or November 2018 SBE meeting: Proposed Change cut scores
California Department of Education

33. Feedback on College/Career Indicator (CCI)
On Equity page of the Dashboard, CCI row reflects “N/A”
Causing confusion and generating many questions.
Is there more explanatory language that we can use? (Example: Status Only) 
California Department of Education

34. This the final step in a series of actions by the SBE related the development of the new accountability system.
The system is designed for continuous improvement in mind.
In March we will begin the process again, reviewing indicators for and making recommendations for changes to the indicators.
California Department of Education

35. New Communication Materials
California Department of Education

36. Flyers on the Dashboard
Sacramento County Office of Education is developing a series of one- and two-page flyers on the Dashboard:
Five-by-Five Colored Grids
Academic Indicator
CCI
Dashboard Alternative School Status (DASS)
English Learner Subgroups
Distribute the Five-by-Five Colored Grids and DASS fliers and ask for feedback on the DASS
California Department of Education

37. Dashboard Release Schedule
California Department of Education

38. Release Schedule for the Fall 2017 Dashboard
Early November 2017
LEA private preview began
Rolling weekly release of indicators
Release of Academic Indicator: November 16
Week of November 27, 2017
Public launch of the Dashboard
California Department of Education

39. Assessment and Accountability Related Items

40. 2017-2018 RCAN Dates (Revised) 2017-2018 RCAN Dates
(Regular Meeting: 9:00 a.m.-12:00 p.m.; Mini Workshop: 12:30 p.m.-2:30 p.m.)
Friday, August 18, 2017
Friday, October 20, 2017
Friday, December 8, 2017
Friday, February 2, 2018 (cancelled due to the conflict with CAASPP Institute)
Thursday, March 22, 2018
Friday, May 18, 2018

41. Training Materials May-June, 2017: 2017 CAASPP Post-Test Workshop
Part 1: Accessing Score, Reports, Resources and Tools
Part 2: Analyzing Summative Assessment Results to Inform Teaching and Learning
September 2017: Preparing Technology for Online Testing
October 3, 2017: 2017 Assessment and Accountability Information Meeting
October 13, 2017: Formative Assessment Training (Part 1)
October 17, 2017: CALPADS Information Meeting
October 17, 2017: ELPAC Test Operations Management System (TOMS) Overview Webcast
October 23, 2017: New CAASPP Coordinator’s Workshop
November 13, 2017: Formative Assessment Training (Part 2)
November 15, 2017: Accessibility and Accommodations
December 11-12, 2017: CAASPP Institute Training Materials

42. Upcoming Conferences/Meetings/Trainings
December 11-12, 2017: COE CAASPP Institute TOT, RCOE-Riverside.
December 14, 2017: ELPAC Administration and Scoring Training, RCOE-Indio.
December 19, 2017 or January 23, 2018: ELPAC Administration and Scoring Training, RCOE-Riverside.
February 1-2, 2018: District CAASPP Institute, Riverside Convention Center.
February 6-7, 2018: NGSS: Exploring Performance Assessments in Science (EPAS) Workshop for Secondary School Educators (Grades 6-12) Leading by SCALE, UC-Riverside Extension.
February 20, 2018: CAASPP Pretest Workshop, UC-Riverside.
April 9-10, 2018: Building Educator Assessment Literacy (BEAL) Mathematics Training Leading by SCALE, RCOE-Riverside.