California Educational Research Association Analysis, Measurement, and Accountability Reporting Division November 30, 2016 Printed by the Riverside County Office of Education

Polling Questions Throughout this presentation we will be asking you a variety of questions to help guide further development of the accountability system. You will need to use your phone to respond to the questions. The polling URL is: http://etc.ch/LTc4

Polling Question What organization are you affiliated with? County Office District Office Research Organization Charter School CDE Other

Agenda Framework for the New Accountability and Continuous Improvement System Overview of the New Accountability System Local Control Funding Formula (LCFF) Evaluation Rubrics : Top Level Data Display Overview of State Indicators Status of the Academic Indicator

Agenda (Cont.) Status, Change, and Assigning a Color: The California Model An In-depth Look into State Indicators: Graduation Rate Indicator Suspension Rate Indicator English Learner Indicator An In-depth Look into the College/Career Indicator (local indicator for initial release)

Agenda (Cont.) In-Depth Look at Performance Categories Overview of Local Indicators Timeline

Framework for the New Accountability and Continuous Improvement System

Accountability Framework Performance Equity Improvement Transparency One single cohort system to meet local, state, and federal needs Preparing All Students for College, Career, Life, and Leadership in the 21st Century, Superintendent’s Task Force on Accountability and Continuous Improvement Report, May 2016.

Accountability and Continuous Improvement System: Components Local Control and Accountability Plan (LCAP) and Annual Update Address eight state priorities (ten for counties) Outline plans for expenditures Report on progress LCFF Evaluation Rubrics State and local indicators Performance standards Support and Assistance System Designed to support continuous improvement

Overview of the New Accountability System

New Accountability System On September 8, 2016, California’s State Board of Education (SBE) approved a new multiple measures accountability system. Rather than having two separate accountability systems (state and federal) as in prior years, the new integrated system captures local, state, and federal requirements.

New Accountability System (Cont.) The new accountability system is based on multiple measures which will provide a more complete picture of what contributes to a positive educational experience for students.

New Accountability System (Cont.) The multiple measures are based on the LCFF state priorities and are divided into two types of indicators: State indicators Local indicators

LCFF Evaluation Rubrics Both state and local indicators will be reported through the LCFF Evaluation Rubrics. The statutory purposes of these rubrics: Support LEAs in identifying strengths, weaknesses, and areas for improvement Assist in determining whether LEAs are eligible for technical assistance Assist the State Superintendent of Public Instruction in determining if LEAs are eligible for more intensive state support/intervention LEAs = local educational agencies

Polling Question Would you like to see the Top-Level Data Display for the new multiple measures accountability system? Yes No

Local Control Funding Formula Evaluation Rubrics: Top Level Data Display

Currently There are Four Reports Equity Report All Student Performance Report Status/Change Report Indicator Cluster Report

Overview of State Indicators

State Indicators Academic Indicator: Graduation Rate Indicator English language arts/literacy (ELA) assessment Mathematics assessment Graduation Rate Indicator Suspension Rate Indicator English Learner Indicator (ELI)—measures progress of English learners (ELs)

Future State Indicators Chronic Absenteeism Rate College/Career Indicator (measures postsecondary preparedness)

Status of the Academic Indicator

November SBE Meeting At the November 2016 SBE meeting, the proposed performance standards recommended for the Academic Indicator were based on the percent of students who scored “Standard Met” or “Standard Exceeded” on the Smarter Balanced Assessments for grades three through eight. Grade eleven assessment results are captured in the College/Career Indicator. SBE=State Board of Education

November SBE Meeting (Cont.) The SBE decided not to release the Academic Indicator using “Standard Met” or “Standard Exceeded” voicing concern that this closely paralleled the Adequate Yearly Progress (AYP) methodology, which would reward schools focusing only on those students who were closest to proficient.

November SBE Meeting (Cont.) Rather, the SBE wants to encourage districts and schools to improve the academic achievement of all students. Therefore, the SBE requested CDE staff to work on a methodology that uses scale scores. CDE= California Department of Education

Next Steps CDE staff met with the testing vendor, Educational Testing Service (ETS), to ensure the validity of using scale scores to measure distance from a standard point. Based on the information obtained from ETS staff, CDE staff will continue to work with the Technical Design Group (TDG) to explore options for this proposed methodology.

Next Steps (Cont.) In January 2017, the CDE plans to bring the a proposed methodology to the SBE with four possible options for the Academic Indicator. Distance from Met Distance from Nearly Met Distance from the Statewide Average Score (by grade) Distance from the Lowest Possible Scale Score

Next Steps (Cont.) Pending SBE’s decisions at this meeting, the CDE will move forward with the methodology and one of proposed criteria or pursue other options.

Next Steps (Cont.) In addition, the SBE will be discussing the definition of the EL student group in the Academic Indicator at their January 2017 meeting. In prior accountability reports, the ELA and mathematics results for the EL student group included results for both EL students and reclassified students.

Next Steps (Cont.) However, the SBE has asked CDE staff to explore removing reclassified fluent English proficient (RFEP) students from the EL student group. CDE staff are exploring multiple options.

Polling Question For the Academic Indicator, which criteria do you prefer for the proposed methodology? Distance From Met Distance From Nearly Met Distance from the Lowest Possible Scale Score Distance from the Statewide Average Score (by grade)

Polling Question For the Academic Indicator, which EL student group definition do you prefer? ELs plus 4-years of RFEP (maximum amount of time allowed under the Every Student Succeeds Act (ESSA) ELs plus 2-Years of RFEP ELs only

Polling Question On a scale of one to five, with five being the highest, how familiar are you with the 5X5 color grid to determine the five performance categories (red, orange, yellow, green, and blue)?

Status, Change, and Assigning a Color: The California Model

The California Model The California Model combines five “Status” and five “Change” levels which results in a performance category (or color) for each of the state indicators. The model provides equal weight to both Status and Change.

Five Status and Five Change Levels Five Status Levels Five Change Levels Very High Increased Significantly High Increased Medium Maintained Low Declined Very Low Declined Significantly

Status and Change Status is based on current year performance. Change is the difference between prior year performance and current year performance. Exception! The Graduation Rate Indicator is the only state indicator that uses a multi-year average rather than prior year data.

Methodology To determine overall performance, the percentile distributions for Status and Change were examined for each indicator: For Status, LEAs and schools were ordered from highest to lowest and four cut points were selected based on the distribution of all LEAs and schools. These cut points created the five Status Levels.

Methodology (Cont.) For Change, LEAs and schools were ordered separately from highest to lowest for positive change and lowest to highest for negative change. Cut points were determined separately for positive and negative change. A total of four cuts were selected which created five Change Levels.

Methodology (Cont.) Cut points will remain in place for a select number of years (e.g., 3 to 5 years) to be determined by the SBE.

Performance Category The combination of Status and Change results in a performance category that is assigned a color for each indicator: Blue Highest Green Yellow Orange Red Lowest

Who Gets a Performance Category? The model will be applied to all LEAs, schools, and student groups with 30 or more students. The data used to determine “30 or more” differs for each indicator. While a performance category (or color) will not be determined for LEAs, schools, or student groups with fewer than 30 students, the Status and Change data will be reported for groups with 11-29 students.

Who Gets a Performance Category? (Cont.) In these instances, an asterisk (*) will be reported to indicate that because there are fewer than 30 students, the LEA/school will not receive a performance category. Data for fewer than 11 students will not be reported due to privacy reasons and a “- - -” will be displayed. Data that is not yet available will be noted with an N/A.

An In-Depth Look into the State Indicators One of the goals of this presentation is to provide the necessary information for you to calculate performance category (or color) for each state indicator.

Before We Dive In…. LEA Data Charter Schools Alternative Schools Because all charter schools are treated as LEAs under the LCFF, charter school data are not rolled up to their authorizing agency’s data. Alternative Schools Since a separate accountability system will be developed for alternative schools, all alternative schools (i.e., Alternative Schools Accountability Model [ASAM]) are also excluded from LEA-level data.

Before We Dive In… (Cont.) Therefore, the LEA-level data for graduation and suspension rates will not match the data reported on the CDE DataQuest Web page.

Polling Question Should alternative schools’ results be included in the LEA-level data even though an alternative accountability system will be developed? Yes, include No, do not include

Graduation Rate Indicator

Who Will Receive a Graduation Rate Indicator? This indicator applies to LEAs, schools, and student groups that have 30 or more students in the four-year graduation cohort. Note: Students who earn a Special Education Certificate of Completion or a General Education Degree (GED) are not counted as high school graduates but are included in the denominator.

Status For the initial release of the LCFF Evaluation Rubrics, the 2014–15 cohort graduation rate, or the class of 2015 graduation data, will be used to determine Status.

Change Prior three-years of four-year cohort graduation data were used to calculate the three-year weighted average to determine Change. Note: This is the only state indicator that uses a three-year weighted average to calculate Change.

Three-Year Weighted Average Formula Class of 2012 Graduates + Class of 2013 Graduates + Class of 2014 Graduates   divided by Students in the Class of 2012 + Students in the Class of 2013 + Students in the Class of 2014

Three-Year Weighted Average Formula (Cont.) It is important to note that if the LEA, school, or student group does not have cohort data for all prior three graduating classes, then the weighted average for Change was calculated using the one or two years of available cohort data.

Current Status (Class of 2015) minus Three-Year Weighted Average Change Formula Current Status (Class of 2015) minus Three-Year Weighted Average

Determine Performance Category (Cont.) Status: 89.4% Change: +4.6%

Future Plans At the SBE’s request, CDE staff will work with the TDG to analyze and investigate the incorporation of the five-year cohort graduation rate into this indicator. Based on the current timeline, implementation of the four-and five-year cohort graduation rates may occur in 2018–19.

Suspension Rate Indicator

Who Will Receive a Suspension Rate Indicator? LEAs, schools, and student groups that have 30 or more students enrolled will receive a Suspension Rate Indicator. The enrollment data are obtained from the California Longitudinal Pupil Achievement Data System (CALPADS).

Suspension Rate Rules If a student is suspended multiple times (at the same school or district), the student is counted as being suspended only once. LEA Example: If a student was suspended: Five times at School A, Twice at School B, and Twice at School C The student would be counted as being suspended once at the LEA.

Status and Change Status: The 2014–15 suspension rate will be used for Status for the initial release of the LCFF Evaluation Rubrics. Change: Change uses current and prior year suspension rates. Change Formula: 2014–15 rate minus 2013–14 rate

Key Differences in Suspension Cut Scores Data simulations revealed that suspension data varies widely among LEA and school type. Therefore, multiple suspension cut scores were set for LEAs and schools based on their type. This resulted in six different sets of cut scores: Three at LEA-level: Elementary, High, and Unified Three at School-level: Elementary, Middle, and High

Key Differences in Suspension Cut Scores (Cont.) Having six different sets of cut scores also means six separate 5 x 5 color grids based on district or school type.

Impact of Multiple Cut Scores Since cut scores were set separately for LEAs and schools, charter schools and single school districts could receive two performance categories (or two colors): (1) at the LEA-level and (2) at the school-level because: Charter schools are treated as LEAs under the LCAP, and Single school districts are treated as schools under the ESSA   LCAP= Local Control and Accountability Plan ESSA= Every Student Succeeds Act

Impact of Multiple Cut Scores (Cont.) Because the LEA and school performance categories (colors) could be inconsistent, the SBE approved holding charter schools and single school districts accountable for the suspension rate cut scores using the school-level cut scores. Therefore, charter schools and single school districts will only be held accountable for their school-level performance category (color).  

Key Differences in the Suspension Goal It is important to remember that for this indicator, the goal is reversed. For all other state indicators, the desired outcome and goal is to achieve a high percent for Status and Change.

Key Differences in the Suspension Goal (Cont.) However, the desired outcome and goal for the Suspension Rate Indicator is to have a low suspension rate, which translates to a low percent for Status and Change. For this reason, the Status and Change levels on the 5 x 5 color grids are in reverse order compared to the grids for the other indicators.

Key Differences in 5 x 5 Grid Level Increased Significantly Increased   Maintained Declined Declined Significantly Very Low Yellow Green Blue Low Orange Medium High Red Very High

Assignment of Performance Category Schools that did not certify (or submit) suspension data in the CALPADS are automatically assigned the Orange performance category.

Determine Performance Category (Cont.) Status: 0.5% Change: +0.2%

Polling Question Currently, a student who is suspended multiple times at the same school (or same district) is only counted once in the suspension rate. How should suspensions be counted in the Suspension Indicator? Include only the student’s first suspension Include all of the student’s suspensions

English Learner Indicator (ELI)

Who Will Receive an ELI? This indicator applies to LEAs and schools that have 30 or more students who took the annual California English Language Development Test (CELDT). Note: Because 86.2% of schools have no significant, or only one significant race/ethnic student group within the EL group, student group data will not be reported for the ELI.

ELI Data Sources The ELI determines progress through the use of two data sources: Annual CELDT results, and EL reclassification

CELDT Data The CELDT has five overall performance levels: Beginning Early Intermediate Intermediate Early Advanced Advanced

CELDT Data (Cont.) Because the CELDT Intermediate performance level has a large range of scale scores, many students stay in the intermediate level for multiple years. As a result, stakeholders advised, and the SBE approved, that this level be divided into two, for accountability purposes only, to recognize the substantial growth that can be made within this particular level.

CELDT Data (Cont.) Therefore, the ELI uses six overall CELDT performance levels: Beginning Early Intermediate Low Intermediate High Intermediate Early Advanced Advanced

ELI Model Annual CELDT takers must advance at least one CELDT performance level from prior year to current year to be included in the numerator of the ELI calculation. Examples: Prior Overall CELDT Performance Current Overall CELDT Performance Does the LEA or School Receive Credit? Early Intermediate Low Intermediate Yes High Intermediate Advanced Beginning No

ELI Model (Cont.) Students who scored Early Advanced or Advanced Proficient in the prior year and maintained that performance level for the current year will be included in the numerator for the ELI calculation. This is the only set of annual CELDT test takers who are not required to advance one CELDT performance level. ELs who were reclassified in the prior year will also be included in the numerator and denominator for the ELI calculation.

Students Reclassified in Prior Year ELI Model (Cont.) Previous CELDT Overall Level Current CELDT Beginning Early Intermediate Low Intermediate High Intermediate Early Advanced Early Advanced or Adv Not Proficient Adv Proficient + Students Reclassified in Prior Year

ELI Formula: Numerator Annual CELDT test takers who: Increased at least one CELDT level compared to the prior year Maintained Early Advanced/Advanced English Proficient ELs who were reclassified in the prior year

ELI Formula: Denominator Total number of annual CELDT test takers ELs who were reclassified in the prior year

Status and Change Status The percent of ELs who moved up a performance level from the 2014 to 2015 CELDT plus ELs who were reclassified between July 1, 2013 to June 30, 2014. Change Difference in Status from current year to prior year.

Assignment of Performance Category Schools that did not test at least 50 percent of their EL population in the CELDT are automatically assigned an Orange performance category. Determination of the 50 percent is based on the EL demographic data reported for mathematics in the Smarter Balanced Assessment file from the testing vendor.

Example 210/250 = 84% 20 students 230/270 = 85% Step 1: Percent of annual CELDT test takers who advanced at least one performance level (or maintained Early Advanced/Advanced English Proficient) on the 2015 overall CELDT compared to the 2014 overall CELDT 210/250 = 84% Step 2: Number of ELs who were reclassified in prior year (2013–14) 20 students Step 3: Add reclassified students to the numerator and denominator of Step 1 and calculate the rate. 230/270 = 85%

Determine Performance Category Status: 85% Change: +2%

Activity You have 10 minutes! Each table has a sheet of paper containing the directions for your task. Your task is determine the performance category based on the data provided, and to populate the LCFF Evaluation Rubrics posted around the room with your performance category (color) results. You have 10 minutes!

An In-Depth Look into the College/Career Indicator

What is the Purpose of the CCI? The College/Career Indicator (CCI) was designed to emphasize that a high school diploma should represent completion of rigorous course work that prepared students for postsecondary.

CCI Measures The CCI currently contains the following measures: Advanced Placement (AP) exam results Dual Enrollment Early Assessment Program (EAP) results for ELA and mathematics (Grade 11) a-g completion Career Technical Education (CTE) pathway completion

CCI Measures (Cont.) Potential measures for the release in 2017–18: International Baccalaureate (IB) State Seal of Biliteracy Golden State Seal Merit Diploma IB Career–related Programme Articulated CTE Pathway

Students with Disabilities Students with the most severe cognitive disabilities (i.e., students who take the California Alternate Assessment) are removed from the calculation of the CCI.

First a Local Indicator Although the CCI was planned to be reported as a state indicator, the SBE approved: Reporting the CCI as a local indicator for the initial release of the LCFF Evaluation Rubrics based on Status only because the current CCI does not contain the Smarter Balanced results. Note: The CDE will pre-populate the CCI data.

Then a State Indicator When the LCFF Evaluation Rubrics are released in fall 2017, the CCI will be reported as a state indicator based on Status only because only one year of Smarter Balanced results will be available. The first time grade eleven Smarter Balanced Assessment results will be available to calculate both Status and Change for the CCI is in 2018–19 (Class of 2016 and 2017).

CCI Formula Students Who Receive a High School Diploma and Meet the CCI Benchmark of “Prepared” divided by Current Year Graduation Cohort

Status Only For the initial release of the CCI (as a local indicator), Status will be calculated based on the Class of 2014. The following years of data will be used for the CCI measures: CCI Measure Data Source Year of Data AP exam The College Board 2014 Grade 11 EAP results STAR file From Educational Testing Service (ETS) 2013 a-g completion CALPADS (Yes/No filled in by LEA ) CTE pathway completion Last 3 yrs in high school (2012, 2013, 2014) Dual enrollment

In-Depth Look at Performance Categories

Green is the Target The target performance category (or color) is Green for all LEAs, schools, and student groups. Performance categories (or colors) tell more than just the current year’s status; they reflect information about status and change. In other words, if your school is improving, that’s already reflected in the performance category.

The Performance Categories Already Reflect Change….. For some indicators, earning Red one year and Orange the next year may not reflect improvement!

Moving From RED to ORANGE Does Not Always Mean Improvement! Example: An LEA has a 2015–16 graduation rate of 82.0%. The LEA’s prior three-year average graduation rate is 89.0% (Change = -7%). As a result, the performance category is: The 2016–17 graduation rate is 79.0%. The updated prior 3-year average graduation rate is 83.0% (Change = -4). As a result the performance category is: Red Orange

The Reverse is also True Example: An LEA has a 2015–16 graduation rate of 92.0%. The district’s prior 3-year average graduation rate is 85.0% (Change = +7%). As a result, the performance category is: The 2016–17 graduation rate is 94.0%. The updated prior 3-year average graduation rate is 91.0% (Change = +3). As a result, the performance category is: Blue Green

Takeaways From These Examples Performance category already reflects change, so: Talking about how a performance category changes over time may not make sense. Discussions about trends should focus on trends in the underlying data. A red, orange, or yellow performance category means that there is more work to be done. A green or blue performance category means that the trajectory of performance is fine.

Overview of Local Indicators

Local Indicators Basic (Priority 1): Appropriately assigned teachers Access to curriculum-aligned instructional materials Safe, clean, and functional school facilities Implementation of State Academic Standards (Priority 2) Parent Engagement (Priority 3) School Climate (Priority 6)

Local Indicators (Cont.) For county offices of education (COEs) only: Coordination of Services for Expelled Students (Priority 9) Coordination of Services for Foster Youth (Priority 10)

Other Local Indicators The following local indicators will be populated by the CDE: College/Career Indicator (initially will be reported in the rubrics as a local indicator for informational purposes only.) Grade eleven Smarter Balanced Assessment results.

Meeting Standards For each local indicator, LEAs are responsible for: Using locally available information to measure progress relative to performance standards and criteria adopted by the SBE. Reporting the results at a regularly scheduled meeting of the local governing board. Reporting the results through the Evaluation Rubrics.

Meeting Standards (Cont.) LEAs will use the following standards to assess performance on the local indicators: Met Not Met Not Met for Two or More Years These results will be populated by the LEAs in the LCFF Evaluation Rubrics.

SBE November Decisions At the November 2016 SBE meeting, the SBE approved the following tools for the Local indicators: Basic (Priority 1): These data are collected through the School Accountability Report Card (SARC).

SBE November Decisions (Cont.) LEAs will use locally available information, including data reported through the SARC, to provide evidence of progress. The Web-based interface system for the Evaluation Rubrics is being developed based on the same data system that supports the SARC template.

SBE November Decisions (Cont.) The goal is for the rubrics system to auto-populate the necessary SARC data for LEAs that use the SARC template.

SBE November Decisions (Cont.) School Climate (Priority 6): At least once every other year, LEAs administer a survey in at least one grade within each grade span served (K-5, 6-8 and 9-12). Provide a narrative summary of the local survey and the analyses of the survey in a text box provided in the Evaluation Rubrics Web-based user interface.

SBE November Decisions (Cont.) Coordination of Services for Expelled Students (Priority 9): COEs determine progress using the self-reflection tool adopted by the SBE. COEs reports results through the local data selection option in the Evaluation Rubrics Web-based user interface.

SBE November Decisions (Cont.) Coordination of Services for Foster Youth (Priority 10) COEs determine progress using the self-reflection tool adopted by the SBE. COEs reports results through the local data selection option in the Evaluation Rubrics Web-based user interface.

Next Steps for Local Indicators The CDE will bring recommended standards for the Implementation of State Academic Standards (Priority 2) and Parent Engagement (Priority 3) to the SBE in January 2017.

Timeline

Timeline The LCFF Evaluation Rubrics will be released in 2017 to LEAs first for their preview. LEA superintendents have been notified by the CDE to designate a “LCFF Rubrics Coordinator” to compile and coordinate feedback and/or responses from stakeholders.

Timeline (Cont.) The LCFF Rubrics Coordinators will be provided access to the LCFF Evaluation Rubrics when they are made available. After LEAs preview the LCFF Evaluation Rubrics, the public release will occur. The CDE is releasing the data to provide LEAs and the public an opportunity to become familiar with the LCFF Evaluation Rubrics before they are used to determine technical assistance or intervention.

Timeline (Cont.) Beginning in the 2017–18 school year, the data reported in the LCFF Evaluation Rubrics will be used to determine LEA eligibility for technical assistance or interventions. In addition, the release of the Evaluation Rubrics in the fall each year will make data available for LEAs to use in the development of their LCAPs.

Timeline (Cont.) Because one of the goals of the new accountability system is continuous improvement, the SBE has established an annual review cycle beginning in March 2017, with any potential action taken no later than September of each year, to continually evolve the Evaluation Rubrics based on user experiences and stakeholder feedback, as well as the availability of new data.

Questions?

Contact Information Questions regarding the Evaluation Rubrics should be directed to the Academic Accountability Unit by phone at 916-319-0863 or by e-mail lcffrubrics@cde.ca.gov