Regional Assessment Network (RAN) Meeting Analysis, Measurement, and Accountability Reporting Division November 16, 2016

Agenda Overview of the September and November State Board of Education (SBE) Meetings Next Steps on the Academic Indicator An In-depth Look into the State Indicators: Graduation Rate Indicator Suspension Rate Indicator English Learner Indicator An In-depth Look into the College/Career Indicator (local indicator for initial release)

Agenda (Cont.) In-Depth Look at Performance Categories Additional Information Questions for RAN Members

Overview of the September and November State Board of Education (SBE) Meetings

September SBE Meeting The SBE adopted the Local Control Funding Formula (LCFF) Evaluation Rubrics, which will report: State indicators (data pre-populated by the CDE) Local indicators (data populated by LEAs) The SBE also established performance standards for select state and local indicators CDE= California Department of Education LEA=local educational agency Local Indicators will be uploaded locally

September SBE Meeting (Cont.) The SBE adopted performance standards for the following state indicators : Graduation Rate Indicator Suspension Rate Indicator English Learner Indicator (ELI) College/Career Indicator (CCI)

September SBE Meeting (Cont.) The SBE directed CDE staff to bring a recommended performance standards for the Academic Indicator to the November 2016 SBE meeting. Two years of Data for

November SBE Meeting The initial recommended performance standards for the Academic Indicator were based on the percent of students who scored “Standard Met” or “Standard Exceeded” on the Smarter Balanced Assessments for grades three through eight. Grade eleven assessment results are captured in the College/Career Indicator. Grades 3 - 8

November SBE Meeting (Cont.) The SBE decided not to release the Academic Indicator using “Standard Met” or “Standard Exceeded” voicing concern that this closely paralleled the Adequate Yearly Progress (AYP) methodology, which resulted in many schools focusing only on those students who were closest to proficient. Concerns about it felt too much like AYP and the bubble kids

November SBE Meeting (Cont.) Rather, the SBE wants to encourage districts and schools to improve the academic achievement of all students in the new accountability system. Therefore, the SBE requested the CDE to work on a methodology that uses scale scores. Focus is on all students

Next Steps for the Academic Indicator

Academic Indicator CDE staff will continue to work with the Technical Design Group (TDG) to explore using a “Distance from Met” methodology, which uses scale scores. CDE staff will also meet with the testing vendor, Educational Testing Service (ETS), to ensure the validity of the “Distance from Met” methodology. Distance from met is an indicator from Charter School conversations.

Academic Indicator (Cont.) In January 2017, the CDE will bring the “Distance from Met” to the SBE as a possible option for the Academic Indicator. Pending SBE’s decisions at this meeting, the CDE will move forward with “Distance from Met” or pursue other options. Bringing Methodology and proposed cut scores in January. Desire is to approve and not wait until March. Board has a strong desire to make this work. Distance Met look at each student how far a students scale score from lowest from met. Average Scale Scores for the school the average student is 50 points above or below. Discussion for status and growth. Distance from Met. Model will be posted 10 days before January meeting. Replace the status portion of the previous academic indicator. Possible weighting of scores. The model is based on distributions will set the cut scores. Scores will be set for a certain number of years and then reset forward as scores improve Performance Indicator Blue, Green, Red You will continue to look at particular students.

An In-Depth Look into the State Indicators The goal of today’s presentation is to cover the calculation method for each of the state indicators so that you can determine the performance category (or color) based on your own data.

Handouts The 5 x 5 colored grid handouts include the cut scores specific for each indicator. These handouts are being provided as a resource and will not be covered in detail: Handout 1: Graduation Indicator Handout 2: Suspension Indicator Handout 3: English Learner Indicator (ELI) Handout 4: College/Career Indicator (CCI) Handout 6: The Denominator for All Indicators

Before We Dive In…. LEA Data Charter Schools Alternative Schools Because all charter schools are treated as LEAs under the LCFF, charter school data are not included in the LEA-level data. Alternative Schools Since a separate accountability system will be developed for alternative schools, all alternative schools (i.e., Alternative Schools Accountability Model [ASAM]) are also excluded from LEA-level data. Important Slide Charter and Alternative

Before We Dive In… (Cont.) Therefore, the LEA-level data for graduation and suspension rates will not match the data reported on the CDE’s DataQuest Web page.

Graduation Rate Indicator

Who Will Receive a Graduation Rate Indicator? This indicator applies to LEAs, schools, and student groups that have 30 or more students in the four-year cohort graduation. See Handout 1. School Level reporting is at 30 or more. LEA level is 15

Status For the initial release of the LCFF Evaluation Rubrics, the 2014–15 four-year cohort graduation rate, or the class of 2015 graduation data, will be used to determine Status.

Change Prior three-years of four-year cohort graduation data were used to calculate the three-year weighted average to determine Change. Note! This is the only state indicator that uses a three-year weighted average to calculate Change.

Three-Year Weighted Average Formula Class of 2012 Graduates + Class of 2013 Graduates + Class of 2014 Graduates   divided by Students in the Class of 2012 + Students in the Class of 2013 + Students in the Class of 2014

Three-Year Weighted Average Formula (Cont.) It is important to note that if the LEA, school, or student group does not have cohort data for all prior three graduating classes, then the weighted average for Change was calculated using the one or two years of available cohort data.

Current Status (Class of 2015) minus Three-Year Weighted Average Change Formula Current Status (Class of 2015) minus Three-Year Weighted Average

Example 1 Topaz High School Status Change: Class of 2015 Graduation Rate: 89.4% Change: Step 1: Obtain Cohort Graduation Data for the Prior Three Years Step 2: Calculate the Weighted Average Step 3: Calculate Change

Number of Students in the Cohort Steps 1 & 2 Prior Three-Year Data Number of Students in the Cohort Number of Graduates Class of 2014 3,346 2,916 Class of 2013 3,343 2,857 Class of 2012 3,558 2,912 Sum 10,247 8,685 Weighted Average: 8,685 / 10,247 = 84.8%

Current Status (Class of 2015) minus Three-Year Weighted Average Step 3 Step 3: Calculate Change Current Status (Class of 2015) minus Three-Year Weighted Average 89.4% minus 84.8% = +4.6%

Determine Performance Category Now that we have the Status and Change for Topaz High School, the school’s performance category (or color) can be determined. Use the 5 x 5 color grid specific for the Graduation Rate Indicator Reminder! Each indicator has its own unique 5 x 5 color grid(s).

Determine Performance Category (Cont.) Status: 89.4% Change: +4.6%

Future Update The SBE has asked the CDE to work with the TDG to analyze and investigate the incorporation of the five-year cohort graduation rate into this indicator. Based on the current SBE timeline, implementation of the four-and five-year cohort graduation rates may occur in 2018–19. TDG =‘s Technical Design Group.

Suspension Rate Indicator

Who Will Receive a Suspension Rate Indicator? LEAs, schools, and student groups that have 30 or more students enrolled will receive a Suspension Rate Indicator. The enrollment data are obtained from the California Longitudinal Pupil Achievement Data System (CALPADS).

Suspension Rate Rules If a student is suspended multiple times, the student is counted as being suspended only once. LEA Example: If a student was suspended: Five times at School A, Twice at School B, and Twice at School C The student would be counted as being suspended once at the LEA. Important Slide

Status and Change Status The 2014–15 suspension rate will be used for Status for the initial release of the LCFF Evaluation Rubrics. Change: Change uses current and prior year suspension rates. Change Formula: 2014–15 rate minus 2013–14 rate

Key Difference in Cut Scores Data simulations revealed that suspension data varies widely among LEA and school type. Therefore, multiple suspension cut scores were set for LEAs and schools based on their type. This resulted in six different sets of cut scores: Three at LEA-level: Elementary, High, and Unified Three at School-level: Elementary, Middle, and High

Key Difference in Cut Scores (Cont.) Having six different sets of cut scores also means six separate 5 x 5 color grids based on district or school type. See Handout 2.

Impact of Multiple Cut Scores Since cut scores were set separately for LEAs and schools, charter schools and single school districts could receive two performance categories (or two colors): (1) at the LEA-level and (2) at the school-level because: Charter schools are treated as LEAs under the LCAP, and Single school districts are treated as schools under the ESSA   LCAP= Local Control and Accountability Plan ESSA= Every Student Succeeds Act

Impact of Multiple Cut Scores (Cont.) Because the LEA and school performance categories (colors) could be inconsistent, the SBE approved holding charter schools and single school districts accountable for the suspension rate cut scores using the school-level cut scores. Therefore, charter schools and single school districts will only be held accountable for their school-level performance category (color).   Due to potential inconsistencies, school-level indictor will be the indicator not the LEA indicator. See previous slides.

Key Difference In Goal It is important to remember that for this indicator, the goal is reversed. For all other state indicators, the desired outcome and goal is to achieve a high percent for Status and Change.

Key Difference in Goal (Cont.) However, the desired outcome and goal for the Suspension Rate Indicator is to have a low suspension rate, which translates to a low percent for Status and Change. For this reason, the Status and Change levels on the 5 x 5 color grids are in reverse order compared to the grids for the other indicators.

Key Difference in 5 x 5 Grid Level Increased Significantly Increased   Maintained Declined Declined Significantly Very Low Yellow Green Blue Low Orange Medium High Red Very High

Assignment of Performance Category Schools that did not certify (or submit) suspension data in the CALPADS are automatically assigned the Orange performance category.

Ruby Elementary School Example 1 Ruby Elementary School Status 2014–15 Suspension Rate: 0.5% Change: Step 1: Obtain prior year rate: 0.3% Step 2: Calculate Change Current Rate minus Prior Rate 0.5% minus 0.3% = +0.2%

Determine Performance Category Now that we know the Status and Change for Ruby Elementary School, the school’s performance category (or color) can be determined. Remember! Use the 5 x 5 color grid specific for elementary schools.

Determine Performance Category (Cont.) Status: 0.5% Change: +0.2%

Emerald Unified School District Example 2 Emerald Unified School District Status 2014–15 Suspension Rate: 3.8% Change: Step 1: Obtain prior year rate: 4.1% Step 2: Calculate Change Current Rate minus Prior Rate 3.8% minus 4.1% = -0.3%

Determine Performance Category Determine Emerald Unified School District’s performance category (color) using the specific 5 x5 color grid for unified school districts.

Determine Performance Category (Cont.) Status: 3.8% Change: -0.3%

English Learner Indicator (ELI)

Who Will Receive an ELI? This indicator applies to LEAs and schools that have 30 or more students who took an annual California English Language Development Test (CELDT). Note: Because 86.2% of schools have no significant, or only one significant race/ethnic student group within the EL group, student group data will not be reported for the ELI.

ELI Data Sources The ELI determines progress through the use of two data sources: Annual CELDT results, and EL reclassification

CELDT Data The CELDT has five overall performance levels: Beginning Early Intermediate Intermediate Early Advanced Advanced

CELDT Data (Cont.) Because the CELDT Intermediate performance level has a large range of scale scores, many students stay in the intermediate level for multiple years. As a result, stakeholders advised, and the SBE approved, that this level be divided into two, for accountability purposes only, to recognize the substantial growth that can be made within this particular level.

CELDT Data Therefore, the ELI uses six overall CELDT performance levels: Beginning Early Intermediate Low Intermediate High Intermediate Early Advanced Advanced Six Performance Levels

ELI Model Annual CELDT takers must advance at least one CELDT performance level from prior year to current year to be included in the numerator of the ELI calculation. Examples: Prior Overall CELDT Performance Current Overall CELDT Performance Does the LEA or School Receive Credit? Early Intermediate Low Intermediate Yes High Intermediate Advanced Beginning No

ELI Model (Cont.) Students who scored Early Advanced or Advanced Proficient in the prior year and maintained that performance level for the current year will be included in the numerator for the ELI calculation. This is the only set of annual CELDT test takers who are not required to advance one CELDT performance level. ELs who were reclassified in the prior year will also be included in the numerator and denominator for the ELI calculation.

Students Reclassified in Prior Year ELI Model (Cont.) Previous CELDT Overall Level Current CELDT Beginning Early Intermediate Low Intermediate High Intermediate Early Advanced Early Advanced or Adv Not Proficient Adv Proficient + Students Reclassified in Prior Year

ELI Formula: Numerator Annual CELDT test takers who: Increased at least one CELDT level compared to the prior year Maintained Early Advanced/Advanced English Proficient ELs who were reclassified in the prior year

ELI Formula: Denominator Total number of annual CELDT test takers ELs who were reclassified in the prior year

Status and Change Status The percent of ELs who moved up a performance level from the 2014 to 2015 CELDT plus ELs who were reclassified between July 1, 2013 to June 30, 2014. Change Difference in Status from current year to prior year. Data Quest Calculations do not work for DQ includes charter and alternative schools. Data Quest Calculatons will be changed in include or exclude charter.

Assignment of Performance Category Schools that did not test at least 50 percent of their EL population in the CELDT are automatically assigned an Orange performance category. Determination of the 50 percent is based on the EL demographic data reported for mathematics in the Smarter Balanced Assessment file from the testing vendor.

Example 1 Gemstone High School Status 210/250 = 84% 20 students Step 1: Percent of annual CELDT test takers who advanced at least one performance level (or maintained Early Advanced/Advanced English Proficient) on the 2015 overall CELDT compared to the 2014 overall CELDT 210/250 = 84% Step 2: Number of ELs who were reclassified in prior year (2013–14) 20 students Step 3: Add reclassified students to the numerator and denominator of Step 1 and calculate the rate. 230/270 = 85%

Current Status minus Prior Status Example 1 (Cont.) Change: Step 1: Calculate Prior Year Status: 83% Step 2: Calculate Change Calculate Change: Current Status minus Prior Status 85% minus 83% = +2%

Determine Performance Category Use the specific 5 x 5 color grid for the ELI and determine the performance category (color) for the EL student group.

Determine Performance Category (Cont.) Status: 85% Change: +2%

An In-Depth Look into the College/Career Indicator

What is the Purpose of the CCI? The College/Career Indicator (CCI) was designed to emphasize that a high school diploma should represent completion of rigorous course work that prepared students for postsecondary.

CCI Measures The CCI Model currently contains the following measures (see Handout 5): Advanced Placement (AP) exam results Dual Enrollment Early Assessment Program (EAP) results for English language arts/literacy (ELA) and mathematics (Grade 11) a-g completion Career Technical Education (CTE) pathway completion

CCI Measures (Cont.) Potential Measures for the release in 2017–18: International Baccalaureate (IB) State Seal of Biliteracy Golden State Seal Merit Diploma IB Career–related Programme Articulated CTE Pathway

Students with Disabilities Students with the most severe cognitive disabilities (i.e., students who take the California Alternate Assessment) are removed from the calculation of the CCI.

First a Local Indicator Although the CCI was planned to be reported as a state indicator, the SBE approved that: The CCI will be reported as a local indicator for the initial release of the rubrics based on Status only. (The CDE will pre-populate the CCI data in the rubrics.)

Then a State Indicator The first time grade eleven Smarter Balanced Assessment results will be available to calculate both Status and Change for the CCI is in 2018–19 (class of 2016 and 2017). As a result, when the LCFF Evaluation Rubrics are released in fall 2017, the CCI will be reported as a state indicator based on Status only.

CCI Formula Students Who Receive a High School Diploma and Meet the CCI Benchmark of “Prepared” divided by Current Year Graduation Cohort

Example 1 By the time Minnie Hause received her high school diploma, she had completed the following during her four years in high school: Completed a-g Scored “Standard Met” in ELA and “Standard Nearly Met” in mathematics on the grade eleven Smarter Balanced Summative Assessments Completed one CTE pathway in marketing

Example 1 (Cont.) Using the CCI Model (table version or graphic format in Handout 5), and based on Minnie’s information on the prior slide, the highest measure that she meets is completion of a-g and CTE pathway within the CCI “Prepared” performance level. As a result, Minnie would be included in the numerator of the Status calculation of the CCI.

Status Only For the initial release of the CCI (as a local indicator), Status will be calculated based on the class of 2014. The following years of data will be used for the CCI measures: CCI Measure Data Source Year of Data AP exam The College Board 2014 Grade 11 EAP results STAR file From Educational Testing Service (ETS) 2013 a-g completion CALPADS (Yes/No filled in by LEA ) CTE pathway completion Last 3 yrs in high school (2012, 2013, 2014) Dual enrollment

In-Depth Look at Performance Categories

GREEN is the Target The target performance category (or color) is GREEN for all LEAs, schools, and student groups. Performance categories (or colors) tell more than just the current year’s status; they reflect information about status and change. In other words, if your school is improving, that’s already reflected in the performance category.

The Performance Categories Already Reflect Change….. For some indicators, earning RED one year and ORANGE the next year may not reflect improvement! (example to follow)

Using the 5 x 5 Grid for the Graduation Rate Indicator…

Moving From RED to ORANGE Does Not Always Mean Improvement! Example: An LEA has a 2015–16 graduation rate of 82.0%. The LEA’s prior three-year average graduation rate is 89.0% (Change = -7%). As a result, the performance category is: The 2016–17 graduation rate is 79.0%. The updated prior 3-year average graduation rate is 83.0% (Change = -4). As a result the performance category is: RED ORANGE

The Reverse is also True Example: An LEA has a 2015–16 graduation rate of 92.0%. The district’s prior 3-year average graduation rate is 85.0% (Change = +7%). As a result, the performance category is: The 2016–17 graduation rate is 94.0%. The updated prior 3-year average graduation rate is 91.0% (Change = +3). As a result, the performance category is: BLUE GREEN

Takeaways From These Examples Performance category already reflects change, so: Talking about how a performance category changes over time may not make sense. Discussions about trends should focus on trends in the underlying data. A red, orange, or yellow performance category means that there is more work to be done. A green or blue performance category means that the trajectory of performance is fine.

Additional Information

Small LEAs, Schools, and Student Groups Small LEAs and schools with a denominator of less than 30 students for many of the indicators will not have performance categories reported. See Handout 6. However, the LCFF Evaluation Rubrics will display Status and Change data for numbers between 11 to 29. Although this will not translate to a color, stakeholders will be able to view the Status and Change data.

Small LEAs, Schools, and Student Groups (Cont.) Not receiving a performance category (or color) means that the LEA/school will not be included in the eligibility list for support or interventions. Data for less than 11 students will not be reported due to privacy reasons.

Demographic Corrections For the initial release of the LCFF Evaluation Rubrics, the opportunity to make demographic corrections for the state indicators and the CCI has passed. For the release of the LCFF Evaluation Rubrics in fall 2017, the demographic correction process will vary for each state indicator.

Demographic Corrections (Cont.) Graduation Rate Indicator The demographic and program participation (i.e., EL, Students with Disabilities [SWDs], and Socioeconomically Disadvantaged [SED]) data are based on data extracted from the CALPADS Operational Data Store (ODS). CALPADS LEA Administrators and Accountability Coordinators will be notified of the (Fall 1) preview period and the correction window timeline.

Demographic Corrections (Cont.) Graduation Rate Indicator (Cont.) The CALPADS Calendar also identifies the submission, certification, and correction windows for the cohort graduation data: http://www.cde.ca.gov/ds/sp/cl/rptcalendar.asp

Demographic Corrections (Cont.) Suspension Rate Indicator The demographic and program participation data for suspension (or discipline) are based on the End-of-Year 3 submissions. LEAs and schools should view the CALPADS Calendar for certification deadlines and the correction window for discipline data.

Demographic Corrections (Cont.) English Learner Indicator CALPADS LEA Administrators and Accountability Coordinators will be notified of the correction window for the CELDT data. The reclassified data will be extracted from the CALPADS ODS mid-to-late June of each year. LEAs and schools should have their data corrected prior to the extraction as there will be no opportunities to correct data after the extraction date.

Demographic Corrections (Cont.) College/Career Indicator Because the CCI is based on the four-year graduation cohort, the demographic and program participation data used for this indicator is tied to the four-year cohort graduation data. As a result, LEAs and schools should review their graduation data during the (Fall 1) preview process and make appropriate corrections since the cohort data will be used for two indicators.

Questions?

Questions for RAN Members Is it appropriate that alternative schools’ results are not included in the LEA-level data? A student who is suspended multiple times at the same school (or same district) is only counted once in the numerator of the suspension rate. Should the numerator capture the number of times the student is actually suspended?