1. Regional Assessment Network (RAN) Meeting
Analysis, Measurement, and Accountability Reporting Division
November 16, 2016

2. Agenda
Overview of the September and November State Board of Education (SBE) Meetings
Next Steps on the Academic Indicator
An In-depth Look into the State Indicators:
Graduation Rate Indicator
Suspension Rate Indicator
English Learner Indicator
An In-depth Look into the College/Career Indicator (local indicator for initial release)

3. Agenda (Cont.) In-Depth Look at Performance Categories
Additional Information
Questions for RAN Members

4. Overview of the September and November State Board of Education (SBE) Meetings

5. September SBE Meeting
The SBE adopted the Local Control Funding Formula (LCFF) Evaluation Rubrics, which will report:
State indicators (data pre-populated by the CDE)
Local indicators (data populated by LEAs)
The SBE also established performance standards for select state and local indicators
CDE= California Department of Education
LEA=local educational agency
Local Indicators will be uploaded locally

6. September SBE Meeting (Cont.)
The SBE adopted performance standards for the following state indicators :
Graduation Rate Indicator
Suspension Rate Indicator
English Learner Indicator (ELI)
College/Career Indicator (CCI)

7. September SBE Meeting (Cont.)
The SBE directed CDE staff to bring a recommended performance standards for the Academic Indicator to the November 2016 SBE meeting.
Two years of Data for

8. November SBE Meeting
The initial recommended performance standards for the Academic Indicator were based on the percent of students who scored “Standard Met” or “Standard Exceeded” on the Smarter Balanced Assessments for grades three through eight.
Grade eleven assessment results are captured in the College/Career Indicator.
Grades 3 - 8

9. November SBE Meeting (Cont.)
The SBE decided not to release the Academic Indicator using “Standard Met” or “Standard Exceeded” voicing concern that this closely paralleled the Adequate Yearly Progress (AYP) methodology, which resulted in many schools focusing only on those students who were closest to proficient.
Concerns about it felt too much like AYP and the bubble kids

10. November SBE Meeting (Cont.)
Rather, the SBE wants to encourage districts and schools to improve the academic achievement of all students in the new accountability system.
Therefore, the SBE requested the CDE to work on a methodology that uses scale scores.
Focus is on all students

11. Next Steps for the Academic Indicator

12. Academic Indicator
CDE staff will continue to work with the Technical Design Group (TDG) to explore using a “Distance from Met” methodology, which uses scale scores.
CDE staff will also meet with the testing vendor, Educational Testing Service (ETS), to ensure the validity of the “Distance from Met” methodology.
Distance from met is an indicator from Charter School conversations.

13. Academic Indicator (Cont.)
In January 2017, the CDE will bring the “Distance from Met” to the SBE as a possible option for the Academic Indicator.
Pending SBE’s decisions at this meeting, the CDE will move forward with “Distance from Met” or pursue other options.
Bringing Methodology and proposed cut scores in January.
Desire is to approve and not wait until March.
Board has a strong desire to make this work.
Distance Met look at each student how far a students scale score from lowest from met.
Average Scale Scores for the school the average student is 50 points above or below. Discussion for status and growth. Distance from Met. Model will be posted 10 days before January meeting.
Replace the status portion of the previous academic indicator.
Possible weighting of scores.
The model is based on distributions will set the cut scores.
Scores will be set for a certain number of years and then reset forward as scores improve
Performance Indicator Blue, Green, Red
You will continue to look at particular students.

14. An In-Depth Look into the State Indicators
The goal of today’s presentation is to cover the calculation method for each of the state indicators so that you can determine the performance category (or color) based on your own data.

15. Handouts
The 5 x 5 colored grid handouts include the cut scores specific for each indicator. These handouts are being provided as a resource and will not be covered in detail:
Handout 1: Graduation Indicator
Handout 2: Suspension Indicator
Handout 3: English Learner Indicator (ELI)
Handout 4: College/Career Indicator (CCI)
Handout 6: The Denominator for All Indicators

16. Before We Dive In…. LEA Data Charter Schools Alternative Schools
Because all charter schools are treated as LEAs under the LCFF, charter school data are not included in the LEA-level data.
Alternative Schools
Since a separate accountability system will be developed for alternative schools, all alternative schools (i.e., Alternative Schools Accountability Model [ASAM]) are also excluded from LEA-level data.
Important Slide Charter and Alternative

17. Before We Dive In… (Cont.)
Therefore, the LEA-level data for graduation and suspension rates will not match the data reported on the CDE’s DataQuest Web page.

18. Graduation Rate Indicator

19. Who Will Receive a Graduation Rate Indicator?
This indicator applies to LEAs, schools, and student groups that have 30 or more students in the four-year cohort graduation. See Handout 1.
School Level reporting is at 30 or more.
LEA level is 15

20. Status
For the initial release of the LCFF Evaluation Rubrics, the 2014–15 four-year cohort graduation rate, or the class of 2015 graduation data, will be used to determine Status.

21. Change
Prior three-years of four-year cohort graduation data were used to calculate the three-year weighted average to determine Change.
Note! This is the only state indicator that uses a three-year weighted average to calculate Change.

22. Three-Year Weighted Average Formula
Class of 2012 Graduates + Class of 2013 Graduates + Class of 2014 Graduates
divided by
Students in the Class of
Students in the Class of
Students in the Class of 2014

23. Three-Year Weighted Average Formula (Cont.)
It is important to note that if the LEA, school, or student group does not have cohort data for all prior three graduating classes, then the weighted average for Change was calculated using the one or two years of available cohort data.

24. Current Status (Class of 2015) minus Three-Year Weighted Average
Change Formula
Current Status (Class of 2015) minus Three-Year Weighted Average

25. Example 1 Topaz High School Status Change:
Class of 2015 Graduation Rate: 89.4%
Change:
Step 1: Obtain Cohort Graduation Data for the Prior Three Years
Step 2: Calculate the Weighted Average
Step 3: Calculate Change

26. Number of Students in the Cohort
Steps 1 & 2
Prior
Three-Year Data
Number of Students in the Cohort
Number of Graduates
Class of 2014
3,346
2,916
Class of 2013
3,343
2,857
Class of 2012
3,558
2,912
Sum
10,247
8,685
Weighted Average:
8,685 / 10,247 = 84.8%

27. Current Status (Class of 2015) minus Three-Year Weighted Average
Step 3
Step 3: Calculate Change
Current Status (Class of 2015) minus Three-Year Weighted Average
89.4% minus 84.8% = +4.6%

28. Determine Performance Category
Now that we have the Status and Change for Topaz High School, the school’s performance category (or color) can be determined.
Use the 5 x 5 color grid specific for the Graduation Rate Indicator
Reminder! Each indicator has its own unique 5 x 5 color grid(s).

29. Determine Performance Category (Cont.)
Status: 89.4%
Change: +4.6%

30. Future Update
The SBE has asked the CDE to work with the TDG to analyze and investigate the incorporation of the five-year cohort graduation rate into this indicator.
Based on the current SBE timeline, implementation of the four-and five-year cohort graduation rates may occur in 2018–19.
TDG =‘s Technical Design Group.

31. Suspension Rate Indicator

32. Who Will Receive a Suspension Rate Indicator?
LEAs, schools, and student groups that have 30 or more students enrolled will receive a Suspension Rate Indicator.
The enrollment data are obtained from the California Longitudinal Pupil Achievement Data System (CALPADS).

33. Suspension Rate Rules
If a student is suspended multiple times, the student is counted as being suspended only once.
LEA Example: If a student was suspended:
Five times at School A,
Twice at School B, and
Twice at School C
The student would be counted as being suspended once at the LEA.
Important Slide

34. Status and Change Status
The 2014–15 suspension rate will be used for Status for the initial release of the LCFF Evaluation Rubrics.
Change:
Change uses current and prior year suspension rates.
Change Formula:
2014–15 rate minus 2013–14 rate

35. Key Difference in Cut Scores
Data simulations revealed that suspension data varies widely among LEA and school type. Therefore, multiple suspension cut scores were set for LEAs and schools based on their type.
This resulted in six different sets of cut scores:
Three at LEA-level: Elementary, High, and Unified
Three at School-level: Elementary, Middle, and High

36. Key Difference in Cut Scores (Cont.)
Having six different sets of cut scores also means six separate 5 x 5 color grids based on district or school type. See Handout 2.

37. Impact of Multiple Cut Scores
Since cut scores were set separately for LEAs and schools, charter schools and single school districts could receive two performance categories (or two colors): (1) at the LEA-level and (2) at the school-level because:
Charter schools are treated as LEAs under the LCAP, and
Single school districts are treated as schools under the ESSA
LCAP= Local Control and Accountability Plan
ESSA= Every Student Succeeds Act

38. Impact of Multiple Cut Scores (Cont.)
Because the LEA and school performance categories (colors) could be inconsistent, the SBE approved holding charter schools and single school districts accountable for the suspension rate cut scores using the school-level cut scores.
Therefore, charter schools and single school districts will only be held accountable for their school-level performance category (color).
Due to potential inconsistencies, school-level indictor will be the indicator not the LEA indicator. See previous slides.

39. Key Difference In Goal
It is important to remember that for this indicator, the goal is reversed.
For all other state indicators, the desired outcome and goal is to achieve a high percent for Status and Change.

40. Key Difference in Goal (Cont.)
However, the desired outcome and goal for the Suspension Rate Indicator is to have a low suspension rate, which translates to a low percent for Status and Change.
For this reason, the Status and Change levels on the 5 x 5 color grids are in reverse order compared to the grids for the other indicators.

41. Key Difference in 5 x 5 Grid
Level
Increased Significantly
Increased
Maintained
Declined
Declined Significantly
Very Low
Yellow
Green
Blue
Low
Orange
Medium
High
Red
Very High

42. Assignment of Performance Category
Schools that did not certify (or submit) suspension data in the CALPADS are automatically assigned the Orange performance category.

43. Ruby Elementary School
Example 1
Ruby Elementary School
Status
2014–15 Suspension Rate: 0.5%
Change:
Step 1: Obtain prior year rate: 0.3%
Step 2: Calculate Change
Current Rate minus Prior Rate
0.5% minus 0.3% = +0.2%

44. Determine Performance Category
Now that we know the Status and Change for Ruby Elementary School, the school’s performance category (or color) can be determined.
Remember! Use the 5 x 5 color grid specific for elementary schools.

45. Determine Performance Category (Cont.)
Status: 0.5%
Change: +0.2%

46. Emerald Unified School District
Example 2
Emerald Unified School District
Status
2014–15 Suspension Rate: 3.8%
Change:
Step 1: Obtain prior year rate: 4.1%
Step 2: Calculate Change
Current Rate minus Prior Rate
3.8% minus 4.1% = -0.3%

47. Determine Performance Category
Determine Emerald Unified School District’s performance category (color) using the specific 5 x5 color grid for unified school districts.

48. Determine Performance Category (Cont.)
Status: 3.8%
Change: -0.3%

49. English Learner Indicator (ELI)

50. Who Will Receive an ELI?
This indicator applies to LEAs and schools that have 30 or more students who took an annual California English Language Development Test (CELDT).
Note: Because 86.2% of schools have no significant, or only one significant race/ethnic student group within the EL group, student group data will not be reported for the ELI.

51. ELI Data Sources
The ELI determines progress through the use of two data sources:
Annual CELDT results, and
EL reclassification

52. CELDT Data The CELDT has five overall performance levels: Beginning
Early Intermediate
Intermediate
Early Advanced
Advanced

53. CELDT Data (Cont.)
Because the CELDT Intermediate performance level has a large range of scale scores, many students stay in the intermediate level for multiple years. As a result, stakeholders advised, and the SBE approved, that this level be divided into two, for accountability purposes only, to recognize the substantial growth that can be made within this particular level.

54. CELDT Data
Therefore, the ELI uses six overall CELDT performance levels:
Beginning
Early Intermediate
Low Intermediate
High Intermediate
Early Advanced
Advanced
Six Performance Levels

55. ELI Model
Annual CELDT takers must advance at least one CELDT performance level from prior year to current year to be included in the numerator of the ELI calculation.
Examples:
Prior Overall CELDT Performance
Current Overall CELDT Performance
Does the LEA or School Receive Credit?
Early Intermediate
Low Intermediate
Yes
High Intermediate
Advanced
Beginning No

56. ELI Model (Cont.)
Students who scored Early Advanced or Advanced Proficient in the prior year and maintained that performance level for the current year will be included in the numerator for the ELI calculation. This is the only set of annual CELDT test takers who are not required to advance one CELDT performance level.
ELs who were reclassified in the prior year will also be included in the numerator and denominator for the ELI calculation.

57. Students Reclassified in Prior Year
ELI Model (Cont.)
Previous CELDT
Overall Level
Current CELDT
Beginning
Early Intermediate
Low Intermediate
High Intermediate
Early Advanced
Early Advanced or
Adv Not Proficient
Adv Proficient +
Students Reclassified in Prior Year

58. ELI Formula: Numerator
Annual CELDT test takers who:
Increased at least one CELDT level compared to the prior year
Maintained Early Advanced/Advanced English Proficient
ELs who were reclassified in the prior year

59. ELI Formula: Denominator
Total number of annual CELDT test takers
ELs who were reclassified in the prior year

60. Status and Change Status
The percent of ELs who moved up a performance level from the 2014 to 2015 CELDT plus ELs who were reclassified between July 1, 2013 to June 30, 2014.
Change
Difference in Status from current year to prior year.
Data Quest Calculations do not work for DQ includes charter and alternative schools.
Data Quest Calculatons will be changed in include or exclude charter.

61. Assignment of Performance Category
Schools that did not test at least 50 percent of their EL population in the CELDT are automatically assigned an Orange performance category.
Determination of the 50 percent is based on the EL demographic data reported for mathematics in the Smarter Balanced Assessment file from the testing vendor.

62. Example 1 Gemstone High School Status 210/250 = 84% 20 students
Step 1: Percent of annual CELDT test takers who advanced at least one performance level (or maintained Early Advanced/Advanced English Proficient) on the 2015 overall CELDT compared to the 2014 overall CELDT
210/250 = 84%
Step 2: Number of ELs who were reclassified in prior year (2013–14)
20 students
Step 3: Add reclassified students to the numerator and denominator of Step 1 and calculate the rate.
230/270 = 85%

63. Current Status minus Prior Status
Example 1 (Cont.)
Change:
Step 1: Calculate Prior Year Status: 83%
Step 2: Calculate Change
Calculate Change:
Current Status minus Prior Status
85% minus 83% = +2%

64. Determine Performance Category
Use the specific 5 x 5 color grid for the ELI and determine the performance category (color) for the EL student group.

65. Determine Performance Category (Cont.)
Status: 85%
Change: +2%

66. An In-Depth Look into the College/Career Indicator

67. What is the Purpose of the CCI?
The College/Career Indicator (CCI) was designed to emphasize that a high school diploma should represent completion of rigorous course work that prepared students for postsecondary.

68. CCI Measures
The CCI Model currently contains the following measures (see Handout 5):
Advanced Placement (AP) exam results
Dual Enrollment
Early Assessment Program (EAP) results for English language arts/literacy (ELA) and mathematics (Grade 11)
a-g completion
Career Technical Education (CTE) pathway completion

69. CCI Measures (Cont.) Potential Measures for the release in 2017–18:
International Baccalaureate (IB)
State Seal of Biliteracy
Golden State Seal Merit Diploma
IB Career–related Programme
Articulated CTE Pathway

70. Students with Disabilities
Students with the most severe cognitive disabilities (i.e., students who take the California Alternate Assessment) are removed from the calculation of the CCI.

71. First a Local Indicator
Although the CCI was planned to be reported as a state indicator, the SBE approved that:
The CCI will be reported as a local indicator for the initial release of the rubrics based on Status only. (The CDE will pre-populate the CCI data in the rubrics.)

72. Then a State Indicator
The first time grade eleven Smarter Balanced Assessment results will be available to calculate both Status and Change for the CCI is in 2018–19 (class of 2016 and 2017).
As a result, when the LCFF Evaluation Rubrics are released in fall 2017, the CCI will be reported as a state indicator based on Status only.

73. CCI Formula
Students Who Receive a High School Diploma and Meet the CCI
Benchmark of “Prepared”
divided by
Current Year Graduation Cohort

74. Example 1
By the time Minnie Hause received her high school diploma, she had completed the following during her four years in high school:
Completed a-g
Scored “Standard Met” in ELA and “Standard Nearly Met” in mathematics on the grade eleven Smarter Balanced Summative Assessments
Completed one CTE pathway in marketing

75. Example 1 (Cont.)
Using the CCI Model (table version or graphic format in Handout 5), and based on Minnie’s information on the prior slide, the highest measure that she meets is completion of a-g and CTE pathway within the CCI “Prepared” performance level.
As a result, Minnie would be included in the numerator of the Status calculation of the CCI.

76. Status Only
For the initial release of the CCI (as a local indicator), Status will be calculated based on the class of The following years of data will be used for the CCI measures:
CCI Measure
Data Source
Year of Data
AP exam
The College Board
2014
Grade 11 EAP results
STAR file
From Educational Testing Service (ETS)
2013
a-g completion
CALPADS
(Yes/No filled in by LEA )
CTE pathway completion
Last 3 yrs in high school
(2012, 2013, 2014)
Dual enrollment

77. In-Depth Look at Performance Categories

78. GREEN is the Target
The target performance category (or color) is GREEN for all LEAs, schools, and student groups.
Performance categories (or colors) tell more than just the current year’s status; they reflect information about status and change. In other words, if your school is improving, that’s already reflected in the performance category.

79. The Performance Categories Already Reflect Change…..
For some indicators, earning RED one year and ORANGE the next year may not reflect improvement! (example to follow)

80. Using the 5 x 5 Grid for the Graduation Rate Indicator…

81. Moving From RED to ORANGE Does Not Always Mean Improvement!
Example: An LEA has a 2015–16 graduation rate of 82.0%. The LEA’s prior three-year average graduation rate is 89.0% (Change = -7%). As a result, the performance category is:
The 2016–17 graduation rate is 79.0%. The updated prior 3-year average graduation rate is 83.0% (Change = -4). As a result the performance category is:
RED
ORANGE

82. The Reverse is also True
Example: An LEA has a 2015–16 graduation rate of 92.0%. The district’s prior 3-year average graduation rate is 85.0% (Change = +7%). As a result, the performance category is:
The 2016–17 graduation rate is 94.0%. The updated prior 3-year average graduation rate is 91.0% (Change = +3). As a result, the performance category is:
BLUE
GREEN

83. Takeaways From These Examples
Performance category already reflects change, so:
Talking about how a performance category changes over time may not make sense.
Discussions about trends should focus on trends in the underlying data.
A red, orange, or yellow performance category means that there is more work to be done. A green or blue performance category means that the trajectory of performance is fine.

84. Additional Information

85. Small LEAs, Schools, and Student Groups
Small LEAs and schools with a denominator of less than 30 students for many of the indicators will not have performance categories reported. See Handout 6.
However, the LCFF Evaluation Rubrics will display Status and Change data for numbers between 11 to 29. Although this will not translate to a color, stakeholders will be able to view the Status and Change data.

86. Small LEAs, Schools, and Student Groups (Cont.)
Not receiving a performance category (or color) means that the LEA/school will not be included in the eligibility list for support or interventions.
Data for less than 11 students will not be reported due to privacy reasons.

87. Demographic Corrections
For the initial release of the LCFF Evaluation Rubrics, the opportunity to make demographic corrections for the state indicators and the CCI has passed.
For the release of the LCFF Evaluation Rubrics in fall 2017, the demographic correction process will vary for each state indicator.

88. Demographic Corrections (Cont.)
Graduation Rate Indicator
The demographic and program participation (i.e., EL, Students with Disabilities [SWDs], and Socioeconomically Disadvantaged [SED]) data are based on data extracted from the CALPADS Operational Data Store (ODS).
CALPADS LEA Administrators and Accountability Coordinators will be notified of the (Fall 1) preview period and the correction window timeline.

89. Demographic Corrections (Cont.)
Graduation Rate Indicator (Cont.)
The CALPADS Calendar also identifies the submission, certification, and correction windows for the cohort graduation data:

90. Demographic Corrections (Cont.)
Suspension Rate Indicator
The demographic and program participation data for suspension (or discipline) are based on the End-of-Year 3 submissions.
LEAs and schools should view the CALPADS Calendar for certification deadlines and the correction window for discipline data.

91. Demographic Corrections (Cont.)
English Learner Indicator
CALPADS LEA Administrators and Accountability Coordinators will be notified of the correction window for the CELDT data.
The reclassified data will be extracted from the CALPADS ODS mid-to-late June of each year. LEAs and schools should have their data corrected prior to the extraction as there will be no opportunities to correct data after the extraction date.

92. Demographic Corrections (Cont.)
College/Career Indicator
Because the CCI is based on the four-year graduation cohort, the demographic and program participation data used for this indicator is tied to the four-year cohort graduation data.
As a result, LEAs and schools should review their graduation data during the (Fall 1) preview process and make appropriate corrections since the cohort data will be used for two indicators.

93. Questions?

94. Questions for RAN Members
Is it appropriate that alternative schools’ results are not included in the LEA-level data?
A student who is suspended multiple times at the same school (or same district) is only counted once in the numerator of the suspension rate. Should the numerator capture the number of times the student is actually suspended?