Accountability Work Group

Accountability Work Group
ADP March 1, 2018

Agenda Opening Grounding Agenda Purpose and Norms Updates
State School/District Performance Framework Input Disaggregated Groups Growth to Standard FEP Students, PSAT, Disaggregation for Dropout and SAT Lunch Confirming Reporting Priorities ESSA Updates and Input School Identification Criteria and Simulated Impact Communication Needs Wrap up ADP

Opening Grounding ADP

Thank you! ADP

Introductions AWG Members Audience Members CDE Staff ADP

Purpose and Goals of the Accountability Work Group
To gather input on the accountability decision points for the state’s School and District Performance Frameworks, as well as on the methods and criteria for identifying schools for comprehensive or targeted support and improvement under the Every Student Succeeds Act (ESSA). The work is focused on school and district accountability, which is tied closely to and dependent upon state assessments. However, assessment options will not be the focus of this work. ADP

11/13/2018 Group Norms Participate – be present and contribute Control your sidebars and technology Self-monitor your balance between listening and speaking Help to ensure that all voices are heard Find the common ground Respect for people and ideas- we all have different perspectives that are crucial for this work Speak your mind – this is a safe space Shared talking points at the end of the meeting Be a problem-solver Identify both issues/concerns and possible solutions Provide feedback on the process and product There may be homework that we need you to do ADP

State and Federal Systems 2017 Reporting and Support Cycle
Updates ESSA Plan Feedback Participation calculations Use of graduation rates State and Federal Systems 2017 Reporting and Support Cycle SPFs and DPFs ESSA School Identifications EASI Application State Board of Education Accountability Bill ADP High level mention of ESSA feedback that’s led to split systems--- saving details for Nazie later

State School/District Performance Framework Input

Purpose Statement | Colorado State Accountability System
The purpose of Colorado’s state school and district accountability system is to provide valid and actionable information regarding the progress of all students in meeting the state standards. This information should support districts, schools and stakeholders in evaluating and improving the effectiveness of their systems, with the goal that each and every student, regardless of background or learning need, meets state expectations for academic achievement and growth and is prepared for post-secondary and workforce success. ADP Looks good here- we can give the background of how this came to be- we use this as a guiding statement for recommendations.

2018 Performance Framework Decisions: Reporting on Race/Ethnicity
Ashley Piche, Josh Perdue, Marie Huchton CDE Accountability & Data Analysis Unit

Overview of Today’s Discussion
Under the current performance frameworks, all race/ethnicity groups other than “White” are reported under the heading of “Minority Students” Ratings associated with the “Minority Students” disaggregated group determine a portion of the points earned on the Achievement, Growth, and PWR indicators. For the 2018 frameworks, race/ethnicity results could be reported in alignment with the ESSA plan I adjusted the language for Rating could be based on the results of the individual race/ethnicity groups (and aggregated non-white group) consistent with Colorado’s plan for federal reporting under ESSA. Under the current performance frameworks, all race/ethnicity groups other than “White” are reported under the heading of “Minority Students” Ratings associated with the “Minority Students” subgroup determine a portion of the points earned on the Achievement, Growth, and PWR indicators. For the 2018 frameworks, it has been proposed that each race/ethnicity group be included separately Using individual race/ethnicity groups for reporting and for ratings determinations would be consistent with Colorado’s plan for federal reporting under ESSA. Under the ESSA plan, students who belong to race/ethnicity groups where schools do not meet the minimum n-count would be reported under a new category, tentatively labeled “Aggregated non-White Students”.

Under the current performance frameworks, all race/ethnicity groups other than “White” are reported under the heading of “Minority Students” Ratings associated with the “Minority Students” disaggregated group determine a portion of the points earned on the Achievement, Growth, and PWR indicators.

For the 2018 frameworks, race/ethnicity results could be reported in alignment with the ESSA plan Rating could be based on the results of the individual race/ethnicity groups (and aggregated non-white group) consistent with Colorado’s plan for federal reporting under ESSA. Under the current performance frameworks, all race/ethnicity groups other than “White” are reported under the heading of “Minority Students” Ratings associated with the “Minority Students” subgroup determine a portion of the points earned on the Achievement, Growth, and PWR indicators. For the 2018 frameworks, it has been proposed that each race/ethnicity group be included separately Using individual race/ethnicity groups for reporting and for ratings determinations would be consistent with Colorado’s plan for federal reporting under ESSA. Under the ESSA plan, students who belong to race/ethnicity groups where schools do not meet the minimum n-count would be reported under a new category, tentatively labeled “Aggregated non-White Students”.

For the 2018 frameworks, race/ethnicity results could be reported in alignment with the ESSA plan For ESSA, students who belong to race/ethnicity groups where schools do not meet the minimum n-count are reported under a new category, tentatively labeled “Aggregated non-White Students.” For ESSA, all groups count towards the school rating; and individual identifications as well. Under the current performance frameworks, all race/ethnicity groups other than “White” are reported under the heading of “Minority Students” Ratings associated with the “Minority Students” subgroup determine a portion of the points earned on the Achievement, Growth, and PWR indicators. For the 2018 frameworks, it has been proposed that each race/ethnicity group be included separately Using individual race/ethnicity groups for reporting and for ratings determinations would be consistent with Colorado’s plan for federal reporting under ESSA. Under the ESSA plan, students who belong to race/ethnicity groups where schools do not meet the minimum n-count would be reported under a new category, tentatively labeled “Aggregated non-White Students”.

Impact of Including Individual Race/Ethnicity Categories
Given the minimum n-count requirements, the shift to including individual race/ethnic groups on performance frameworks would have a significant impact on how non-White students are represented in the frameworks, compared to the current method. These impacts center around two major issues: With minimum n-count requirements, non-White students as a whole would see their representation in the disaggregated race/ethnicity reporting reduced when each race/ethnicity group are reported separately - this is true even with the introduction of a new category of “Aggregated non-White Students.” All students will still be included in the “All” category. I added the box on the side of this slide. I think we really need to make sure it’s clear that this is only for the race/ethnicity disagg reporting. The new “Aggregated non-White Students” category would, by definition, lack consistent meaning, so would pose challenges to interpreting results.

Minimum N-Counts | Reduced Representation of non-White Students
Consider the following example of a fictional school - Mountain Elementary: Mountain Elementary has achievement scores for 20 Hispanic students, 12 Black students and 3 Native American students. The school has no scores for students in any of the other non-White race/ethnicity groups. Can you all make sure to make a caveat about the fact that this is just an example- and not one that occurs all that often (I don’t think) Hispanic Students Black Students Native American Students N = 20 N = 12 N = 3

On the performance frameworks, the minimum N-size for reporting achievement results is 16. Under the proposed disaggregation of race/ethnicity groups, Hispanic students (N=20) would be reported as an individual race/ethnicity disaggregated group on Mountain Elementary’s school performance framework. Black students (N=12) and Native American students (N=3) will not meet minimum N-size requirements and would not be reported as an individual disaggregated group. Hispanic Students Black Students Native American Students N = 20 N = 12 N = 3

For ESSA, all race/ethnicity disaggregated groups that are not reported individually are aggregated into the tentatively named “Non-White Aggregated Group” At Mountain Elementary, the “Non-White Aggregated Group” would consist of the 12 Black students and 3 Native American students that cannot be reported as individual disaggregated groups. Hispanic Students Black Students Native American Students N = 20 N = 12 N = 3

The total N-count for the “Non-White Aggregated Group” at Mountain Elementary is 15 which is under the minimum requirement for reporting The “Non-White Aggregated Group” would therefore not be represented on the school’s framework. Hispanic Students Black Students Native American Students N = 20 N = 12 N = 3

Mountain Elementary Example Current State Framework Proposed Individual Race/Ethnicity Categories

Takeaways from the Mountain Elementary example: Under existing reporting practice, the 15 scores for black and Native American students would be included in the “Minority Students” category along with the scores for Hispanic students, so that results for all 35 students would be represented in the race/ethnicity disaggregated group. With the proposed changes, only 20 of the 35 total scores for non-White students would be reported in the race/ethnicity disaggregated group. The impacts would not be evenly distributed, with smaller race/ethnicity groups seeing much larger proportional reductions in the race/ethnicity disaggregated group reporting than the state’s largest non-White group, Hispanics. While the number of students impacted at individual schools might seem small, the following analysis shows that the cumulative impacts across the state would be quite large.

Analysis | 2017 Performance Framework Results
On the 2017 School Performance Frameworks, the English Language Arts achievement metrics reported under the “All Students” group represented 99.5% of all eligible results.* The results reported under the “Minority Students” disaggregated group on the frameworks represented 98.3% of eligible achievement outcomes for non- White students. The representation of non-White students within the “Minority Students” disaggregated group was on par with the representation that White students would have had if they had been reported as a distinct disaggregated group. *Note: For the School Performance Frameworks, eligible records are those where students had valid scores and met requirements around continuous enrollment, expelled status, and home-school status. The exclusion of 0.5% of eligible records reflects the minimum n-count requirements, which means that results are not reported in cases where there are fewer than 16 records available at a given grade level within a school.

Analysis | 2017 Frameworks Reported with Disaggregated Race/Ethnicity Groups
Shifting to a system where individual race/ethnicity groups are included as separate disaggregated groups would result in reduced representation for non-White students on the School Performance Frameworks This reduction in the representation of non-White students would occur even with the introduction of a new “Aggregated non-White” disaggregated group.

Analysis | 2017 Frameworks Reported with Disaggregated Race/Ethnicity Groups
Whereas 98.3% of English Language Arts achievement outcomes for non-white students were reported under the “Minority Students” subgroup on the 2017 frameworks, only 94.0% would have been included had the state adhered to the method that has been proposed for ESSA reporting. The number of non-White students whose results would be excluded from the frameworks would increase by 9,244: from 3,753 under the current system to 12,997 under the proposed new system. This reduction in the representation of non-White students would occur even with the introduction of a new “Aggregated non-White Subgroup”.

Analysis | Reduced Representation of non-White Students
In comparison, under the current method where non-White students are reported together under the “Minority Students” disaggregated group, no individual race/ethnicity category has fewer than 95.0% of results included in accountability determinations. While Hispanic students would have roughly equivalent representation under either system, all other race/ethnicity groups – particularly the smallest groups – would see their representation reduced significantly.

Analysis | Reduced Representation of non-White Students in School-Level Reporting
The slides presented to this point have been aimed at illustrating the cumulative impact of the proposed changes on the representation of non-white students across the state. The next series of slides looks at the proportion of schools with Achievement results for non-white students that would meet the requirements for having these results included in the performance frameworks. It is also important to understand how these changes impact the information that is available to evaluate the performance of individual schools.

For the 2017 performance frameworks, there were 1,086 schools with elementary school level Achievement results. 96% of schools saw these results reported under the “All Students” category. 1,070 schools had elementary school level results for non-White Students. 89% of schools saw results for non-White students reported under the “Minority Students” category. For comparison, there were 1,079 schools with elementary school level results specifically for White Students. If “White Students” had been a separate reporting category, 87% of schools would have met the n-count requirements to have these results reported.

With the proposed changes, very few schools would have enough results for individual race/ethnicity groups to be included in the performance frameworks, except for the Hispanic group. Under the proposed reporting scheme, only 35% of elementary schools would see all of their results for non-White students represented in the frameworks, in contrast to 88.6% when using the “Minority Students” category.

“Aggregated non-White Students” | Inconsistent Meaning
The composition of the group is contingent by definition, so that the category would have no fixed meaning outside of the context of a particular subsection of a report for a specific school or district. Variation based on level of aggregation Because the composition of the group is determined by n-count, the individual race/ethnicity groups included within it may change depending on the level of aggregation. Variation within individual frameworks Because minimum n-count differ for performance indicators (i.e. 16 for achievement & 20 for growth), the composition of the group may differ from one indicator to another. Variation over time Even slight changes in a district or school’s demographics from one year to the next can impact which race/ethnicity groups get included in the group each year, which may impact historical comparisons. In addition to concerns about how the proposed change will impact the representation of non-White students in the performance frameworks, the Accountability Unit is concerned about how audiences will make sense of the “Aggregated non-White Students” group.

“Aggregated non-White Students” | Inconsistent Meaning
To get a sense of the difficulties that would occur in trying to contextualize and make meaning out of this new category, consider the differences that would exist in the the composition of the “Aggregated non-White Students” group under the Achievement indicator versus the composition that would exist under the Growth indicator. Because the set of growth results is smaller, and because the n-count for growth results is higher, fewer schools would have results reported for individual race/ethnic groups, and more results would be grouped together under the “Aggregated non-White Student” label. The table below illustrates how these differences would play out across the state in the School Performance Frameworks. For many schools, the Achievement results reported for individual race/ethnicity groups would have no corresponding data point in the Growth results. Moreover, the “Aggregated non-White Student” group reported under the Achievement indicator would not be directly comparable to the “Aggregated non-White Student” group appearing under the Growth indicator. This same type of mismatch would occur with results reported under the PWR indicator, and similar complications would arise in trying to make comparisons between grade levels, between schools and districts, or between the same school from one year to the next.

Proposed Options | OPTION 1
Option 1: Continue reporting Minority group and assign points contributing to overall school rating. Under the current performance frameworks, all race/ethnicity groups other than “White” are reported under the heading of “Minority Students” Ratings associated with the “Minority Students” disaggregated group determine a portion of the points earned on the Achievement, Growth, and PWR indicators.

Option 2: Remove Minority group. Report individual race/ethnicity categories and Aggregated non-White group and assign points contributing to overall school rating. Using individual race/ethnicity groups for reporting and for ratings determinations would be consistent with Colorado’s plan for federal reporting under ESSA. Under the current performance frameworks, all race/ethnicity groups other than “White” are reported under the heading of “Minority Students” Ratings associated with the “Minority Students” subgroup determine a portion of the points earned on the Achievement, Growth, and PWR indicators. For the 2018 frameworks, it has been proposed that each race/ethnicity group be included separately Using individual race/ethnicity groups for reporting and for ratings determinations would be consistent with Colorado’s plan for federal reporting under ESSA. Under the ESSA plan, students who belong to race/ethnicity groups where schools do not meet the minimum n-count would be reported under a new category, tentatively labeled “Aggregated non-White Students”.

Option 3: Continue reporting Minority and assign points contributing to overall school rating. Additionally, report individual race/ethnicity categories, but do not assign points or report Aggregated Non-White. Under the current performance frameworks, all race/ethnicity groups other than “White” are reported under the heading of “Minority Students” Ratings associated with the “Minority Students” subgroup determine a portion of the points earned on the Achievement, Growth, and PWR indicators. For the 2018 frameworks, it has been proposed that each race/ethnicity group be included separately Using individual race/ethnicity groups for reporting and for ratings determinations would be consistent with Colorado’s plan for federal reporting under ESSA. Under the ESSA plan, students who belong to race/ethnicity groups where schools do not meet the minimum n-count would be reported under a new category, tentatively labeled “Aggregated non-White Students”.

AWG Recommendation Option 1 Continue reporting Minority group and assign points contributing to overall school rating. Option 2 Remove Minority. Report individual race/ethnicity categories and Aggregated non-White group and assign points contributing to overall school rating. Option 3 Continue reporting Minority and assign points contributing to overall school rating. Additionally, report individual race/ethnicity categories, but do not assign points or report Aggregated Non-White. Do you want to add, or mention while you talk about an “option 4”- if someone has another idea. (I think someone may suggest pulling Asian out of Minority…)

2019 State Frameworks: Growth to Standard
Marie Huchton, Unit of Accountability and Data Analysis

Creating Growth-to-Standard Measures
In addition to knowing how much growth a student has made in comparison to her peers, it is also useful to know if the student is on-track to attaining a particular proficiency standard within a set time- frame, for example the CMAS Meets Expectations or WIDA ACCESS English proficient cut-points. Due to the recent assessment transitions for both CMAS and WIDA ACCESS, we have the opportunity to reframe and recalculate these growth-to-standard measures for the state performance frameworks. To meet ESSA submission requirements, work has already been done to create this new “on-track” growth-to-standard measure for English learners on the WIDA ACCESS.

Creating Growth-to-Standard Measures
The 2018 ESSA identification calculations will be including this English language proficiency (WIDA ACCESS) on-track measure. CDE is planning to publish student-level WIDA ACCESS growth-to-standard results this summer and potentially include in the 2018 SPF/DPF report for informational purposes (i.e. not for points). We are also working to develop new CMAS ELA/Math and PSAT/SAT EBRW/Math growth-to-standard measures in the coming months and hope to release results for informational purposes either within or alongside the performance frameworks (again, not for points). For the 2019 performance frameworks, we are hoping to include both the WIDA ACCESS and CMAS/PSAT/SAT growth-to-standard metrics for points in the rating calculations.

Today’s Presentation We will review the work that has been done to create the English Language Proficiency (ELP) growth-to- standard metric, breaking down the required component steps and the rationales behind analytic decisions that have been made. CDE is planning to engage in a similar multi-step building process for the CMAS ELA/Math and PSAT/SAT EBRW/Math growth-to-standard measures. Future AWG meetings will likely dig deeper into these new CMAS/PSAT/SAT measures, but we wanted start with the work done this fall around WIDA ACCESS to ground the conversation.

Establishing English Language Proficiency

Colorado’s English Language Proficiency Assessment
WIDA ACCESS for ELLs transitioned from paper-and- pencil to a new online assessment referred to as ACCESS 2.0 for spring 2017 testing. WIDA set new proficiency levels that reflect the increased rigor of academic language requirements in the new College and Career Ready standards and assessments Graphic from WIDA presentation “2017 Score Changes Based on Standard Setting”, SEA Webinar February 23, 2017.

Colorado’s English Language Proficiency Assessment
To set their new proficiency levels, WIDA identified “the point where WIDA English language proficiency (ELP) levels (1.0 x to 6.0) display similar distributions to English only (EO) students, or ELP levels are above the proficient cutscore, given EOs are on average above the cutscore.” (Cook, 2016) The underlying scale did not change and they referred to shifting the cuts as “stretching the rubber band” Previous Cuts 350 475 600 225 100 1.0 2.0 3.0 4.0 5.0 6.0 New 2017 Cuts 350 475 600 225 100 1.0 2.0 3.0 4.0 5.0 6.0

Redesignation Eligibility Criteria- 2017
CLDE stakeholders directed CDE to match as closely as possible the previous counts of elementary and high school students eligible for redesignation. Decided upon updated 4.0 Overall and 4.0 Literacy requirement on revised ACCESS 2.0 proficiency levels Estimated counts of students that would be eligible for redesignation : Count % of Current EL population Change from Current Count All Grades 23635 22.6% 3,317 K-5 13727 21.2% -227 6-8 4678 22.7% 3,557 9-12 5230 27.2% -13

Timeline for Attaining English Language Proficiency

Only two consecutive school years of monitoring.
Progression of English Learners from Identification to Full English Proficiency 5 – 7 YEARS 2 YEARS Identification Instruction Re-designation Monitoring Exiting Incoming student with home language other than English is assessed to determine whether he/she is an English Learner. Student is supported in their English language development along with access to grade level academic standards. Student demonstrates they are Fluent English Proficient (FEP) and can transition successfully to classrooms, with minimal English language development support. Students must still receive classroom differentiated instruction and assessment, if needed. Only two consecutive school years of monitoring. Students who have formally exited no longer need formal English language development programming. Redesignation is… A process that English learners undergo when they attain established criteria that demonstrates they are Fluent English Proficient (FEP) and can transition successfully to classrooms, with minimal English language development support. CDE has approved the following pathways to redesignation for Pathway 1 – Access 2.0 Assessment Data to Initiate Redesignation Process 4.0 Literacy and 4.0 Overall composite scores Pathway 2 – Access 2.0 Assessment Data to Initiate Redesignation Process 4.0 Literacy and <4.0 Overall composite or no overall composite score reported Pathway 3 – Local Data to Initiate Alternate Redesignation Process Evidence aligned to CAS to show grade level proficiency in reading and writing Monitoring is… Only two consecutive school years of monitoring Must still receive differentiated instruction and classroom assessment, if needed At the end of each monitor year, student progress should be evaluated, using district determined criteria Must be monitored to ensure progress toward exit statusAfter two consecutive school years of monitoring: Exit 1 Status Reenter as LEP or NEP proficiency level and re-enter ELD program Exiting is… Upon completion of two consecutive school years of monitoring, a student is eligible to exit from an English language development program Students who have formally exited no longer need formal English language development programming

Previous ACCESS 1.0 Timeline to Proficiency
Percent of Colorado Students Increasing 1 or More Proficiency Levels on 2013 ACCESS 1.0 in 1 year: All Grades Combined When starting at proficiency levels 1, 2, or 3, a student had a better than 50% chance of increasing at least one proficiency level in one year, which seemed like a reasonable expectation When starting at level 4 a student had less than a 50% chance of increasing at least one proficiency level in one year, implying that 2 years to get to level 5 was a more realistic trajectory

Previous ACCESS 1.0 Stepping-Stone Timeline
1 year 1 year to hit Literacy 5 2 years

ACCESS 2.0 The transition to WIDA ACCESS 2.0 has made it difficult to track changes in student proficiency over time. Only the two most recent years of data can be directly compared 2016 re-standardized domain and composite scale scores with online/paper mode adjustment 2017 domain and composite scale scores with online/paper mode adjustment

ACCESS 2.0 Overall Proficiency Level Increases in 1 Year
Highlighted above, are the most likely outcomes for students starting at the indicated Overall proficiency level after 1 year Level 1 students move to Level 2 Level 2 students move to Level 3 Level 3 students stay at Level 3 Level 4 students stay at Level 4

New ACCESS 2.0 Proficiency Level Increases in 1 Year
The column to the far right shows the likelihood of a student at this starting Overall proficiency level in 2016 increasing 1 or more levels in 1-year. Notice that only students starting at Level 1 and Level 2 have above a 50% or greater chance of moving up. Students at Level 3 have only a 27.8% chance of moving up, indicating that two or more years would likely be required to move to Level 4

ACCESS 2.0 Student Progress 2016 to 2017
Level 1 Level 2 Level 3 Level 4 Level 5 5.3%* 10.7%* 27.8%* 54.5%* 77.3%* * Percent of students increasing 1 proficiency level on ACCESS 2.0 from 2016 to 2017 62% of students scoring at Level 1 Overall in 2016 are enrolled in Kindergarten and have a greater likelihood of increasing 1 or more levels within 1 year than middle or high-schoolers. Elementary students are more likely to move up 1+ levels in one year regardless of starting proficiency level than students at middle or high school. Level 1 to 2+ Level 2 to 3+ Level 3 to 4+ Level 4 to 5+ Level 5 to 6 Elementary 81.6% 59.3% 33.9% 16.7% 8.6% Middle 56.2% 37.9% 19.9% 5.2% 0.4% High 53.5% 45.8% 19.1% 5.4% 1.2%

Proposed Timeline Expectations
Level 1 Level 2 Level 3 Level 4 Level 5 5.3%* 10.7%* 27.8%* 54.5%* 77.3%* 3 years 2 years 1 year 62% of students scoring at Level 1 Overall in 2016 are enrolled in Kindergarten and have a greater likelihood of increasing 1 or more levels within 1 year than middle or high-schoolers. Elementary students are more likely to move up 1+ levels in one year regardless of starting proficiency level than students at middle or high school. * Percent of students increasing 1 proficiency level on ACCESS 2.0 from 2016 to 2017 On average across all grades, a 6-year timeline was recommended for a student to move from non-English proficient newcomer to re-designated fluent-English proficient. Students entering at higher levels of proficiency will have fewer years to achieve re-designation.

Proposed Timeline for Progression
On average across all grades, a 6-year timeline was recommended for a student to move from non-English proficient newcomer to re-designated fluent-English proficient. The 6-year timeline is broken down into the following proficiency stepping-stone targets: Students entering at higher levels of proficiency will have fewer years to achieve re-designation. Remind again that these are preliminary timeframes and further discussion is being planned for the spring

Progress in Attaining English Language Proficiency

Progress in Attaining English Language Proficiency
Growth To Standard (On-Track Measure) Students making enough progress to attain English language proficiency in the designated time period Measure of “enough growth” for students to reach language proficiency (also known as adequate growth) Growth needed to reach next level of language proficiency is compared to actual growth percentile Adequate growth was included in the performance frameworks in Was previously part of federal Title III accountability under NCLB and the Title I ESEA Waiver ESSA requires goals and targets for students making progress Level 1 Level 2 Level 3 Level 4 Level 5 The board can define the determination of “adequate” growth, our definition of progress in attaining ELP.

Percent On-Track: All Grades
Level 1 Level 2 Level 3 Level 4 Level 5 39.4%* 68.3%* 45.6%* 66.6%* 76.9%* 3 years 2 years 1 year * Percent of students on-track to reach proficiency goals within allotted timeframes Students initially at Levels 1, 2 and 3 are expected to increase 1 proficiency level in 1, 2, or 3-years. Students initially at Levels 4 and 5 are expected to maintain their current proficiency level or increase one or more levels (as possible) within 1-year

Percent On-Track: By Grade
Pretty clear delineation between g1-5 and g6-12

School Distributions of Percent On-Track by EMH
Removing Schools with N<20 and AECs Much higher proportions of elementary students on-track to make ELP targets. Plan to norm by EMH level

ELP On-Track Sub-Indicator Ratings
Preliminary 2017 Does Not Meet, Approaching, Meets, Exceeds targets set using school distributions by individual EMH levels at the 15th-50th-85th percentiles. Plan to re-norm after reviewing 2018 WIDA ACCESS 2.0 data and ensuring growth results are usable for accountability reporting. For 2018, anticipate only ESSA identification calculations will include ELP on-track measure.

Extending On-Track Methodology to CMAS
We want to get your informal opinion on whether the process described for creating the ELP AGP On-track metric seems reasonable for moving forward building the CMAS ELA and Math and PSAT/SAT EBRW and Math metrics as well.

2018 Performance Framework Decision Items
Marie Huchton, Unit of Accountability and Data Analysis

How should PSAT 9 and PSAT 10 data be combined for 3-year reporting?

Background Colorado started testing 10th graders on the PSAT in spring of For the 2018 performance frameworks, we will have PSAT 10 results for the years , , and (3 years total) For the first time this march, Colorado will be testing grade 9 students with the PSAT. For the 2018 performance frameworks, we will have PSAT 9 results for (1 year total)

Decision Item for Today
As we create the 3-year version of the performance framework, how many years of PSAT 9 and PSAT 10 should we include? Proposal #1- 1 year PSAT 9 and 3 years PSAT 10 Proposal #2- 1 year PSAT 9 and 1 year PSAT 10

Proposal 1: 1 year PSAT 9 and 3 years PSAT 10
Pros Uses all available assessment data Consistent with what we did for 2017 reporting Cons Inconsistent grade representation Inconsistent with other metrics on 3-year SPF/DPFs Note: Also needs to be decided for ESSA reporting

Proposal 2: 1 year PSAT 9 and 1 year PSAT 10
Pros Same amount of data for both grades Would remove potential impact of 2017 PSAT participation issues Cons Inconsistent with other metrics on 3-year SPF/DPFs Note: Assume that 3-year sub-indicator targets will be normed on 2018 data and then revisited as additional data become available.

Comparison of Alternatives
Pros Cons ESSA Factors Proposal 1: 1 year PSAT 9 and 3 years PSAT 10 - Uses all available assessment data - Consistent with what we did for 2017 reporting - Inconsistent grade representation - Inconsistent with other metrics on 3-year SPF/DPFs - Also needs to be decided for ESSA reporting - Assume that 3-year sub-indicator targets will be normed on 2018 data and then revisited as additional data become available. Proposal 2: 1 year PSAT 10 - Same amount of data for both grades - Would remove potential impact of 2017 PSAT participation issues

AWG Recommendation As we create the 3-year version of the performance framework, how many years of PSAT 9 and PSAT 10 should we include? Proposal #1- 1 year PSAT 9 and 3 years PSAT 10 Proposal #2- 1 year PSAT 9 and 1 year PSAT 10

Should Dropout rate and SAT results be reported by disaggregated group (Minority/Race/ethnicity, FRL, EL, IEP) like the other framework sub-indicators and graduation rate?

Decision Item for Today
Should Dropout rate and SAT results be reported by disaggregated group like the other framework sub- indicators and grad rate? Current Practice- Do not report disaggregated group results for Dropout rate and SAT Mean SS Alternative Proposal- Include disaggregated group results for Dropout rate and SAT Means SS Note: If the decision is made to report out individual race/ethnicity categories, this structure will also be applied to Dropout and SAT disaggregated reporting.

Current Performance Framework Reporting
The CO SAT and Dropout sub-indicators are only reported for the All Students Group, not by disaggregated group

Alternative Proposal Performance Framework

Current Scenario: Do not report disaggregated group results for Dropout rate and SAT Mean SS
Pros Aligns with previous practice Cons Does not align with disaggregated reporting for all other sub-indicators Note: ESSA requires disaggregated group reporting for all sub-indicators (will include dropout and SAT)

Proposal: Include disaggregated group results for Dropout rate and SAT Means SS
Pros Aligns with all other framework sub-indicators being reported by disaggregated group (Note: If the decision is made to report out individual race/ethnicity categories, this structure will also be applied to Dropout and SAT disaggregated reporting. ) Cons Makes the framework reports longer

Comparison of Alternatives
Pros Cons ESSA Current: Do not report disaggregated group results for Dropout rate and SAT Mean SS - Aligns with previous practice - Does not align with disagg reporting for all other sub-indicators - ESSA requires disaggregated group reporting for all sub-indicators (will include dropout and SAT) Proposal: Include disaggregated group results for Dropout rate and SAT Means SS - Aligns with all other framework sub-indicators being reported by disagg group

AWG Recommendation Should Dropout rate and SAT results be reported by disaggregated group like the other framework sub- indicators and grad rate? Current Practice- Do not report disaggregated group results for Dropout rate and SAT Mean SS Alternative Proposal- Include disaggregated group results for Dropout rate and SAT Means SS

CDE Reporting Priorities Feedback

CDE Reporting Priorities | Background
In the spring of 2016, the AWG provided recommendations to CDE’s commissioner on desired enhancements and revisions to the state’s performance frameworks (Accountability 2.0). The AWG was divided into small work groups to tackle framework topic areas that were identified as issues requiring revisions based on input received by CDE from stakeholders. The communications and reporting group met with district communications experts and reviewed several reporting resources to help inform recommendations on updating current displays and reporting features for the frameworks.

CDE Reporting Priorities | AWG Recommendations
Recommended Tiered Reporting Structure

Recommended Tiered Reporting Structure Under the tiered reporting structure recommended, Level 1 would likely provide sufficient information to ensure that most parents and community members understood the core indicators used to evaluate districts and schools.

Recommended Tiered Reporting Structure Level 2 would be primarily used by school and district staff and could also be accessed by those in the broader public interested in acquiring details underlying Level 1, specific to how the accountability determinations were made. Level 2 would provide a drill-down feature for each indicator to evaluate performance on all sub-indicators used to inform indicator ratings, including disaggregated performance data by disaggregated groups and trend data.

Recommended Tiered Reporting Structure Level 3 data would provide the most comprehensive data on schools and districts. This level may also incorporate more complex data points (e.g., standard errors, effect sizes, etc.) to assist with broader evaluation efforts of school performance. Depending on the data included at this level and technology platforms available, Level 3 may also provide locked-views to enable school and district staff to access and review data that cannot be publicly shared due to data privacy restrictions.

CDE Reporting Priorities | AWG Feedback
Please take some time over lunch to provide any feedback around the proposed tiered reporting structure as we explore how to put this recommendation into action! Feedback forms can be the center of your table

Lunch & Reporting Priorities Feedback

ESSA Updates

ESSA Plan Submissions Round 1 Plan submitted ~ May 9, 2017
USDE feedback / phone call ~ August 9, 2017 USDE feedback / letter ~ August 11, 2017 Round 2 Revisions submitted ~ October 23, 2017 USDE feedback ~ December 2017 Round 3 Revisions submitted ~ February 6, 2018 USDE feedback ~ coming soon Next steps ~ TBD

When We Met on September 21, 2017…
Reviewed the required changes at that time, gathered input, revised and resubmitted plan based on your feedback: Achievement and Participation Cannot exclude parent refusals K-2 Schools Develop methods and criteria for identifying schools for support and improvement Achievement Goals and Targets Set different long-term achievement goals for student groups so that all groups are increasing over time English Learner Progress Goals and Timelines Establish cut-scores for ELP, FEP re-designation, timeline to proficiency and method for measuring progress towards established targets Other Indicator Move science achievement out of academic achievement indicator to SQSS, in addition to reduction of chronic absenteeism and dropout rates Areas of Further Clarification

Since Then… Received a second round of feedback from USDE with additional changes requested: Academic Achievement Long-Term Goals Methodology meets requirements but the rate of increase is not enough – long- term goal for some groups is still below meets expectations Clarification on Non-Participant Calculations Must count non-participants, above 5%, as not proficient Graduation Rate Must include points for both 4-year and can add points for extended year Cannot use completion rate for identification of AECs for comprehensive support and improvement – low graduation rate category Identification of Small Schools for ESSA Support and Improvement Must have method and criteria for identifying small schools for ESSA support and improvement, if the school does not meet minimum N with 3-years of data More Areas of Further Clarification

ESSA School Identification and Simulated Impact

Simulated Impact Prioritization could change for some schools Additional schools may be identified

Additional Considerations
Identification of K-2 schools Inclusion of ELP progress, when available Moving from “Minority students” group to each major racial/ethnic group Identification of small schools Inclusion of additional indicators of school quality and student success Change in chronic absenteeism Dropout rates

ESSA/State Communication Needs

Moving Forward with Multiple Ways to Look at Data
We have learned from our years with AYP, SAR and district accreditation in Colorado Too many different determinations can water down the meaning of the data We know we need to be proactive in explaining the purpose of the state and federal requirements and making sense out of them We would like your help thinking through the most important/impactful work we can do

Help Us Prioritize What Questions Will Arise? What are the Best Tools/Strategies to Address those Questions?

Wrap-Up

Thank you! ADP

Feedback What Worked? What to Improve?

Accountability Work Group

Similar presentations

Presentation on theme: "Accountability Work Group"— Presentation transcript:

Similar presentations

About project

Feedback

Log in

Auth with social network:

Accountability Work Group

Similar presentations

Presentation on theme: "Accountability Work Group"— Presentation transcript:

Similar presentations

About project

Feedback