Technical Advisory Panel (TAP) Meeting

Technical Advisory Panel (TAP) Meeting
January 17, 2019

Technical Advisory Panel
Welcome! Introductions

Topics to Cover HB Rulemaking Update Colorado Accountability Theory of Action Feedback Recap of Growth to Standard creation so far and what we have left to do Growth to Standard Analysis TAP Statement

HB Rulemaking

CO State Accountability | Areas of Interest
Current State Board of Education Conversation Setting Achievement, Growth & PWR Targets on Performance Frameworks Achievement, Growth and Postsecondary & Workforce Readiness (PWR) Weighting on Performance Frameworks Addition of a new Growth-to-Standard (criterion-based growth) metric to Performance Frameworks Rulemaking Process for House Bill Bill passed in Spring 2018 concerning adjustments to the accountability system for the elementary and secondary public education system

Current State Board of Education Conversation | Timeline
2019 Target Setting HB SBE Rulemaking Hearings Earliest Date of Implementation: 2020 School & District Performance Frameworks Additional time will be allowed as needed to fully engage in rulemaking conversation.

CO State Accountability | Important Dates

CO State Accountability | Growth to Standard
Growth to standard describes student growth towards meeting grade level expectations as defined by the underlying assessment (i.e. on CMAS, how much growth would a 3rd grader need to show to be ‘on track’ to reaching the next performance level within a certain amount of time?) CDE staff is working with the Technical Advisory Panel (TAP) - composed of district administrators from across the state - to determine how this metric will be developed. The Growth to Standard metric will be developed by February 2019 to allow time to evaluate impact data (models) prior to rulemaking sessions. The TAP is analyzing historical student growth data to ensure that student-level goals are ambitious, yet attainable. The metric currently being explored: Utilizes a ‘stepping stone’ methodology (how long does it take a student to move from Level 1 to Level 2; from Level 2 to Level 3?) as opposed to the ‘reaching proficiency within 3 years or by 10th grade’ methodology that was utilized with the previous version of adequate growth. Allows students to show progress on a yearly basis. All of the TAP’s conversations are recorded and posted online, and there is time at each meeting for public comment. Please follow along if you are interested!

CO State Accountability | Engage in the Conversation!
Stakeholder Feedback Survey | Focused on key areas the SBE will be addressing during rulemaking Presentations and Handouts Draft HB Rules & Table of Contents HB Fact Sheets Additional Resources | Focused on key areas the SBE will be addressing during rulemaking CDE and SBE Contact Information

Feedback Sessions | What We’ve Heard
Pace and Timing Allow the appropriate amount of time to have thoughtful accountability conversations Gather authentic stakeholder feedback on all areas that will be addressed during the rulemaking process Allow appropriate lead time for any adjustments Develop and evaluate models to determine the impact of any potential adjustments Colorado Values Ensure that Colorado’s values are reflected in accountability policy Define and/or adhere to the purpose of school and district accountability There is a value reflected in the inclusion of growth measures on performance frameworks

CO Accountability Theory of Action

Colorado’s Quality Schools Theory of Action If we,
Colorado’s system of school and district accountability is primarily designed to prioritize and maximize support for schools and districts identified for academic improvements. Colorado’s Quality Schools Theory of Action If we, Stakeholder Action State Action Learn More IDENTIFY Identify schools and districts for additional support based on student academic outcomes CDE creates School & District Performance Frameworks and identifies schools under ESSA Why are school and district identifications based on student academic outcomes? What student academic outcomes are included in performance frameworks? Performance Frameworks 101 How are schools identified under ESSA? ASSESS NEEDS AND PLAN Schools and districts assess needs and select strategies for improvement CDE offers supports and ensures all schools and districts engage in Unified Improvement Planning. What is Colorado’s approach to school and district improvement planning? Unified Improvement Planning Process & Template IMPLEMENT Schools and districts implement selected strategies for improvement CDE offers supports and allocates resources to align with identified stakeholder needs What is Colorado’s approach to school and district support and improvement? Systems Framework for School & District Improvement (WestEd Four Domains) EASI Application INTERVENE Low performing schools and districts take more rigorous action CDE supports SBE, districts and schools through Performance Watch process. State Board of Education directs action at the local level. What is Performance Watch? Want feedback on this level of Theory of Action. Prompting questions: Then, CDE, school districts and schools can effectively partner to build capacity to better meet the educational needs of all students and work to ensure that all schools meet state performance expectations.

Theory of Action Next Steps
Examine how specific components of the accountability system align with CO’s Theory of Action: School & District Performance Frameworks Federal School Identifications State Assessments Unified Improvement Planning School & District Supports Performance Watch (Accountability Clock)

Growth-to-Standard: Update
Marie Huchton, Accountability & Data Analysis January 17, 2019

Growth to Standard Overview | Recap

Growth-to-Standard Requirement in SB18-1355
Required performance indicator for inclusion in annually- determined school and district rating calculations: “Student academic growth to standards, based on students progress toward meeting the state standards… or for students who meet grade-level expectations on the state standards, progress toward higher levels of achievement, if available, as measure by the statewide assessments.” (1)(a)(III) Which means a growth to standard metric needs to measure a student’s progress towards meeting a target level of performance within a given timeframe. And this metric needs to update/incorporate observed progress over time.

How long to achieve the target(s)?
CMAS ELA and Math Growth to Standard- Initial Theoretical Decision Points What target(s)? Should the target be set to “Meets State Expectations” or should interim targets be used for Catch Up trajectories? How long to achieve the target(s)? How many years should students be given to attain their target performance level? Should that vary by grade, content area, and/or initial performance level? How does the target update over time? Does the clock start over every year or should this be a set trajectory where we track student progress from the first test result? To be successfully on-track, do students have to maintain the gains made? How do we report? Do we report students below proficient (Catch Up) and above proficient (Keep Up) separately? Or combined? TBD- January In Progress TBD- February

What Target(s)? Should the target be set to “Meets State Expectations” or should interim targets be used for Catch Up trajectories? Having reviewed historical data, very few students starting at the lowest performance levels attain proficiency within 4 years. The proportions of students moving up one or more achievement levels in 1, 2, 3, or 4-years varies by content grade level and starting achievement level, but in general is around 40-60% of students. Maintaining gains is quite difficult, with nearly half of all students dropping back down to their original proficiency level at some point.

2018 SGP Results and Relationship to Observed Achievement Trajectories- ELA, All Students
Achievement Level 2018 L L L L L5 For the All Students group combining across grades in ELA, moving either up or down one or more achievement levels requires significantly higher (or lower) than average growth. Students with typical growth tend to stay at the same achievement level from one year to the next (notable exception for level 5) L L L L L1 Achievement Level 2017

2018 SGP Results and Relationship to Observed Achievement Trajectories- Math, All Students
Achievement Level 2018 L L L L4 L5 Similar to ELA, for the All Students group combining across grades in Math, moving either up or down one or more achievement levels requires significantly higher (or lower) than average growth. Students with typical growth tend to stay at the same achievement level from one year to the next (similar notable exception for level 5) L L L L L1 Achievement Level 2017

What Target(s)? TAP recommended using interim targets and a “stepping-stone” model based on observed data and theoretical considerations. Emphasizes the gains over the course of a year, rather than solely focusing on did students hit the minimum expectation for a grade level. Sets realistic goals for student improvement given observed historical student performance.

Today’s Focus Areas What target(s)? Should the target be set to “Meets State Expectations” or should interim targets be used for Catch Up trajectories? How long to achieve the target(s)? How many years should students be given to attain their target performance level? Should that vary by grade, content area, and/or initial performance level? How does the target update over time? Does the clock start over every year or should this be a set trajectory where we track student progress from the first test result? To be successfully on-track, do students have to maintain the gains made? How do we report? Do we report students below proficient (Catch Up) and above proficient (Keep Up) separately? Or combined? TBD- January In Progress TBD- February

Growth to Standard “How does the target update over time?”

CMAS ELA and Math Growth to Standard- Initial Theoretical Decision Points What target(s)? Should the target be set to “Meets State Expectations” or should interim targets be used for Catch Up trajectories? How long to achieve the target(s)? How many years should students be given to attain their target performance level? Should that vary by grade, content area, and/or initial performance level? How does the target update over time? Does the clock start over every year or should this be a set trajectory where we track student progress from the first test result? To be successfully on-track, do students have to maintain the gains made? Do we report students below proficient (Catch Up) and above proficient (Keep Up) separately? Or combined? TBD- January In Progress TBD- February

Hypothetical 2016 On Track Prediction Outcomes- 3 Scenarios for Setting Target Growth Percentiles
Within a stepping-stone model there are different possibilities for setting individual target student growth percentiles and on track predictions in the baseline year and then subsequent years and then tracking whether the predicted outcomes were correct. Using 2016 as the baseline year and tracking through the 2018 outcomes, these three target scenarios were explored: Maintain Initial 2016 Targets Maintain Initial 2016 Targets Until Attained, then Reset Reset Targets Every Year

On Track Up Prediction Outcomes- Maintaining Initial 2016 Targets
grade grade grade 5 grade 6 2015- Prior Current/Base Future Future 2 The year 0 projection is always 100% accurate because we already know what happened between the prior and current year. L4 L3 SGP= 87 2016 CU L1toL2_y0 = On Track Correctly Predicted 722 L2 TGP=56 TGP=47 TGP=44 675 L1

On Track Prediction Outcomes- Maintaining Initial 2016 Targets
grade grade grade 5 grade 6 2015- Base Prior Baseline Current Future 1 If we maintain the 2016 trajectory and target growth percentiles (TGPs) and compare these against 2017 outcomes, our predictive accuracy goes down, averaging 82.3% for ELA and 82.7% for Math, combining Catch Up and Keep Up across Elementary grades. L4 L3 SGP= 87 2016 CU L1toL2_y0 = On Track Correctly Predicted 722 2016 CU L1toL2_y1 = On Track Incorrectly Predicted L2 TGP=56 TGP=47 TGP=44 688 675 SGP= 8 L1

Comparing the 2016 trajectory and TGPs against the 2018 outcomes reduces the accuracy of our On Track predictions a little bit more, averaging 75.2% for ELA and 76.1% for Math combining Catch Up and Keep Up across Elementary grades. grade grade grade 5 grade 6 2015- Base Prior Baseline Outcome Current L4 L3 SGP= 87 2016 CU L1toL2_y1 = On Track Correctly Predicted 722 2016 CU L1toL2_y1 = On Track Incorrectly Predicted 2016 CU L1toL2_y2 = On Track Incorrectly Predicted L2 TGP=56 TGP=47 TGP=44 688 688 675 SGP= 8 SGP= 29 L1

Under the Maintain scenario, each student would have a single starting point and all future progress would be measured against that starting point. Doesn’t account for students moving between levels grade grade grade 5 grade 6 2015- Base Prior Baseline Outcome Current L4 L3 SGP= 87 2016 CU L1toL2_y1 = On Track Correctly Predicted 722 2016 CU L1toL2_y1 = On Track Incorrectly Predicted 2016 CU L1toL2_y2 = On Track Incorrectly Predicted L2 TGP=56 TGP=47 TGP=44 688 688 675 SGP= 8 SGP= 29 L1

On Track Prediction Outcomes- Maintaining Initial 2016 Targets Until Attained, then Reset
grade grade grade 5 grade 6 2015- Prior Current Future Future 2 Again, using the year 0 TGP is always 100% accurate because we know what actually happened between the prior and current year. L4 L3 SGP= 87 2016 CU L1toL2_y0 = On Track Correctly Predicted 722 L2 TGP=56 TGP=47 TGP=44 675 L1

Students scoring at or below the same proficiency level in 2015 and 2016 maintain their 2016 TGPs, while students moving up 1+ PLs have their targets reset to their 2017 TGPs. Looking at the 2017 outcomes, our predictive accuracy drops significantly, averaging 64.7% for ELA and 70.4% for Math, combining Catch Up and Keep Up across Elementary grades. grade grade grade 5 grade 6 2015- Prior Prior Current Future 1 L4 2017 CU L2toL3_y0 = Not On Track Correctly Predicted L3 TGP=71 TGP=58 722 L2 688 675 SGP= 8 L1

There are now four possible combinations of student proficiency trajectories- No movement from 15 to 16, or 16 to 17 – use 2016 TGP No movement from 15 to 16, upward movement from 16 to 17- use 2018 TGP Upward movement from 15 to 16, no movement from 16 to 17- use 2017 TGP Upward movement from 15 to 16 and 16 to 17- use 2018 TGP grade grade grade 5 grade 6 2015- Prior Prior Current Future 1 2017 CU L2toL3_y0 = Not On Track Correctly Predicted L3 TGP=71 TGP=58 722 2017 CU L2toL3_y1 = Not On Track Correctly Predicted L2 688 688 675 SGP= 8 SGP= 29 L1

On Track Prediction Outcomes- Reset Targets Every Year
grade grade grade 5 grade 6 2015- Prior Current Future Future 2 Again, using the year 0 TGP is always 100% accurate because we know what actually happened between the prior and current year. L4 L3 SGP= 87 2016 CU L1toL2_y0 = On Track Correctly Predicted 722 L2 TGP=56 TGP=47 TGP=44 675 L1

grade grade grade 5 grade 6 2015- Prior Prior Current Future 1 Since the targets reset each year, our predictions will always be 100% accurate because the slate of prior target expectations and timelines is wiped clean each year. L4 2017 CU L2toL3_y0 = Not On Track Correctly Predicted L3 TGP=71 TGP=58 722 L2 688 675 SGP= 8 L1

grade grade grade 5 grade 6 2015- Prior Prior Prior Current Since the targets reset each year, our predictions will always be 100% accurate because the slate of prior target expectations and timelines is wiped clean each year. L4 L3 722 2018 CU L1toL2_y0 = Not On Track Correctly Predicted L2 TGP=34 688 675 688 SGP= 29 L1

Observed School Aggregations of 3 Year On Track Target Scenarios

Observed 2018 School Aggregations of 2016 3 Year On Track Targets
School level aggregations of observed 2018 student On/Off Track flags for each of the above Year Target Scenarios then compared outcomes to see how different the inferences of average student performance become depending upon which target-setting methodology is used.

Comparison of 2018 School Aggregated Percent On Track 2016 3 yr Target Scenarios- All Students, ELA

Comparison of 2018 School Aggregated Percent On Track 2016 3 yr Target Scenarios- All Students, Math

Maintaining Initial Targets
Pros Cons Considerations - Conceptually simple, single baseline for each child - Explicitly tracks whether students have attained target within given timeframe - Really only appropriate if we were using a single at-benchmark target, rather than stepping-stone approach - Doesn’t allow for trajectories to reset if students attain next level of performance - Schools potentially held responsible for student target trajectories established at previous schools - At some point would need to reset even if student still hadn’t made progress - When aggregated at the school and district-level, would result in similar % On Track Catch Up averages to Reset scenario, but less dispersion

Maintaining Initial Targets Until Attained, then Reset
Pros Cons Considerations - Explicitly tracks whether students have attained target within given timeframe - Recognizes upward movement and resets targets to achieve next performance level within new time frame - Calculationally most complex, having to keep track of original as well as reset trajectories - Inconsistent baseline target years- students not moving between levels would maintain their original targets forever, while other students making progress would have updated targets every year - When aggregated at the school and district-level, would result in lowest % On Track distributions of all scenarios, especially for Catch Up - Targets based on more prior years of data are more precise - Assumes upward movement learning progression - Schools potentially held responsible for student target trajectories established at previous schools

Pros Cons Considerations - Consistent with stepping stone methodology for achieving incremental goals within shorter timeframes - Aligns with previous TCAP AGP methodology - Fairly simple to explain - Generous metric, giving credit both for students who have moved and those whose current growth, if sustained over time, would move up. - Clock resets every year and never checks that students actually reached the target within allotted timeframe. - When aggregated at the school and district-level, would result in similar % On Track Catch Up averages to Maintain scenario, but greater dispersion - Strategic communication will be required to make clear the options considered by CDE and TAP and the decision-making rationale if settling upon a metric that looks similar to previous.

TAP Recommendation Does the TAP support moving forward using the Reset Methodology?

Growth to Standard “How long to achieve targets?”

CMAS ELA and Math Growth to Standard- Initial Theoretical Decision Points What target(s)? Should the target be set to “Meets State Expectations” or should interim targets be used for Catch Up trajectories? How long to achieve the target(s)? How many years should students be given to attain their target performance level? Should that vary by grade, content area, and/or initial performance level? How does the target update over time? Does the clock start over every year or should this be a set trajectory where we track student progress from the first test result? To be successfully on-track, do students have to maintain the gains made? Do we report students below proficient (Catch Up) and above proficient (Keep Up) separately? Or combined? TBD- January TBD- February

How many years should students be given to attain their target performance level? Should that vary by grade, content area, and/or initial performance level? Using a stepping stone approach, a student’s target depends upon their starting proficiency level Allowing the targets to reset every year ensures the expectation for each student is to move up to the next level

Grade 3, Achievement Level 1 Cohort- L1/L2 Target Percentiles by Prior Scale Score
In 1 Year: In 2 Years: In 3 Years: In 4 Years: g3 to g g3 to g g3 to g g3 to g7 This progression of plots shows that, for grade 3 students scoring at level 1 in the prior year, the targets necessary to move to level 2 become more similar as more future years are added, regardless of how close/far a student was initially from the level 2 cut-score.

Achievement Level 1 Cohort- Target Growth Percentile Distributions by Timeframe
This graphic shows the distributions of target student growth percentiles for students starting at Level 1 and expected to move to Level 2, within 1 year (i.e. the current year), 2 years, 3 years, or 4 years. The mean TGPs decrease from year 1 to year 3, but bump back up for year 4. N=23,710

For students starting at Level 2 needing to move to Level 3, slightly above average growth will always be required, regardless of the timeframe allotted. N=30,666

Moving from Level 3 to Level 4 requires even higher average growth for students than the other Catch Up scenarios. N=45,187

For students already Meeting State Expectations (Level 4), it is fairly easy to maintain Level 4 for 1 Year, and becomes increasingly difficult as time goes on though the target growth percentiles are always below 50. N=67,400

For students Exceeding State Expectations (Level 5), it is very easy to maintain Level 4 for 1 Year, and becomes somewhat more difficult as time goes on though the target growth percentiles are always below 20. N=10,681

Target Growth Percentile Distributions by Timeframe- Math
Math shows similar patterns, though the target growth percentiles are always more rigorous than for ELA

Given these trends in student-level targets, it should not be surprising that For students starting below proficient (Catch Up), longer timeframes would yield lower adequate growth percentiles and an increased likelihood a student will be identified as “On Track” For students starting at/above proficient (Keep Up), longer timeframes would yield higher adequate growth percentiles and the possibility students may not stay “On Track” over time How do these “On Track” results aggregate up at the school level?

School-level Distributions of % On Track by Timeframe- Grade 4 ELA, Catch Up L1 to L2
Given a 1 year timeframe, the average % of students On Track to go from L1 to L2 is 53.6%, if we increase the timeframe to 2 years, it becomes 58.6%, and after either 3 or 4 years the average is 60.9%.

An average of 54.0% of students are On Track to move between L2 and L3 in 1 year, and 59.8% are on track to move within 2, 3, or 4 years.

An average of 40.5% of students are On Track to move between L3 and L4 in 1 year, 47.5% within 2 years, 47.9% within 3 years and 51.7% within 4 years.

School-level Distributions of % On Track by Timeframe- Grade 4 ELA, Keep Up L4 to L4
Given a 1 year timeframe, the average % of students at L4 On Track to stay at or above L4 is 83.9%, over a 2 year timeframe it drops to 77.2%, and over 3 and 4 years drops to around 65%.

School-level Distributions of % On Track by Timeframe- Grade 4 ELA, Keep Up L5 to L4
An average of 99.4% of students at L5 are On Track to stay at at/above L4 over 1 year, 94.7% over 2 years, 85.1% over 3 years, and 81.0% over 4 years.

School-level Distributions of % On Track by Timeframe- Grade 4 ELA, Grade 4 ELA, Catch Up
Combining the individual Catch Up trajectories together, schools average 49.1% of below- proficient students On Track to move up one proficiency level within 1 year, 55.0% in 2 years, 55.9% in 3 years and 57.4% in 4 years.

School-level Distributions of % On Track by Timeframe- Grade 4 ELA, Keep Up
And combining the Keep Up trajectories together, schools average 86.6% of at/above-proficient students On Track to stay at or above L4 over 1 year, 80.1% over 2 years, 68.9% over 3 years and 67.8% over 4 years.

School-level Distributions of % On Track by Timeframe- Grade 4 ELA, All Trajectories
And combining all the Catch Up and Keep Up trajectories together, schools average 63.2% of students On Track over 1 year, 64.1% over 2 years, 60.2% over 3 years and 60.7% over 4 years.

School-level Distributions of % On Track by Timeframe- Grade 4 Math
Differences between Math timeframes are greater for CU and reduced for KU

Considerations for Determining Timeline to Target
Other grade levels and content areas show similar patterns Choosing a shorter timeframe will result in smaller proportions of Catch Up students and greater proportions of Keep Up students being identified as “On Track” and a metric that is more closely aligned to achievement outcomes (and demographics) and less to growth. Choosing a longer timeframe will result in larger proportions of Catch Up students and smaller proportions of Keep Up students being identified as “On Track” and a metric that is more closely aligned to growth outcomes.

School-level % On Track Distributions- Comparisons by Timeframe
Longer target timeframes allow for better differentiation among schools, since the Keep Up results do not pile-up quite so heavily at the top of the scale

The Catch Up results are more aligned with the All Trajectory results when the 3 year target timeframes are used

Given longer target timeframes, both the Keep Up and All Trajectory results show more differentiation among schools and are more highly correlated.

Given longer target timeframes, the correlation between the % On Track across All Trajectories and the Mean Scale Score decreases

Given longer target timeframes, the correlation between % On Track All Trajectories and MGP increases considerably

Given longer target timeframes, the % On Track across All Trajectories is less correlated with % FRL

CDE Thoughts Across all grades, content areas, and CUKU combinations, the 3-year timeline consistently appears as the sweet spot: Greater similarity between the Catch Up and Keep Up distributions Results show higher correlations with growth and lower correlations with achievement metrics 4 year results are often similar to 3 year, so little benefit in extending timeline further CDE will investigate whether the 7th and 8th grade targets can also be meaningfully played out to 3 years, rather than being truncated at 2 years and 1 year.

TAP Recommendation Does the TAP support using a 3-year timeline for students across all grades and content areas?

TAP Statement

Technical Advisory Panel
Meeting Summary: Suggested future analysis TAP recommendations from this meeting Public Comment Close Meeting Next Scheduled Meeting, February 5th(Tues), noon-4 at Broadway, 19th Floor, conference room.

Technical Advisory Panel (TAP) Meeting

Similar presentations

Presentation on theme: "Technical Advisory Panel (TAP) Meeting"— Presentation transcript:

Similar presentations

About project

Feedback

Log in

Auth with social network:

Technical Advisory Panel (TAP) Meeting

Similar presentations

Presentation on theme: "Technical Advisory Panel (TAP) Meeting"— Presentation transcript:

Similar presentations

About project

Feedback