PUBLIC SCHOOLS OF NORTH CAROLINA STATE BOARD OF EDUCATION DEPARTMENT OF PUBLIC INSTRUCTION 1 Review of the ABCs Standards SBE Issues Session March 2, 2005

PUBLIC SCHOOLS OF NORTH CAROLINA STATE BOARD OF EDUCATION DEPARTMENT OF PUBLIC INSTRUCTION 2 ABCs Accountability Basic skills with high education standards Control (at the local level)

PUBLIC SCHOOLS OF NORTH CAROLINA STATE BOARD OF EDUCATION DEPARTMENT OF PUBLIC INSTRUCTION 3 ABCs Accountability first year for grades K first year for high school

PUBLIC SCHOOLS OF NORTH CAROLINA STATE BOARD OF EDUCATION DEPARTMENT OF PUBLIC INSTRUCTION 4 Why Review Now? A provision in HB 1414 requires a comprehensive review of ABCs standards This same provision requires that the modified standards be in place for the school year

PUBLIC SCHOOLS OF NORTH CAROLINA STATE BOARD OF EDUCATION DEPARTMENT OF PUBLIC INSTRUCTION 5 Objectives of the Testing Program Consistent with 3 purposes in G.S. 115C Assure all HS graduates possess essential skills and knowledge Provide a means of identifying strengths and weaknesses in education process in order to improve instruction Make system accountable to the public for results

PUBLIC SCHOOLS OF NORTH CAROLINA STATE BOARD OF EDUCATION DEPARTMENT OF PUBLIC INSTRUCTION 6 1.Public and understandable 2.Multiple sources 3.Interpretations at student, school and LEA level 4.All students and groups 5.Explicit weighting 6.Avoids erroneous judgments CRESST Standards (Center for Research on Evaluation, Standards, and Student Testing)

PUBLIC SCHOOLS OF NORTH CAROLINA STATE BOARD OF EDUCATION DEPARTMENT OF PUBLIC INSTRUCTION 7 Related Issues Achievement levels Need clearer definitions Teachers still code same percent as Level I as in the past Teachers anticipate students’ successes regardless of previous achievement levels Process issues Various business process issues NC WISE will help Resolutions already in process

PUBLIC SCHOOLS OF NORTH CAROLINA STATE BOARD OF EDUCATION DEPARTMENT OF PUBLIC INSTRUCTION 8 ABCs Current Formulas Only at school level Statewide average scale score growth, index of true proficiency, index of regression to the mean More complex at the HS level Links multiple subjects State of the art for its time

PUBLIC SCHOOLS OF NORTH CAROLINA STATE BOARD OF EDUCATION DEPARTMENT OF PUBLIC INSTRUCTION 9 Recommendations Reviewed Technical Advisory Committee Standing group of recognized experts Supported proposal with suggestions for additional analyses Helped define high growth Provided input for minor adjustments

PUBLIC SCHOOLS OF NORTH CAROLINA STATE BOARD OF EDUCATION DEPARTMENT OF PUBLIC INSTRUCTION 10 Findings from Analyzing Results of Original Formulas Saw-toothed pattern: As a cohort moved through grades, % of schools making growth peaked and dipped Not necessarily tied to curriculum implementation

PUBLIC SCHOOLS OF NORTH CAROLINA STATE BOARD OF EDUCATION DEPARTMENT OF PUBLIC INSTRUCTION 11 More Findings EOC formulas: Drop some students from calculations Complex Curriculum changes can be problematic

PUBLIC SCHOOLS OF NORTH CAROLINA STATE BOARD OF EDUCATION DEPARTMENT OF PUBLIC INSTRUCTION 12 Fundamental Issues Change in curriculum prompts new test edition Change in one subject impacted other Change in curriculum expectations not mirrored in growth expectation Required conversions to old scale Saw-toothed effect

PUBLIC SCHOOLS OF NORTH CAROLINA STATE BOARD OF EDUCATION DEPARTMENT OF PUBLIC INSTRUCTION 13 Proposed Solutions Issue: Saw-toothed effect Solution: Averaging test performance across years increases stability

PUBLIC SCHOOLS OF NORTH CAROLINA STATE BOARD OF EDUCATION DEPARTMENT OF PUBLIC INSTRUCTION 14 How It Works Look at two previous years when possible Average, to find the “typical” performance of the student Use 3 years of actual student data for decisions

PUBLIC SCHOOLS OF NORTH CAROLINA STATE BOARD OF EDUCATION DEPARTMENT OF PUBLIC INSTRUCTION 15 Proposed Solutions Issue: Effects of curricula revisions Tie to curriculum (The relative difficulty of the grade level curricula may have shifted) Current method relies on equating study to “link” to old scale Solution: Use a scale that adjusts to difficulty of curriculum (base expectation on actual curriculum)

PUBLIC SCHOOLS OF NORTH CAROLINA STATE BOARD OF EDUCATION DEPARTMENT OF PUBLIC INSTRUCTION 16 How It Works Use students’ actual performance as the basis of the Change Scale (in the standard-setting year) Convert all scores to the Change Scale Calculate expectations using the Change Scale

PUBLIC SCHOOLS OF NORTH CAROLINA STATE BOARD OF EDUCATION DEPARTMENT OF PUBLIC INSTRUCTION 17 Proposed Solutions Issue: EOC Formula issues EOC sequence not coupled in the same manner as EOGs Solution: Simplify the formulas Fewer predictors – keep more students in the analyses Use of Change Scale allows smoother transitions between test editions

PUBLIC SCHOOLS OF NORTH CAROLINA STATE BOARD OF EDUCATION DEPARTMENT OF PUBLIC INSTRUCTION 18 How It Works Find the previous test(s) with the highest correlations to the current year’s tests Examples: Algebra I based on 8 th grade math Biology based on English I and 8 th grade reading Physics based on Chemistry and Geometry

PUBLIC SCHOOLS OF NORTH CAROLINA STATE BOARD OF EDUCATION DEPARTMENT OF PUBLIC INSTRUCTION 19 Proposed Solutions Issue: Changes in curricula and test editions Solution: Uncouple reading and math Predict reading using only reading Correlations are similar

PUBLIC SCHOOLS OF NORTH CAROLINA STATE BOARD OF EDUCATION DEPARTMENT OF PUBLIC INSTRUCTION 20 How It Works Original formulas used average of math and reading to provide additional information for prediction Instead, use 2 years of reading for reading prediction Correlations are better Outliers have less effect

PUBLIC SCHOOLS OF NORTH CAROLINA STATE BOARD OF EDUCATION DEPARTMENT OF PUBLIC INSTRUCTION 21 Proposed Solutions Issue: Conversions to old scales Multiple levels of issues Approaching third edition of math tests Solution: Use a scale-neutral system Allows true spanning of test editions No need to reset growth standards

PUBLIC SCHOOLS OF NORTH CAROLINA STATE BOARD OF EDUCATION DEPARTMENT OF PUBLIC INSTRUCTION 22 How It Works Based on the standard setting year of the new test edition Use the actual student performance on the test; convert to the Change Scale Use the actual state standard deviation to form the points on the Change Scale Maintain these conversion tables until curriculum and test edition change

PUBLIC SCHOOLS OF NORTH CAROLINA STATE BOARD OF EDUCATION DEPARTMENT OF PUBLIC INSTRUCTION 23 The Proposed Formulas Based on groups of students spanning 8 years of ABCs data Includes over 500,000 students per grade and subject (grades 3-8) Use a consistent scale Simplified to allow easier understanding Reviewed by outside experts

PUBLIC SCHOOLS OF NORTH CAROLINA STATE BOARD OF EDUCATION DEPARTMENT OF PUBLIC INSTRUCTION 24 AC = CS – (0.9 ATPA) Academic Change = the difference between the student’s actual performance on the curriculum-based test for this grade and his/her typical performance CS = current year score ATPA = average of two previous assessments

PUBLIC SCHOOLS OF NORTH CAROLINA STATE BOARD OF EDUCATION DEPARTMENT OF PUBLIC INSTRUCTION 25 AC = CS – (0.9 ATPA) The 0.9 accounts for regression to the mean All analyses point to this formula being fair Works for all grades and subjects Use 0.8 when only one previous score is available (there is more regression to the mean when using one previous test rather than the average of two)

PUBLIC SCHOOLS OF NORTH CAROLINA STATE BOARD OF EDUCATION DEPARTMENT OF PUBLIC INSTRUCTION 26 Anticipated Impact Results are different, especially in years with a test edition change Different formulas, different standards, different results

PUBLIC SCHOOLS OF NORTH CAROLINA STATE BOARD OF EDUCATION DEPARTMENT OF PUBLIC INSTRUCTION 27 How It Works – Expected Growth Calculate student level AC per subject and grade Average AC across grades and subjects within a school Grades 3-8: positive or “0” meets expected growth HS: include weighted dropout change, competency pass rate change and CTP/CUP change Consistent with original model Positive or “0” meets expected growth

PUBLIC SCHOOLS OF NORTH CAROLINA STATE BOARD OF EDUCATION DEPARTMENT OF PUBLIC INSTRUCTION 28 How It Works – High Growth TAC recommended use of a ratio Ratio of students with positive AC to those with negative AC (called c-ratio) HS: include dropouts in denominator; CTP/CUP graduates and competency pass rate change in numerator 1.50 or higher met high growth (if met expected growth)

PUBLIC SCHOOLS OF NORTH CAROLINA STATE BOARD OF EDUCATION DEPARTMENT OF PUBLIC INSTRUCTION 29 Advantages More stability Fewer fluctuations between years Clear indication of change in curriculum implementation Adjusts expectations to the curriculum Simpler formulas

PUBLIC SCHOOLS OF NORTH CAROLINA STATE BOARD OF EDUCATION DEPARTMENT OF PUBLIC INSTRUCTION 30 Advantages: What the Technical Experts Said More resilient due to two years’ prior performance Different results in test edition change years than previous Equitable Regression coefficients appropriate for the test relationships

PUBLIC SCHOOLS OF NORTH CAROLINA STATE BOARD OF EDUCATION DEPARTMENT OF PUBLIC INSTRUCTION 31 Advantages: Student Information Provides information on student performance C-ratio allows insight into how well schools or grades are implementing curriculum

PUBLIC SCHOOLS OF NORTH CAROLINA STATE BOARD OF EDUCATION DEPARTMENT OF PUBLIC INSTRUCTION 32 Relationship to (CRESST) Standards 1.Public and understandable  2.Multiple sources  3.Interpretations at student, school and LEA level  4.All students and groups  5.Explicit weighting  6.Avoids erroneous judgments 

PUBLIC SCHOOLS OF NORTH CAROLINA STATE BOARD OF EDUCATION DEPARTMENT OF PUBLIC INSTRUCTION 33 Review of the ABCs Standards SBE Issues Session March 2, 2005