1. Employing Empirical Data in Judgmental Processes Wayne J. Camara National Conference on Student Assessment, San Diego, CA June 23, 2015

2. 2 Performance Levels –Licensing and Certification - Pass/Fail –Graduation Tests – Advanced (Regents) /Honors/Pass/Fail –AP Exams – 5, 4, 3, 2, 1 –State Tests/NAEP – Advanced/Proficient/Basic/Below Basic Performance Levels established based on –Judgmental methods (item, person) –Benchmarked performance (external data)

3. 3 Content – performance level (Proficient Ohio Graduation Test, 2006) Students performing at the Proficient level apply mathematical concepts, terms and properties to problem situations. Most times, students can solve problems with two or more steps or decision points. They usually make appropriate decisions about what to do and can use informal reasoning and problem-solving strategies. Students typically can interpret or provide a visual or symbolic representation to match a problem situation and purpose. Students communicate mathematical thinking and solutions using a combination of informal and mathematical language. Performance level descriptors

4. 4 Empirical benchmark (ACT, 2013): Students attaining the College Readiness benchmark have a 50% probability of attaining a grade >B and a 75% probability of attaining a grade > C in college algebra across a nationally representative sample of colleges.

5. 5 Item difficulty (p-values) Gap between actual and perceived item difficulty Internal consistency of raters (as a group, consistency of individual rater to other raters) _______________________________________________ Impact on total group (% in each category or passing) Impact of subgroups (differential impact) Trends – across years, subjects, grades, regions _______________________________________________ Within tables and across tables The timing of the data (after ratings, before next rating, Policy panel, State Board) Data Incorporated into Judgmental Processes

6. 6 Setting Cut Scores 1.Regression Based Methods Expectancy Tables (Set insurance rates, establish probabilities). Using PSAT scores to predict probability of passing specific AP examinations (Ewing, Camara & Millsap, 2006) 2.Norm-Referenced Fixed percent (by admin, study)) Based on SEM 3.External Outcomes AP grades (3=C, 4=B, 5=1) Normative Methods and Impact Data

7. 7

8. 8 Empirical – Predictive Models Briefing Book (or data book) - Haertel et al, 2012 Evidence based standard setting – McClarity et al., 2012 Judgmental Standard Setting with post hoc external data (incorporated or separate) Judgmental Standard setting setting without external data

9. 9 –Internal studies – bridge studies, linking studies, correlational studies –External studies – college course data, Job performance, training success, course enrollment, NAEP, PISA, ASVAB, Workeys, ACT –Consider content-construct relationship, data quality, motivation –Methods – equipercentile linking, Regression or projection (OLS, logistic, hierarchical) –Panelists – content, policymakers, horizontal vs vertical lines of expertise –Setting neighborhoods vs cut scores Evaluating External Data Before It is Introduced 9

10. 10 Three Methods of Incorporating Outcome Data in Standard Setting 10 Final Cut Scores At end of rating process (Round 2 of 3) show panelists relevant outcome data along with any other impact data, and allow discussion. Outcome Data as Impact with Standard Setting Panel Bring a wide range of outcome and correlational data to the table. Data Collection Bring relevant outcome data to a separate policy panel changed with evaluating the standards / cut scores before acceptance. Outcome Data with Separate Policy Panel

11. ACT College Readiness Benchmarks 11 ACT Math Benchmark

12. 12 ACT Test (1-36) Freshmen Course (50% B + 75% C) 20052013Bench mark 2013 (2005) % Meeting 2013 Cohort Logistic Reg Slope Coeff. EnglishEnglish Comp.0.1200.13218 79% MathCollege Alg.0.1950.20422 73% ReadingSocial Science0.1090.13522 (21) 74% ScienceBiology0.1690.20123 (24) 77%

13. ROUND 1 PROFICIENT LEVEL RATING FORM* 13 A prob B or higher prob C or higher prob WY StateNational 201320142013 0.130.400.6742 51 0.150.420.68394050 0.160.440.69373848 0.170.460.71353647 0.190.480.72333545 0.200.500.73313443 0.220.520.74303141 0.240.540.7629 39 0.250.560.7727 37 0.270.580.7825 35 0.290.600.79222332 0.320.620.8020 30 * This is the math rating form. The reading and science forms will have slightly different probabilities and percentages.

14. ESTABLISHING CUT SCORES FOR FOUR LEVELS OF ACHIEVEMENT To divide the achievement scale into Below Basic, Basic, Proficient, and Advanced, we will focus on the lower borderline of each achievement level. BasicProficient Advanced BasicProficientAdvancedBelow Basic LowCollege and Career ReadyHigh Lower Borderline Basic Lower Borderline Proficient Lower Borderline Advanced 14

15. PROFICIENT ACHIEVEMENT LEVEL First, we will focus on the lower borderline of the Proficient achievement level. BasicProficient Advanced BasicProficientAdvancedBelow Basic LowCollege and Career ReadyHigh Lower Borderline Basic Lower Borderline Proficient Lower Borderline Advanced 15

16. 16 370 // 390 300 // 330 Proficient State Test 2013 Advanced State Test 2013 Raw Score = 50% items correct MATH HIGH SCHOOL CUT SCORE STUDIES AND IMPACT – NEW MATH TEST 2014

17. 17 ACT College Ready in State 370 // 390 300 // 330 Proficient State Test 2013 ACT College Ready in US Advanced State Test 2013 College Placement test cut score Raw Score = 50% items correct MATH HIGH SCHOOL CUT SCORE STUDIES AND IMPACT – NEW MATH TEST 2014

18. 18 ACT College Ready in State 370 // 390 300 // 330 NAEP Proficient NAEP Advanced Proficient State Test 2013 ACT College Ready in US Advanced State Test 2013 College Placement test cut score Raw Score = 50% items correct MATH HIGH SCHOOL CUT SCORE STUDIES AND IMPACT – NEW MATH TEST 2014

19. 19 ACT College Ready in State 370 // 390 300 // 330 NAEP Proficient NAEP Advanced Proficient State Test 2013 ACT College Ready in US Advanced State Test 2013 75% Prob of > B college math 75% Prob of A college math 75% Prob of > B in HS Alg II College Placement test cut score Raw Score = 50% items correct 75% Prob of > B in HS Alg II MATH HIGH SCHOOL CUT SCORE STUDIES AND IMPACT – NEW MATH TEST 2014

20. 20 ACT College Ready in State 370 // 390 300 // 330 NAEP Proficient NAEP Advanced Proficient State Test 2013 ACT College Ready in US Advanced State Test 2013 75% Prob of > B college math 75% Prob of A college math 75% Prob of > B in HS Alg II College Placement test cut score Raw Score = 50% items correct 75% Prob of > B in HS Alg II MATH HIGH SCHOOL CUT SCORE STUDIES AND IMPACT – NEW MATH TEST 2014 PROFICIENT

21. 21 ACT College Ready in State PERCENT 80% 70% 60% 50% 40% 30% 20% 10% NAEP Proficient NAEP Advanced Proficient State Test 2013 ACT College Ready in US Advanced State Test 2013 75% Prob of > B college math 75% Prob of A college math 75% Prob of > B in HS Alg II College Placement test cut score Raw Score = 50% items correct 75% Prob of > B in HS Alg II MATH HIGH SCHOOL CUT SCORE STUDIES AND IMPACT – NEW MATH TEST 2014 PROFICIENT

22. Math Proficient level – Comparison information 22

23. 23 ACT – WY Empirical Standard Setting Results ACT ScoreProbability Percentage at or Above A B or higher C or higher National 2014 Wyoming 2014 Mathematics Advanced270.460.730.851610 Proficient210.160.440.694739 Basic160.050.220.518583 Reading Advanced280.380.690.851914 Proficient230.220.530.763934 Basic170.110.330.637570 Science Advanced290.470.770.9186 Proficient230.180.490.773732 Basic180.070.260.597370

24. 24 ACT – WY Empirical Standard Setting Results