Setting Performance Standards Grades 5-7 NJ ASK NJDOE Riverside Publishing May 17, 2006

2 What do we mean by “Standard Setting ” ? Content standards: specify the content that should be learned by students – given implementation of the state standards and an opportunity to learn. Performance standards: specify the precise level of performance required on an assessment to achieve some category or outcome. How much must students know and be able to do?

3 Proficiency Levels on NJ ASK Advanced Proficient Partially Proficient Cut Score Advanced Cut Score

4 NJ ASK Grades 5-7 Standard Setting Phase 1  20 panelists on the Math Grades 5-7 & LAL Grade 7 panels  19 panelists on the LAL Grades 5-6 panels  Used a Modified Angoff procedure to calculate cut scores in a 2-day workshop Phase 2  Representatives from Phase 1 and state and local policy makers reviewed results  Examined consistency across grades and subjects

5 The Angoff Procedure Research-based procedure used since the early 1970s The most commonly used standard setting method — used in many other state testing programs as well as on certification tests Has undergone many modifications over the years and is often referred to as the Modified Angoff or Extended Angoff procedure

6 Angoff Procedure (con’t.) Original Angoff procedure asked panelists to examine each item on a test and determine whether a student who was just barely Proficient would answer it correctly A common modification is to ask “If we had 100 barely Proficient students in the room, how many of them would answer this item correctly?”

7 The Angoff Task Review each item For MC items, determine how many out of 100 just barely Proficient and just barely Advanced Proficient students should answer each item correctly For OE items, determine the average score a just barely student at each performance level should receive

8 Validity Considerations The process was well-planned – see standard setting proposal. All policy, operational and technical decisions were reasonable and well- documented. The judges were satisfied with the process and its results – see the evaluation results.

9 The Final Decision on Cut Scores – Reminder Legitimate authority should have all the information they want or need to make an informed decision At a minimum it should include:  Overall results – Phase 1 and Phase 2  Variance in judgments and in measurement  Impact on subpopulations in the state  Impact on accountability  Documentation of the process (with as much detail as is necessary)

10 Ratings Cut scores are determined over three rounds of ratings Ratings were made independently After each set of ratings, panelists saw their cut scores and the average cut scores There was a discussion between each round

11 Calculating the Cut Score Consider the MC ratings as probabilities We sum the probabilities to obtain a cut score for the MC items and add the average scores for the OE items to get the total cut score ItemRatingPossible Total= = 3.3 points out of a possible 6 points

12 “Just Barely” Performance The idea of just barely performance is key to the Angoff procedure We spent time talking about what it means to be just barely Proficient and Advanced Proficient Our discussions were driven by the NJ ASK LAL and Math Grades 5-7 Performance Level Descriptors (PLDs)

13 Phase 2 Meeting Policymakers and stakeholders gathered to review the results of Phase 1 in light of NJ ASK program goals. Reviewed results from Phase 1 along with standard deviation of judgments and impact data. Compared results of Grades 3, 4 & 8 LAL and Math with Grades 5-7 LAL and Math and discussed the need for consistency across grades and content areas.

14 Phase 2 Results: Final Cut Score Recommendations Committee produced cut score recommendations for NJDOE and the State Board:

15 Evidence the Results Were Technically Sound Survey Results (see handout) Standard deviation of judgments in each round:

16 Distribution of Students by Subgroup LAL Grade 5

17 Distribution of Students by Subgroup LAL Grade 6

18 Distribution of Students by Subgroup LAL Grade 7

19 Distribution of Students by Subgroup Math Grade 5

20 Distribution of Students by Subgroup Math Grade 6

21 Distribution of Students by Subgroup Math Grade 7

22 Final Recommended Cutscores LAL Grades 5-7 (with LAL Grades 3,4 &8)

23 Final Recommended Cutscores Math Grades 5-7 (with Math Grades 3,4 &8)

24 Key Statistics from Surveys