1 New England Common Assessment Program (NECAP) Setting Performance Standards

2 Purpose Provide data to establish the following three cut scores for Reading, Grades 3-8 and Writing, Grades 5 and 8: –Proficient/Proficient with Distinction –Partially Proficient/Proficient –Substantially Below Proficient/Partially Proficient

3 What is Standard Setting? Set of activities that result in the determination of threshold or cut scores on an assessment We are trying to answer the question: –How much is enough?

4 What is Standard Setting Data collection phase Policy/Decision making phase

5 Many Standard Setting Methods Angoff Body of Work Bookmark

6 Choice of Method is Based on Many Factors Prior usage/history Recommendation/requirement by some policy making authority Type of assessment

7 Choice of Method is Based on Many Factors Weighing all these factors, it was determined that the methods to be used are: –Reading: Bookmark –Writing: Body of Work They are both well-established procedures that have been successfully used on many assessments They have produced defensible results

8 Choice of Method is Based on Many Factors Bookmark is appropriate for assessments that consist primarily of multiple-choice items but also include some constructed- response items Body of Work method works well for assessments that consist primarily of constructed-response items

9 Standard Setting vs. Standards Validation Standard setting –Generally three rounds: Round 1: individual ratings Round 2: table group discussion of Round 1 results, followed by second round ratings Round 3: whole room discussion of Round 2 results, followed by final ratings

10 Standard Setting vs. Standards Validation Standards validation –Starting cut points are provided to panelists at the beginning of the process –Two rounds: Round 1: table group discussion of starting cut points, followed by first round ratings Round 2: whole group discussion of Round 1 results, followed by second round ratings The process you will be following will be closer to a standards validation process

11 Details for Standards Validation using the Bookmark and Body of Work Procedures

12 Starting Cut Points Starting cuts for Reading and Writing were determined using teacher judgment data: –At the time of testing, teachers were asked to categorize each student into an achievement level category, based on classroom performance (not test performance) –Using these categorizations as well as the students’ test scores, Measured Progress staff calculated starting cut points

13 Starting Cut Points Analyses of teacher judgment data indicate: –Good participation rates –Teachers appear to have been conscientious in completing the task In other words, available evidence supports the validity of the starting cuts

14 Starting Cut Points However, standard setting is also a critical part of the process because: –the teachers based their judgments on classroom performance, not performance on NECAP –different teachers have different conceptions of what it means to be, for example, Partially Proficient –they did not have the benefit of discussion with their colleagues from across the three states –etc.

15 Starting Cut Points Both sources of information are important. Therefore: –You are free to recommend changing the starting cuts, but we don’t expect to see major changes –The changes you recommend should be based on the test content and the Achievement Level Descriptions

16 The Bookmark Procedure (Reading) The Bookmark procedure is a standard setting method that uses a book of items (ordered from easiest to hardest) Panelists place bookmarks in that book of items

17 W hat is the bookmark procedure?

18 What is the bookmark procedure? For purposes of NECAP standard setting, you will be provided with starting cut points and will either validate those cuts, or recommend modifications to them

19 A Technical Detail regarding the Bookmark Method Note that the ordered item cut point for a given cut does not equal the raw score a student must obtain to be categorized into the higher achievement level For example, the Substantially Below Proficient/ Partially Proficient starting cut for grade 3 reading is between ordered items 6 and 7; however, a student must obtain at least 20 points on the test in order to be classified into the Partially Proficient level

20 How to Place a Bookmark A few concepts you will need to know: –The starting cut points –The achievement level descriptions –‘Borderline’ students –What knowledge, skills, and abilities (KSAs) are needed to answer each question

21 How to Place a Bookmark Start at the beginning of the ordered item book Evaluate whether at least 2 out of 3 students demonstrating skills at the ‘borderline’ of Partially Proficient would correctly answer item 1 Moving through the book, make this evaluation of each item The bookmark should go where you no longer think 2 out of 3 Partially Proficient ‘borderline’ students would correctly answer the question. You may decide that the starting cuts are in the right place, or you might recommend moving them.

22 How to Place a Bookmark Item Number Would at least 2 out of 3 students who demonstrate skills at the Partially Proficient 'borderline' correctly answer this question? 1Yes No 10No 11No 12No 13No 14No 15No …

23 How to Place a Bookmark In the example, the bookmark would go between items 8 and 9 However, it won’t be that easy; there will be gray areas Place one bookmark for each cut score You will have the opportunity to discuss your bookmark placements and change them if desired

24 How to Place a Bookmark To place your bookmarks you will need to be familiar with the achievement level descriptions and the assessment items

25 How to Place a Bookmark Don’t worry, we have procedures, materials and staff to assist you in this process.

26 Any questions about the Bookmark Procedure?

27 The Body of Work Procedure (Writing) A standard method that uses approximately 30 complete sets of student work, including their answers to open-ended items and a display of their responses to the multiple- choice items Bodies of work cover the range of possible scores and are presented in order from lowest to highest total score

28 What is the Body of Work procedure? You will classify each BOW into the achievement level in which you feel it belongs: –Proficient with Distinction –Proficient –Partially Proficient –Substantially Below Proficient

29 The Body of Work Procedure Prior to beginning the process of rating the BOWs, you will: –take the assessment –become very familiar with the Achievement Level Descriptions –define, with your grade-level group, ‘borderline’ students –individually review the full set of BOWs

30 Making Your Ratings Beginning with the first BOW, and referring to the Achievement Level Descriptions and the definition of ‘borderline’ students, determine into which achievement level the BOW should be categorized Continue through each BOW in turn, assigning each to an achievement level You may agree with the categorizations based on the starting cuts, or you may recommend changing some of them

31 Any questions about the Body of Work Procedure?

32 What Next? After this session, you will break into grade-level groups, where you will: –take the assessment to familiarize yourself with the test items –Reading groups: complete the item map, which is a document that will help you with the bookmark placement process –discuss the Achievement Level Descriptions and develop definitions of “borderline” Partially Proficient, Proficient, and Proficient with Distinction students

33 What Next? You will: –discuss the starting cut points in table groups and do the first round of ratings –discuss the first round ratings as a whole room then do the second round of ratings

34 What Next? After the second round of ratings, you will have an opportunity to provide feedback about the Achievement Level Descriptions As the final step, we will ask you to complete an evaluation of the standard setting process

35 Good Luck!