1. Partial Credit Scoring for Technology Enhanced Items
CCSSO National Conference on Student Assessment
22 June 2016 1

2. Overview Possible scoring rules for technology enhanced items (TEIs)
Dichotomous
Partial credit scoring (polytomous)
Evaluating whether partial credit scoring is “working”
Study methods & results
Additional considerations
Do TEIs with more correct answers take longer to answer?
Are TEIs with more correct answers more difficult?
Do TEIs take more time to answer compared to multiple-choice items?
Should TEIs be worth more points than multiple-choice items?
How might partial credit scoring impact student subgroups?

3. Possible Scoring Rules for TEIs
Dichotomous
Currently used in Virginia
Students must select all correct answers to receive 1 point; all other responses receive 0 points
Polytomous
Virginia is looking to implement partial credit scoring in the future for TEIs
Students get “full credit” if they select all correct answers
Different methods allow for different ways to determine how partial credit versus no credit is determined
This research looked at three possible methods:
“N Method”
“N-1 Method”
“N/2 Method”

4. Partial Credit Scoring: N Method
N = the total number of correct answers for a particular TEI
Each correct response receives partial credit
For example, if there are 5 correct answers, each correct answer could be worth 1 point for a total of 5 possible points
Items with different numbers of correct answers will have different numbers of score points
Could scale each item to have the same score range (e.g., 0-1), but there will be N+1 score categories (e.g., 0, 1/5, 2/5, 3/5, 4/5, and 1)

5. Partial Credit Scoring: N Method
Benefits:
perceived face validity
might be similar to how credit would be awarded in the classroom for multi-part items
Limitations:
each score point may not really be discriminating between distinct levels of content knowledge
may be difficult to obtain sufficient numbers of students within all score categories
having items with different score ranges may make meeting a test blueprint more challenging

6. Partial Credit Scoring: N-1 Method
N = the total number of correct answers for a particular TEI
“N-1 Method” – Students get partial credit if they select all but one correct answer; if they select two or more incorrect answers, they receive no credit
N-1 Method will have 3 score categories for all items (e.g., 0, 1, 2)
Example: If an item has 5 correct answers and the student selects all 5 correct, they receive full credit. If the student selects 4 of the 5 correct answers, they receive partial credit, and if they get 0-3 correct answers, they receive no credit.

7. Partial Credit Scoring: N-1 Method
Benefits:
all items, regardless of the number of correct answers, have three possible score categories
having the same score range for all TEIs will make meeting a test blueprint more straight-forward
with only three score categories, sparse data is less likely to be a problem, which will make parameter estimation more stable
Limitations:
one-size-fits-all scoring approach may not work well for very different TEI types
tends to be a very conservative partial credit scoring approach such that very few students may obtain partial credit (dichotomous outcome)

8. Partial Credit Scoring: N/2 Method
N = the total number of correct answers for a particular TEI
“N/2 Method” – Students get partial credit if they select at least half of the correct answers; if they select less than half of the correct answers, they receive no credit
N/2 Method will have 3 score categories for all items (e.g., 0, 1, 2)
N/2 Method and N Method are identical for items with 2 correct answers
Example: If an item has 5 correct answers and the student selects all 5 correct, they receive full credit. If the student selects 3 or 4 of the 5 correct answers, they receive partial credit, and if they get 0-2 correct answers, they receive no credit.

9. Partial Credit Scoring: N/2 Method
Benefits:
all items, regardless of the number of correct answers, have three possible score categories
having the same score range for all TEIs will make meeting a test blueprint more straight-forward
with only three score categories, sparse data is less likely to be a problem, which will make parameter estimation more stable
Limitations:
one-size-fits-all scoring approach may not work well for very different TEI types

10. Is PCS “Working”? Evaluation Criteria
Number of Students in Each Score Category
Do we have enough data to estimate differences in difficulty from one category to the next?
Are we making a meaningful differentiation between groups of students?
As item scores increase, is the total test score increasing?
Are item-total correlations increasing?
Are we improving measurement precision, or just adding “noise” into student scores?

11. Methods
Focused on newly field-tested TEIs in one grade of science and one grade of mathematics
Analysis focused on Hot Spot and Drag and Drop TEIs that required two or more correct answers for full credit.
Note: This study used items that were not specifically developed with partial credit scoring in mind. Virginia intends to begin development of TEIs that are created with partial credit scoring considered from the outset moving forward.

12. Results Item Type Number of Students Scoring Approach Dichotomous N/21 2
Hot Spot
5,231
31.27
39.07
26.91
34.00
3,938
26.42
35.99
23.68
26.64
3,904
32.51
35.61
21.83
32.62
Drag-and-Drop
3,939
29.34
35.38
21.13
29.53
3,955
30.62
37.59
30.22
31.08
5,089
25.43
35.72
17.62
26.77
Item Type
Number of Students
Scoring Approach N 1 2 3 4 5
Hot Spot
5,231
24.50
27.40
34.00
39.07
3,938
23.68
26.64
35.99
3,904
13.00 –
22.64
27.76
33.00
35.61
Drag-and-Drop
3,939
25.25
17.00
29.53
35.38
3,955
31.26
23.11
30.71
31.54
37.59
5,089
17.50
17.71
26.42
27.25
35.72

13. Results Item Type Scoring Approach Dichotomous N/2 N Hot Spot 0.42
0.51
0.35
0.17
0.19
Drag-and-Drop
0.20
0.39
0.38
0.34

14. Results N Method N-1 Method N/2 Method
Many score categories with very few students, especially for TEIs with a larger numbers of correct answers
Many cases where the total score did not increase as the item score increased
N-1 Method
Very few students would receive partial credit using this method
N/2 Method
Most consistent with how content experts assigned partial credit scoring to TEIs
Appears to work well with Virginia items in terms of having sufficient numbers of students within each of the three score categories
Did not appear to result in huge improvements in measurement properties (item total correlations), but also did not systematically decrease technical quality of scores

15. Do TEIs with More Correct Answers Take Longer to Answer?

16. Are TEIs with More Correct Answers More Difficult?

17. Do TEIs take More Time than MC Items to Answer?
Subject
TEI MC
Min
Max
Average
Science
28.32
214.73
81.45
58.50
Math
76.49
466.98
185.55
132.29

18. Should TEIs be Worth More Points than MC Items?
Yes?
On average TEIs take more time to answer
TEIs often require more than one student interaction, while MC items require students to select only one answer
TEIs often used to try to measure higher level skills
TEIs look similar to items that might receive more than a 0/1 score in the classroom

19. Should TEIs be Worth More Points than MC Items?
No?
Number of interactions not correlated with the amount of time students spend on an item
TEIs not necessarily more difficult than MC items when scored dichotomously
Often each interaction is not measuring a distinct skill, but is just more thoroughly evaluating a single skill

20. How Might Partial Credit Scoring Impact Student Subgroups?
TEIs make up 15-20% of the items on Virginia assessments
Partial credit scoring essentially splits the category of students who received 0 points under dichotomous scoring into 2 or more score categories—no credit and various degrees of partial credit
This may help lower performing students show some content mastery
However, each type and subtype of TEI often requires understanding of a different mechanism for responding (e.g., various forms of clicking and/or dragging)
This may result in an additional obstacle for English Language Learners and Poor Readers who may struggle to understand item-specific response directions
The Virginia Department of Education provides practice items and released items to give students and schools an opportunity to become familiar with different types of TEIs

21. Take-Aways
Not all technology enhanced items support partial credit scoring (e.g., one blank FIB items)
For items that do support partial credit scoring, determining which responses merit partial credit is not always straight-forward
There are many methods for applying partial credit, some which may work better than others for a given context
Partial credit scoring may not result in vast improvements in measurement precision but may still enhance face validity and public perception