1. Field Test Analysis Report: SAS Macro and Item/Distractor/DIF Analyses
Prepared by
Yi-Hsin Chen, Chunhua Cao, and Stephanie Green
College of Education at USF
Presented at the meeting of
the Central Florida Assessment Collaborative (CFAC)
May 20th, 2014, Orlando Florida

2. Agenda of This Presentation
SAS macro for CTT test/item analysis, IRT 2PL model, and Mantel-Haenszel differential item functioning (DIF) analysis
Introduction of statistical concepts for test/item development
Item Analyses: CTT and IRT
Distractor Analysis
DIF Analysis

3. SAS macro for test/item, 2PL, DIF analyses

4. SAS Macro Outputs A SAS macro developed for this project
There are six excel outputs
Test score statistics
Frequencies of options for each item
Item analysis statistics
Distractor analysis
DIF
2PL item parameter
Available upon request at

5. Test Score Statistics

6. Frequencies of Options

7. Item Analysis Statistics

8. Item Analysis Statistics

9. Distractor Analysis

10. DIF Analysis

11. Statistical Concepts of Test Scores

12. Sample size N: Sample size 85, 60, 70, 44, 59, 89, 99, 79, . , 100
USED_N: Sample size used for analysis without missing data
one missing data
USED_N = 9

13. Central Tendency MEAN: Arithmetic average
Most frequently reported measure of central tendency
Sum of scores divided by number of scores

14. Test Statistics: Central Tendency
MEDIAN (Q2): the score at the 50th percentile
half of the examinees score above median, and half score below median
110
105
100 95 90
Median = / 2 = 97.5
110
105
100 95 90
Median = 100

15. Percentiles
Percentile is considered when we consider the percentage of scores that fall below a given point
They are very useful for interpreting an individual student’s performance
Q1: The score is at the 25th percentile
Q1 = 10, indicating 25 percent of the students’ scores below 10 points
Q3: The score is at the 75th percentile

16. Variability
Range
Subtract lowest score (Minimum) from highest score (Maximum)
This is a rough measure of variability
High score = 90
Low score = 50
Range = ? (40)
High score = 100
Low score = 50
Range = ? (50)
High score = 90
Low score = 30
Range = ? (60)

17. Variability Standard Deviation (SD):
an average points that deviates from the mean score
A measure of the amount of variability in examinees’ total scores
Large SD = large variability (heterogeneity)
Small SD = small variability (homogeneity) (scores cluster closer to the mean)

18. Variability Deviation Scores Squared 100-92= 8 82 = 64
100-92= = 64
= = 16
= = 4
= = 0
= -2 (-2)2 = 4
= -12 (-12)2 = 144
232 =  (X-Mean)2
SD =  (X-Mean) = = N
Scores
100 96 94 92 90 80
Mean = 92
6.22

19. Skewness and Kurtosis SKEWNESS:
a measure to tell the shape of the score distribution, such as positive or negative skewness or symmetry
KURTOSIS:
a measure of the "peakedness" of the score distribution

20. Skewness

21. Skewness
a roughly negatively skewed distribution (bar chart)

22. Skewness

23. Skewness
a roughly positively skewed distribution (bar chart)

24. Kurtosis
Different kurtosis values
K > 0
K = 0
K < 0

25. Reliability: Cronbach’s Alpha
A measure of the test reliability, indicating the internal consistency of the test
Sample dependent
Different samples may obtain different reliability with the same test
Ranges from 0 to 1
0.7 and above: good internal consistency

26. Standard Error of Measurement
SEM (Standard Error of Measurement)
SEM = STD * 1−𝑟𝑒𝑙𝑖𝑎𝑏𝑖𝑙𝑖𝑡𝑦
A higher reliable test can cause smaller SEM

27. Statistical Concepts of Item Analysis

28. Item Analysis Why care? –
Item analysis helps you identify problems with your items (or scoring)
These problems can be corrected, resulting in a better test, and better measurement

29. Item Analysis When is it useful? –
Item analysis is most useful when you are developing a bank, or pool, of items that you will continue to use
It can be used when evaluating standardized tests
It is also a useful tool, anytime students have complained about an item
It can be used to identify mis-keyed items

30. Item Difficulty (p-value)
Item difficulty (proportion correct):
the proportion of examinees tested that answered the item correctly
# of students who responded correctly
total # of students who responded
p =
Ncorrect
Ntotal
p =

31. Item Difficulty (p-value)
p can range from 0 to 1.0
A rough level of item difficulty (p)
.80 and above moderately easy to very easy (mastery)
moderate
.30 and below moderately difficult to very difficult

32. Item Discrimination Discrimination can be computed using correlation
This shows the relationship between a single item and the total test
It is expected that students with high scores answer the item correctly
rpb = (point-biserial) correlation
between item score and total score

33. Item Discrimination Corrected point-biserial correlation:
A statistic similar to point-biserial correlations
The score of the individual item is taken out of the total score so that the contribution of the item itself is removed from the correlation
This statistic is more accurate to represent item discrimination

34. Item Discrimination Two ability groups (upper and lower) approach
Median score is used to divide the students into two groups
Discrimination coefficient (D-value) = percentage correct in the upper group – percentage correct in the lower group
Ranges from -1 to 1
An item with higher and positive D-value indicates a good discriminating item
An item with a negative D-value suggests that the lower achieving group did better on an item than the higher achieving group, indicating a poor item

35. Item Discrimination A rough scale of item discrimination (D)
D can range from -1 to 1
.30 and above moderate to high discrimination
little to no discrimination
0 and below negative discrimination (unwanted)

36. Item Difficulty and Discrimination
Relationship between item difficulty and discrimination
there can be little discrimination:
if nearly everyone gets the item right, or
if nearly everyone gets the item wrong
there can be maximum discrimination:
if about half the people got the item right, and about half got the item wrong

37. Item Difficulty and Discrimination
Item Difficulty
Max Discrimination
Relationship between item difficulty and potential discrimination

38. Alpha If an Item Deleted
“The Alpha If Deleted” shows what would happen to the internal consistency when the item is deleted
When the test_alpha_deleted coefficient goes up, compared with the original test-alpha, it indicates that without the deleted item, the test can be more reliable (that item can be removed from the test)
When the test_alpha_deleted coefficient goes down, it means that deleting that item is not a good thing and also indicates that item is a good item

39. Statistical Concepts of Distractor Analysis

40. Distractor Analysis
used to determine which distractors students find attractive
consider the proportion of (total) students choosing each option
compare the number of examinees selecting each option in the High and Low groups, or
Example:
Proportion of total examinees selecting each option a* b c d
Total
.78
.11
.03
.08

41. Selecting upper and lower groups
Upper and Lower groups are needed:
to hand-compute D-values, and
for distractor analysis when comparing numbers of examinees
To select Upper and Lower groups:
arrange the tests by total score
separate out the tests for each group
top half becomes Upper group, and
bottom half becomes Lower group

42. Selecting upper and lower groups
Upper and Lower groups are needed:
to hand-compute D-values, and
for distractor analysis when comparing number of examinees
To select Upper and Lower groups:
Upper group: top half (50%) or top 33%
Lower group: bottom half (50%) or bottom 33%

43. Example 1: distractor analysis
1. The capital of Switzerland is
Bern.
Zurich.
Lucerne.
Geneva.
Numbers in the High and Low groups who selected each option a* b c d
Upper 13 1
Lower 3 2 9

44. Example 2: distractor analysis
2. The most important part of test planning is creating:
sound instruction.
a test blueprint.
an item analysis plan.
the grading curve.
Numbers in the High and Low groups who selected each option a b* c d
Upper 1 8
Lower 2

45. Example 3: distractor analysis
3. Which type of essay item contains the most explicit instructions to students?
extended response
fixed response
explicit response
restricted response
Numbers in the High and Low groups who selected each option a b c* d
Upper 3 1 2 14
Lower 4 7 8

46. Statistical Concepts of 2PL IRT model Analysis

47. Two-Parameter Logistic Model
Alpha represents item discrimination
The value is positive
Beta represents item difficulty with the mean of 0 and the SD of 1
Items with the negative values = easy items
Items with the positive values = hard items

48. Statistical Concepts of DIF Analysis

49. Differential Item Functioning
A major concern regarding using the psychological measures is that these measures may “work differently” or be either “for or against” a particular group of examinees (e.g., gender or ethnicity)
When a test item unfairly favors one group over another, it can be said to show differential item functioning or DIF

50. Uniform or consistent DIF

51. Non-uniform or crossing DIF

52. Mantel Haenszel chi-square1
Total
Reference Bt At
NRt
Focal Dt Ct
NFt
M0t
M1t Tt
subscript t = individual raw score

53. Mantel Haenszel chi-square
Controlling for the observed score, we want to see if the proportion correct for the focal group is equal to that for the reference group on an item
The MH statistic consists of a series of 2x2 contingency tables
MH = 1 : No DIF
MH < 1: DIF and favor the focal group (dummy=0) if p < .05
MH > 1: DIF and favor the reference group (dummy=1) if p < .05

54. Field Test Analyses

55. Test Statistics for Three Subjects
Precalculus N
210
USED_N
MEAN
9.748
STD
2.978
MIN 2 Q1 8
MEDIAN 10 Q3 11
MAX 20
SKEWNESS
0.378
KURTOSIS
0.679
ALPHA
0.506
SEM
2.093
STATISTIC
Anatomy N
269
USED_N
MEAN
12.364
STD
3.337
MIN 4 Q1 10
MEDIAN 12 Q3 15
MAX 21
SKEWNESS
-0.102
KURTOSIS
-0.447
ALPHA
0.533
SEM
2.281
STATISTIC
Phy-Sci N
183
USED_N
MEAN
12.852
STD
4.141
MIN 4 Q1 10
MEDIAN 13 Q3 16
MAX 25
SKEWNESS
0.088
KURTOSIS
-0.658
ALPHA
0.626
SEM
2.531

56. Item Difficulty Item difficulty Pre-calculus (21 items) 0-0.1
(2 Items) 19, 3
(1 Item) 14, 1
(14 Items) 21, 10, 11, 18, 12, 20, 8, 16, 17, 15, 14, 6, 13, 2
(3 Items) 7, 5, 9
Item difficulty
Anatomy
(27 items)
0-0.1
(0 items)
(2 items) 13, 2
(6 items) 16, 8, 27, 3, 10, 20
(14 items) 17, 4, 9, 14, 11, 5, 18, 26, 25, 7, 15, 22, 21, 24
(3 items) 12, 19, 1
(2 items) 23, 6
Item difficulty
Physical Science
(31 items)
0-0.10
(1 item) 22
(2 items) 11, 28
(5 items) 16, 27, 9, 6, 20
(12 items) 30, 18, 12, 25, 31, 15, 2, 19, 24, 13, 26, 29, 21, 10, 23, 8, 7, 4, 3, 17, 5, 14
(1 item) 1
0 items

57. Item Discrimination (Corrected point-biserial correlation)
Value
Pre-calculus
21 items
Negative Value
(1 Item) 19
0-0.10
(2 Items) 11, 3
(13 Items) 9, 10, 2, 18, 8, 5, 17, 21, 1, 13, 14, 4, 12
(5 Items) 16, 15, 7, 6, 20
Above 0.30
(0 Items)
Value
Pre-calculus
27 items
Negative Value
(3 items) 3, 17, 13
0-0.10
(6 items) 16, 9, 20, 10, 27, 2
(9 items) 15, 4, 11, 26, 18, 12, 5, 14, 8
(9 items) 7, 1, 25, 22, 23, 21, 24, 6, 19
Above 0.30
0 items
Value
Physical Science
31 items
Negative Value
(6 items) 11, 22, 20, 12, 31, 1
0-0.10
(2 items) 21, 5
(8 items) 23, 19, 6, 28, 25, 18, 10, 16
(6 items) 8, 3, 15, 2, 30, 27
Above 0.30
(9 items) 13, 7, 17, 24, 29, 9, 14, 26, 4

58. Item Discrimination (Two-Group Approach)
Value
Pre-calculus
21 items
Negative Value
(0 Items)
0-0.10
(3 Items) 19, 3, 18
(7 Items) 21, 14, 9, 17, 10, 2, 11
(7 Items) 1, 5, 12, 13, 8, 7, 16
Above 0.30
(4 Items) 15, 20, 4, 6
Value
Anatomy
27 items
Negative Value
(0 items)
0-0.10
(5 items) 13, 3, 16, 17,2
(7 items) 27, 9, 10, 20, 12, 6, 26
(11 items) 8, 4, 11, 1, 15, 18, 23, 15, 5, 7
Above 0.30
(5 items) 25, 24, 19, 22, 21
Value
Number of items
Negative Value
(2 items) 11, 22
0-0.10
(4 items) 20, 1, 12, 31
(8 items) 28, 21, 23, 5, 19, 25, 6, 16
(6 items) 10, 3, 8, 27, 18, 9
Above 0.30
(10 items) 15, 30, 2, 17, 13, 24, 14, 7, 29, 26, 4

59. Alpha Difference (Alpha and Alpha When deleted)
Pre-Calculus
21 items
Negative Value
(2 Items) 19, 11
(3 Items) 3, 9, 10
(5 Items) 2, 8, 18, 5, 17
Above 0.01
(14 Items) 21, 1, 13, 14, 4, 12, 16, 7, 15, 6, 20
Alpha Difference
Anatomy
27 items
Negative Value
(7 items) 3, 17, 9, 16, 13, 20, 10
(3 items) 27, 2, 15
(4 items) 4, 11, 26, 18
Above 0.01
(13 items) 12, 5, 14, 8, 1, 7, 25, 23, 6, 22, 21, 19, 24
Alpha Difference
Physical Science
31 items
Negative Value
(8 items) 11, 20, 12, 31, 22, 1, 21, 5
(6 items) 23, 19, 6, 28, 25, 18
(3 items) 10, 16, 8
Above 0.01
(14 items) 3, 15, 2, 27, 30, 13, 7, 17, 24, 9, 29, 14, 26, 4

60. Item Analysis Summary
The test with reliability (alpha) less than .5 needs to be worried
Too hard item (e.g., p-value < 0.1 or 0.2) or/and too easy (e.g., p-value close to 1) items may be revisited
Revisiting Items with a negative value of discrimination is warranted, especially for the two-group item discrimination
Items with negative alpha difference between the original test alpha and the test alpha when deleted are not good, either

61. DIF Results: Precalculus
Girls = 0
Boys = 1
Favor boys

62. DIF Results: Precalculus
Girls = 0
Boys = 1
Favor girls

63. DIF Results: Anatomy
Girls = 0
Boys = 1
Favor boys

64. DIF Results: Anatomy
Girls = 0
Boys = 1
Favor girls

65. DIF Results: Anatomy
Girls = 0
Boys = 1
Favor boys

66. DIF Results: Anatomy
Girls = 0
Boys = 1
Favor girls

67. DIF Results: Anatomy
Girls = 0
Boys = 1
Favor girls

68. DIF Results: Physical Science
Girls = 0
Boys = 1
Favor boys

69. DIF Results: Physical Science
Girls = 0
Boys = 1
Favor boys

70. DIF Results: Physical Science
Girls = 0
Boys = 1
Favor girls

71. Distractor Analysis: Typical Problems and Solutions

72. Precalculus: Item 29 Frequency Row Pct Table of groupB by r19 groupBD
Total
LOWER GROUP
5 9.80 51
UPPER GROUP
8 7.02
114 61 53 38 13
165
Frequency Missing = 76

73. Precalculus: Item 29
The item is a hard item (p = 0.18)

74. Precalculus: Item 3 Frequency Row Pct Table of groupB by r3 groupB r3
Table of groupB by r3
groupB r3 A B C* D
Total
LOWER GROUP
4 6.45 62
UPPER GROUP
6 4.55
132 60 85 39 10
194
Frequency Missing = 47
Add items 3 and 1 for worse items
Add item 21 as a good item with high discrimination and high difficulty
Add two items with moderate difficulty and high discrimination
The item is a hard item (p = 0.162)

75. Precalculus: Item 3 The item is a hard item (p = 0.19)
Add items 3 and 1 for worse items
Add item 21 as a good item with high discrimination and high difficulty
Add two items with moderate difficulty and high discrimination
The item is a hard item (p = 0.19)

76. Precalculus: Item 1 Frequency Row Pct Table of groupB by r1 groupB r1
Table of groupB by r1
groupB r1 A B C D*
Total
LOWER GROUP
5 8.62 58
UPPER GROUP
8 6.20
129 37 76 13 61
187
Frequency Missing = 54
The item is a hard item (p = 0.253)

77. Precalculus: Item 1
The item is a hard item (p = 0.30)

78. Precalculus: Item 14 Table of groupB by r14 groupB r14 - A B C D*
Total
LOWER GROUP
5 5.56
8 8.89
3 3.33 90
UPPER GROUP
5 4.55
4 3.64
110 10 12 7
116 55
200
Frequency Missing = 10

79. Precalculus: Item 14 The item is challenging (p = 0.26)
Option C may be the potential key
Or students have a misconception on this item

80. Precalculus: Good Item
Table of groupB by r21
groupB
r21 - A B* C D
Total
LOWER GROUP 99
UPPER GROUP
111 56 34 66 28 26
210
The item is challenging (p = 0.266)
Discriminating well

81. Precalculus: Good Item
The item is challenging (p = 0.31)
Discriminating well
However, this item shows DIF and favors girls

82. Summary for Precalculus
Some items need to revisit:
Items: 19, 3, 1, and 14
Develop some easy items (p= )
Two DIF items
Items 4 and 21

83. Anatomy: Hard Item The item is a hard item (p = 0.271)
Table of groupB by r3
groupB r3 A B C* D
Total
LOWER GROUP
108
UPPER GROUP
7 4.38
160 27 52 73
116
268
Frequency Missing = 1
The item is a hard item (p = 0.271)
Not discriminating well

84. Anatomy: Hard Item The item is a hard item (p = 0.271)
Not discriminating well

85. Anatomy: Potential Miskey
Table of groupB by r16
groupB
r16 A B C D*
Total
LOWER GROUP
108
UPPER GROUP
160
144 43 22 59
268
Frequency Missing = 1
The item may have a miskey of Option D
The possible correct key is Option A (Majority of the upper group chose this option)

86. Anatomy: Potential Miskey
The item may have a miskey of Option D
The possible correct key is Option A (Majority of the upper group chose this option)
Or there is a misconception on this item

87. Anatomy: Good Item The item has moderate difficulty level(p = 0.491)
Table of groupB by r25
groupB
r25 A B C D*
Total
LOWER GROUP
109
UPPER GROUP
8 5.03
159 21 52 63
132
268
Frequency Missing = 1
The item has moderate difficulty level(p = 0.491)
Discriminating well

88. Anatomy: Good Item The item has moderate difficulty level(p = 0.491)
Discriminating well

89. Summary for Anatomy
The p-value of the items look good, with half of the items being moderate difficult, almost one quarter of them being easy, and almost one quarter being difficulty
No negative discrimination items using the two-group approach (a good sign)
The test alpha is low (0.533)
DIF: Items 14, 19 (favoring boys) and items15, 22, 26 (favoring girls)

90. Physical Science: Item too hard
Table of groupB by r28
groupB
r28 - A B C D*
Total
LOWER GROUP
2 2.33 86
UPPER GROUP
4 4.12
8 8.25 97 22 23 81 27 30
183
The item is a hard item (p = 0.164)

91. Physical Science: Item too hard
The item is a hard item (p = 0.164)

92. Physical Science: Potential Miskey
Table of groupB by r11
groupB
r11 A B C* D
Total
LOWER GROUP
8 9.30 86
UPPER GROUP
9 9.28 97
101 37 28 17
183
The item may have a miskey of Option C
The possible correct key is Option A (Majority of the upper group chose this option)

93. Physical Science: Potential Miskey

94. Physical Science: Good Item
Table of groupB by r27
groupB
r27 A B C D*
Total
LOWER GROUP 86
UPPER GROUP
9 9.28 97 22 49 67 45
183
The item has moderate difficulty level(p = 0.491)
Discriminating well

95. Physical Science: Good Item

96. Summary for Physical Science
Some items need to revisit:
Items: 6, 11, 12, 22
Potential miskey item: 11
Develop some easy items (p= )
DIF: Items 3 and 4 (favoring boys) and Item 7 (favoring girls)