The ABC’s of Pattern Scoring Dr. Cornelia Orr

Slide 2 Vocabulary Measurement – Psychometrics is a type of measurement Classical test theory Item Response Theory – IRT (AKA logistic trait theory) 1, 2, & 3-parameter IRT models Pattern Scoring

Slide 3 General & Specialized Measurement Assign numbers to objects or events Ex. – hurricanes, earthquakes, time, stock market, height, weight Psychometrics Assigning numbers to psychological characteristics Ex. – achievement personality, IQ, opinion, interests

Slide 4 Different Theories of Psychometrics Classical Test Theory Item discrimination values Item difficulty values (p-values) Guessing (penalty) Number correct scoring Item Response Theory a)Item discrimination values b)Item difficulty values c)Guessing (pseudo- guessing) values Pattern scoring Similar constructs – Different derivations

Slide 5 Different Methods of Scoring Number-Correct Scoring Simple Mathematics Raw scores (# of points) –Mean, SD, SEM, % correct Number right scale Score conversions –Scale scores, percentile ranks, etc. Pattern Scoring Complex Mathematics Maximum likelihood estimates –Item statistics, student’s answer pattern, SEM Theta scale (mean=0, standard dev=1) Score conversions –Scale scores, percentile ranks, etc.

Slide 6 Comparison: Number Correct and Pattern Scoring Similarities The relationship of derived scores is the same, e.g., –High correlation, (0.95) of number right scores and scale scores –Scale score has the same percentile rank for both methods Differences Methods of deriving scores The number of scale scores possible –Number right = limited to the number of items –IRT = unlimited or is limited by the scale (ex )

Slide 7 Choosing the Scoring Method Which model? Simple vs. Complex? Best estimates? Advantages/Disadvantages? Ex. – Why do the same number correct get different scale scores? Ex. – Flat screen TV – how do they do that?

Slide 8 Advantages of IRT and Pattern Scoring Better estimates of an examinee’s ability –the score that is most likely, given the student’s responses to the questions on the test (maximum likelihood scoring) More information about students and items are used More reliability than number right scoring Less measurement error (SEM)

Slide 9 Disadvantages of IRT and Pattern Scoring Technical - Complex Mathematics – –Difficult to understand –Difficult to explain Not common – Not like my experience. Perceived as “Hocus Pocus”

Slide 10 Item Characteristic Curve (ICC)

Slide 11 Examples Effect Of Item Difficulty No Type a b c 1 1 MC MC MC MC MC Response Patterns (1=correct) Pattern SEM SS Answering more difficult items (b-parameter) can result in higher scores.

Slide 12 Examples 5 Items (Effects of Item Discrimination) No Type a b c 1 MC MC MC MC MC Response patterns (1=correct) Pattern SEM SS Answering more discriminating items (a-parameter) can result in higher scores.