Lab 5: Item Analyses

Quick Notes Load the files for Lab 5 from course website – Lab 5.ppt Item.proj.xls

Item Analyses – Introduction We perform item analyses to evaluate individual items. The basic idea is to compute a series of specific descriptive statistics for each item and then evaluate those statistics.

When Can You Do Item Analyses? When responses to items are scored as correct or incorrect. Achievement tests Exams Written driving tests NOT on most attitude or personality measures. Why?

Key Aspects of an Item Analysis Item Difficulty Item Discrimination – 2 Indices

Item Difficulty 1 Item responses are recoded: – Wrong answers are coded as 0 – Right answers are coded as 1 The mean of each recoded item is the percentage of people who got the item correct. It ranges from 0 (no one got the item right) to 1 (everybody got it right).

Example Imagine that you ask 100 people a question. 56 people give the right answer and 44 people give the wrong answer. What is the item mean?

Item Difficulty 2 Traditional method of calculating item difficulty: p p = proportion of people who got the answer correct on an item. – Ranges from 0.00 to 1.00 – The higher the value, the easier the item (i.e., more people got it correct) p =.77 means that 77% of individuals got the item right

An Easy Item What is the name of the daughter of Tom Cruise and Katie Holmes? a) Samantha b) Suri c) Apple d) Sunshine e) Nicole

An Easy Item What is the name of the daughter of Tom Cruise and Katie Holmes? a) Samantha b) Suri c) Apple d) Sunshine e) Nicole Item difficulty is: p =.91

A Difficult Item What TV show won the 2006 Emmy for the best dramatic series? a) Prison Break b) 24 c) The Sopranos d) Lost e) CSI Las Vegas

A Difficult Item What TV show won the 2006 Emmy for the best dramatic series? a) Prison Break b) 24 c) The Sopranos d) Lost e) CSI Las Vegas Item difficulty is: p =.42

Summary: Item Difficulty When few people get an item correct, the item is difficult and p is low. When a lot of people get an item correct, the item is easy and p is high. In general test construction, you want items to range in difficulty. Average item difficulty on the test equals the mean test score. – If you want the mean of your test to be 75%, average of item difficulties should be 75%.

Item Difficulty Analysis for Exam 1 Look at the item analysis file for the exam. Where can you find the item difficulty for each item? Which item is the easiest? Which is the most difficult? What is the average item difficulty?

Item Discrimination An index of how well an item differentiates between people who did well on the test (and presumably knew the material) and those who did not do well (and presumably did not know the material). 2 Methods: – Method 1: Take p for upper range of people (e.g., top 25%) and subtract p for lower range of people (e.g., bottom 25%) on an item – Method 2: item-total correlation correlate item score and test score

Item Discrimination Item discrimination values should always be positive. The higher the value, the better the item predicts how well people did on the test.

Item Discrimination for Exam 1 Look at the Exam 1 item analysis. Where can you find the item discrimination for each item? Which item discriminates best? Which worst?

Relation between Item Difficulty and Item Discrimination I Consider a very easy item, e.g. item 44 What percentage of the upper 25% of people on the test got it right? What percentage of the lower 25%?  This item does not discriminate well between those who did well on the test and those who did not. This is called a ceiling effect.

Relation between Item Difficulty and Item Discrimination Consider a moderately difficult item, e.g., item 15 What percentage of the upper 25% of people on the test got it right? What percentage of the lower 25%? Items with moderate difficulty tend to have higher discrimination. If a person can get an item right or wrong, I can make a judgment about whether the person has high or low ability relative to others.

Homework I l Report and interpret the item difficulty for items 1 – 5. (2 points) l Which item in the first exam is the most difficult? Which is the easiest? (1 point) l Report and interpret both types of item discrimination for items 1 – 5. (2 points) l Which item has the highest discrimination? Which has the lowest? (1 point) 

Homework II 5. Imagine the scoring office gives you these statistics for four different items. For each item, report whether – you would keep the item on the test or – revise the item in future versions of the test and explain your decisions! (4 points) Item 1: p = 0.95; discrimination index = 0 Item 2: p = 0.1; discrimination index = 0.40 Item 3 p = 0.1; discrimination index = 0 Item 4 p = 0.5; discrimination index = 0.60

Your Paper

Picking a Topic for Your Paper The most important first step is finding a topic in psychology that interests you. You can choose a topic from any area of psychology (e.g. biological, cognitive, clinical, community, developmental, educational, personality, social). The question must be one that can be addressed with empirical research. You must have at least 2 variables of interest (e.g., 1 dependent and 1 independent variable) Write down 3 possible research topics and indicate why you are interested in these questions. You will discuss these in lab next week.