Paired Comparisons Testing Dimensions

Studying attributes of items in a cultural domain Free listing is a way to define the contents of a domain. Pile sorts and triad tests are ways to measure similarity. But we want to do more. We want to describe the attributes of the relations among the elements of a domain. To evaluate the items on an attribute, we can use several techniques: ratings, rankings, and paired comparisons.

Rating scales vs. paired comparisons Ratings are the most common way to measure attributes in the social sciences How dangerous is [this illness]:   1 2 3 4 5 This produces many tied scores How many points should we allow? An even number or an odd number of choices? In some research, you don’t want to allow people to equivocate Scales are often the best we can do for measuring attributes, but for short lists of items, rank-ordered data is better.

Paired comparisons Paired comparisons produce perfect rank-ordered lists. There are N(N-1)/2 pairs of items, so there are 6 pairs in 4 items:   AB AC AD BC BD CD   Each element occurs N – 1 times and each element is paired with each other element. You can count how many times each element “wins” in a paired comparison of items.

How to do PC’s in Anthopac Produce a person-by-pair matrix. Convert this matrix into a person-by-item matrix where cells indicate the number of times, for each person, each item “won” in its contest. This matrix can be summed by columns to produce a vector, indicating, for each item, over all informants, how many times the item won. Anthropac does all this when it imports paired comparison data.

The method of paired comparisons is an alternative way to get rank orderings of a list of items in a domain. There are n(n–1)/2 pairs of things in a list of things. Suppose 5 colors: red, green, yellow, blue, and brown. The next slide shows the paired comparison test to find an informant’s rank-ordered preference for these five colors.

In each of the following pairs of colors, please circle the one you like best: RED GREEN RED YELLOW RED BLUE RED BROWN GREEN YELLOW GREEN BLUE GREEN BROWN YELLOW BLUE YELLOW BROWN BLUE BROWN

Order effects The pairs of colors are listed in such a way that you can see how the 10 of them exhausts the possibilities for five items. For a real test, scramble the order of the pairs to ensure that the order of the items in a list doesn’t influence the choices that informants make. Use Anthropac to do this.

Some more examples of paired comparisons Choose the animal in each pair that is more [vicious, exotic, expensive] Choose the illness in each pair that is more [scary, hard to treat, life threatening] Choose the food in each pair that is [better for you, harder to find, best for children] Choose the crime in pair that [you’re most afraid of, deserves more punishment, harder to stop]

How Anthropac calculates the rank order in paired comparisons For each informant, count up how many times each item in a list ‘‘wins’’—that is, how many times it was circled. For a list of illnesses you expect cancer to win over most other illnesses. It gets interesting when compare the average rank ordering across ethnic groups of, say, high blood pressure and diabetes.

Plusses and minuses of paired comparisons Plus: People make one judgment at a time -- easier than rank ordering a list of items by staring at all the items at once. Plus: You can read a list of pairs to nonliterate informants. Minus: With 20 items, informants make separate 190 judgments. With 60 pile-sort cards, informants make 1770 judgments – but it’s all in their heads.