Bias & Fairness in Tests (Rust & Golombok) Important to note: serious consequences follow from test results! Think about it from the client’s perspective Selection test: You don’t get the job Academic test: You lose a year’s work Clinical tests: get/don’t get help Important to make the right decisions based on the results

Bias and fairness How “correct” our decisions are can be thought of in terms of two properties: Fairness - The social justice issues surrounding the employment of the test Bias - A statistical artefact in the test which makes it respond differently to different groups

Fairness in tests This is not so much a flaw the test as of the use of the test Can only consider fairness in terms of the societal norms which apply We can examine the pattern of decisions which have been made based on that test

Fairness and Justice Depending on the focus on authority in a society, tests might be applied more or less strictly An “unfair test” is one whose consequences do not match the value system of the society

Fairness: an example Imagine we set a super hard exam for 206F Two thirds of the class fails the test I can do two things: Accept the marks as they are and see you next year Adjust the test marks to increase the pass rate Which is the fair thing to do?

Fairness: an example If the society’s norms are such the institution is emphasised over theindividual, accepting the results is the fair thing to do If the society’s norms are such that the individual is emphasised over the institution, adjusting the results is the fair thing to do

Bias in tests A bias exists in a test if gives different results for different populations Example: Army Alpha Soldiers almost always scored lower than officers

Bias: good or bad? Good: Bias can be used to identify which population a client belongs to Should he be officer or soldier? Bad: Creates a false impression of difference between groups Foreign language ability can “reduce” intelligence

Bias: Good or Bad? Irrelevant: If the test only really gets used by one population, who cares? Decide on the importance of bias based on the situation Bias is never an issue of “right” or “wrong” - it is a purely statistical concept

Forms of bias Bias can appear in three forms: Item bias Intrinsic test bias Extrinsic test bias We can examine each of these sources of bias separately and address each individually

Item bias The bias exists in individual questions eg. a questions about dollars, quarters and dimes would be biased Linguistic bias (idioms, slang, etc) common interracial bias This type of item is common in IQ tests (!)

Identifying item bias Do item analysis I.e. check out each item of the test separately Identify possible relevant subgroups Work out the “facility value” of each question for each group the proportion of people who get it right

Item bias: example Imagine we have a test with 3 questions We think it might be language biased Look at the groups: native english speakers, others Work out facility value Native: A: 0.68 B: 0.96 C: 0.57 Other: A: 0.59 B: 0.24 C: 0.59 Big difference in item B, so it is biased

Item offensiveness Eg: Shown an engineer and a psychologist “Which is smarter?” Related to item bias offensive items not necessarily biased Offensive items should be removed May interfere with subsequent items

Intrinsic test bias The test has different mean scores for different groups Does not exist in specific questions (Item bias), but rather a general phenomenon Common in language groups It is a matter of degree

Causes of intrinsic bias Tests created with a specific group in mind are biased in this way Other groups perform worse The more different the group, the bigger the difference

Extrinsic test bias When the test is unbiased, but decisions made using the test are biased Eg. a test finds a true difference, and this leads to one group getting selected more than another Extrinsic bias is the overlap between fairness and bias

Extrinsic bias: example On the SAT, poorer children tend to perform worse than richer children This is a real difference - poorer children have less access to the requirements to academic success The SAT was used to select university applicants poorer children were selected far less frequently

Extrinsic bias and ideology Do you believe in “true differences” Or do you rather believe in “unexplored potrential”? It is a fact that poorer kids did worse at university (in the USA) Do we use this as a basis for not selecting them? No way to take a “scientific” ideology-free standpoint on extrinsic bias