1. Cutting scores Using tests to improve decisions: Cutting scores & base rates

2. Cutting scores What is a cutting score or cutting line? How shall we evaluate how good any given test is?

3. Cutting scores Test scores Criterion Positive prediction: “This person will succeed.” Negative prediction: “This person will fail.” Low scoreHigh score Definitely Bad Definitely good Acceptable

4. Cutting scores Test scores Criterion Positive prediction: “This person will succeed.” Negative prediction: “This person will fail.” Low scoreHigh score Definitely Bad Definitely good Acceptable False positive: Incorrectly diagnosed. False negative: Incorrectly undiagnosed Cutting line

5. Cutting scores Test scores Criterion Positive prediction: “This person will succeed.” Negative prediction: “This person will fail.” Low scoreHigh score Definitely Bad Definitely good Acceptable Low false positive rate High false negative rate Cutting line Rewarding incompetence

6. Cutting scores Test scores Criterion Positive prediction: “This person will succeed.” Negative prediction: “This person will fail.” Low scoreHigh score Definitely Bad Definitely good Acceptable High false positive rate Low false negative rate Cutting line Ignoring competence

7. Cutting scores How shall we evaluate how good a test is? Three things need to be taken into account: i.) The size of the correlation between test scores and criterion - The higher the correlation, the narrower the scatterplot (i.e. the ellipse) and the smaller the error rates

8. Cutting scores How shall we evaluate how good a test is? Three things need to be taken into account: ii.) The base rate iii.) The cutting score What is the relation between these two measures?

9. Cutting scores The relation between base rate and cutting score Example from Meehl: –Group A: 415 well-adjusted soldiers –Group B: 89 mal-adjusted soldiers –A scale diagnosed 55% of Group B, and only 19% of Group A, so the authors advocated its use

10. Cutting scores Example: Assume N = 10,000 500 are bad. 55% (275) are classified as bad 9500 are good. 81% (7695) are not classified as bad. (7695 + 275)/10000 = 79.97% are correctly classified. Why should this bother us? We could have correctly classified 95% without using a test!

11. Cutting scores Let’s use Bayes’ Theorem