Chapter 8 Making Judgments and Decisions Intuitive Predictions Judgment Aids Utility Analysis chapter 8 Makeing Judgments and Decisions

chapter 8 Makeing Judgments and Decisions JUDGMENTS OF VALIDITY Evaluating a Measure Without statistical validation – judge job relatedness based on sequential questions: Is trait related to performance (link KSA -> task)? is the construct a valid measure of the KSA? Give an example 12 questions to ask Be prepared to answer the questions for a measure you may use chapter 8 Makeing Judgments and Decisions

12 Questions to ask….to evaluate a measure Clear idea of the attribute (construct) measured? Was the measurement method consistent with the construct? Was the stimulus content appropriate? Unambiguous? Relevant to the measure? Was the development done skillfully with a pilot study? Single attribute or heterogeneous domain? Are scores stable over time? How do you know? Do the scores show convergent validity? Do the relationships show divergent validity? Does the attribute relate to job performance? Does the predictive hypothesis require other attributes equally important? Could there be a non-monotonic relationship? Are criteria measured validly? And predict with reasonable accuracy? chapter 8 Makeing Judgments and Decisions

MANAGERIAL USE OF ASSESSMENTS What do managers want? Fill vacancy asap Want the best people Want it cheap What do managers know about testing? Little; no training in psychometrics or test theory Nothing about construct validity Misinformed on test validity Often distrust testing Some overestimate the validity of tests (especially when over-sold by test vendors (e.g. DISC, MBTI) chapter 8 Makeing Judgments and Decisions

JUDGMENTS AS PREDICTIONS AND DECISIONS What is “common sense”, “intuition?” Give some examples. A thought question - Statistical v. judgmental Prediction What are the pitfalls with each? Which is generally superior? (Meehl, ‘54) Prediction and Decision Without Statistics Usually the most important jobs have few people Why is this a problem for using statistics? How do you deal with this? Combining Information for Overall Assessment Average assessments/use common scale chapter 8 Makeing Judgments and Decisions

chapter 8 Makeing Judgments and Decisions JUDGMENT AIDS Clarifying Judgments to Be Made JA is essential! –it makes management focus on what it will take to do the job Procedural Planning Make a detailed protocol and document it Why is this a good thing to do? Developing Assessment Scales Use the same scale or different scales? A structured interview? Effect size alternatives Common Language Effect Sizes (CLES) Relative Risks (RR) –probability of positive/negative outcome) Behavioral Effect Size Display (Rosenthal) Table 8.2 Expectancy Charts What is a theoretical expectancy chart? How is it used? Give an example Expectancy Graphs Use selection ratio and possible cut scores – to illustrate selection costs to show validity impact (for managers to understand) Utility Analysis Brogden (‘49), Cronbach & Gleser (‘57) Show relative pay off (average dollar pay off) Caveat (GIGO) Concluding Comments Judgments must be rational and include all infromation available chapter 8 Makeing Judgments and Decisions