1. Four Linear Judgment Models
Psychology 466: Judgment & Decision Making Instructor: John Miyamoto 10/05/2017: Lecture 02-2
Note: This Powerpoint presentation may contain macros that I wrote to help me create the slides. The macros aren’t needed to view the slides. You can disable or delete the macros without any change to the presentation.

2. Outline Four linear judgment models
Why are psychologists interested in linear judgment models?
Model 1: Multiple regression applied to objective data.
Model 2: Model of the Judge Multiple regression applied to the judge's prediction
Model 3: Regression model with importance weights
Model 4: Unit weighting model (very crude regression model)
How to compute predictions with these models.
What do these models show about human judgment?
Psych 466, Miyamoto, Aut '17
Reminder: Diagram of Lens Model

3. The Lens Model of Egon Brunswik
Figure 3.1 of Hastie & Dawes.
To-be-judged criterion = the thing you are trying to predict.
Cues = things you can observe about the criterion
Judgment = the “judge’s” prediction (you are the judge)
The lens model is a conceptualization of the structure of typical judgment problems.
Psych 466, Miyamoto, Aut '17
Four Linear Judgment Models

4. Four Linear Judgment Models
Using multiple regression on objective data for which the true state is known.
This method produces a proper linear model that is statistically the best way to generate accurate predictions.
Using multiple regression on judgment data where the true state is not known.
This method produces a model of the judge (abbreviated as MUD or Model of the jUDge)
SMART (Simple Multi-Attribute Rating Technique)
A.k.a. Importance weighting model. A.k.a. Multiattribute Utility Technique (MAUT)
This method produces an improper linear model.
Unit Weighting Model
Relevance of these Four Models to Assignment 1
Psych 466, Miyamoto, Aut '17

5. Relevance of these Four Methods to Assignment 1
Four Linear Judgment Models
Proper Linear Model
Model of the Judge (MUD)
SMART (a.k.a. importance weighting model; a.k.a. MAUT)
Unit Weighting (a type of improper linear model)
Assignment 1: Explain how to use a linear judgment procedure to choose 1 UW course to take.
Minimize issues of major or minor requirements.
Compare intuitive judgment and linear judgment models.
On Assignment 1, you only need to explain 1 of 4 methods.
Introduction to Baron’s College Admission Decision Problem
Psych 466, Miyamoto, Aut '17

6. Next: Explain These Four Methods on a Concrete Judgment Problem
Concrete Problem: Predicting college performance (GPA) based on information in a college application.
Predictions based on a linear model – how to produce them
Predictions based on a model of the judge – how to produce them
Predictions based on SMART method
Predictions based on unit weights – how to produce them
Assignment 1: Pick one of these four methods. Describe this method and then discuss its strengths and weaknesses relative to other methods (including intuitive judgment.
Psych 466, Miyamoto, Aut '17
Baron's College Admission Problem as an Example of Lens Model

7. Baron's College GPA Judgment Problem
Using the terminology of the lens model, the GPA judgment problem looks like this:
Terminology
Example
Criterion (what we want to predict)
COL = College GPA of high school student
Cues (these are things we know)
GPA = High school GPA
SAT = SAT test scores
REC = recommendations (converted to ratings)
ESS = essay quality (converted to a rating)
Judgment (this is our prediction)
PRE = Estimate (guess) of student’s future college GPA
Table Showing Data & Variables for the College Admission Example
Psych 466, Miyamoto, Aut '17

8. Judgment Data Used in Baron's Chapter 20
COL = college GPA. This is the criterion This is what the judge wants to predict.
SAT = SAT score; REC = judge's rating of the recommendation; ESS = judge's rating of the student's essay; GPA = high school GPA These are the cues.
Comment re Qualitative Variables
Psych 466, Miyamoto, Aut '17

9. Four Ways to Compute a Statistical Prediction Model
Next
Method 1: Multiple regression applied to existing data Called a “proper linear model”
Method 2: Multiple regression applied to a judge’s predictions Called a “model of the judge”
Method 3: SMART Method with "importance" weights Called the SMART method or importance weighting method
Method 4: Unit weighting model Called the unit weighting model or unit weighting method
Multiple Linear Regression
Psych 466, Miyamoto, Aut '17

10. Prediction Equation with 4 Predictor Variables
COL = the criterion = to-be-predicted quantity
SAT, REC, ESS, GPA are the cues (predictor variables)
Prediction Equation
PRE = ·SAT ·REC ·ESS ·GPA  5.161
Example: If a high school student has SAT = 1200, REC = 3.7, ESS = 3.9, GPA = 3.2, then we predict
PRE = ·(1200) ·(3.7) ·(3.9) ·(3.2) – 5.161 =
How to Use the Multiple Regression Model to Predict New Cases
Psych 466, Miyamoto, Aut '17

11. Four Ways to Compute a Statistical Prediction Model
Method 1: Multiple regression applied to existing data Called a “proper linear model”
Method 2: Multiple regression applied to a judge’s predictions Called a “model of the judge”
Method 3: SMART Method with "importance" weights Called the SMART method or importance weighting method
Method 4: Unit weighting model Called the unit weighting model or unit weighting method
Next
Model of the Judge
Psych 466, Miyamoto, Aut '17

12. Method 2: Model of the Judge
Researcher has available the scores for SAT, REC, ESS and GPA.
The values of the criterion (COL) are NOT available.
Researcher asks the judge to make intuitive, global predictions for these cases. This produces the column labeled "JUD” (next slide).
Not Available
Not Available
Same Slide Except JUD Column Added to Table
Psych 466, Miyamoto, Aut '17

13. Method 2: Model of the Judge
Researcher has available the scores for SAT, REC, ESS and GPA.
The values of the criterion (COL) are NOT available.
Researcher asks the judge to make intuitive, global predictions for these cases. This produces the column labeled "JUD."
Not Available
Not Available
Regression Equation for the Model of the Judge
Psych 466, Miyamoto, Aut '17

14. Method 2: Model of the Judge
It is an accident that in this example, JUD and MUD are identical
Compute a regression model that predicts JUD (Model of the jUDge or MUD).
Example: “Policy Capturing”.
Not Available
Not Available
See ‘e:\p466\nts\baron.quant.jdmt.r-code.docm’ for R-code that computes the prediction from the model of the judge.
Example:
MUD = (–5x10–18)·SAT ·REC ·ESS ·GPA  4.8
Use the MUD model to predict college GPA (COL) for these cases or future cases.
Pstch 466, Miyamoto, Aut '17
Discussion of Model of the Judge

15. Four Ways to Compute a Statistical Prediction Model
Method 1: Multiple regression applied to existing data. Called a “proper linear model”
Method 2: Multiple regression applied to a judge’s predictions Called a “model of the judge”
Method 3: SMART Method with "importance" weights Called the SMART method or importance weighting method
Method 4: Unit weighting model Called the “unit weighting model or unit weighting method”
Next
SMART Method
Psych 466, Miyamoto, Aut '17

16. Method 3: The SMART Method
Z-Scores
R-code for the table to the right is in ‘e:\p466\nts\baron.quant.jdmt.r-code.doc’
Table on the left shows the initial data.
Table on the right shows the z-scores for the predictor variables, and the predicted rating for each student.
Predicted Score = 2·Z.sat + 3·Z.rec + 1·z.ess + 2·Z.gpa
The decision maker has to produce the importance weights based on subjective judgment.
Psych 466, Miyamoto, Aut '17
Example: Computing the Predicted Score for One Student

17. Four Ways to Compute a Statistical Prediction Model
Method 1: Multiple regression applied to existing data. Called a “proper linear model”
Method 2: Multiple regression applied to a judge’s predictions Called a “model of the judge”
Method 3: SMART Method with "importance" weights Called the SMART method or importance weighting method
Method 4: Unit weighting model Called the “unit weighting model or unit weighting method”
Next
Unit Weighting Method
Psych 466, Miyamoto, Aut '17

18. Method 4: Unit Weighting Model
Z-Scores
R-code for the table to the right is in ‘e:\p466\nts\baron.quant.jdmt.r-code.doc’
Table on the right shows the z-scores for the predictor variables, and the predicted rating for each student.
Example for Case 1: (2.39) + 1(.32) + 1(.41) + 1(1.58) = 4.70
Findings for the Unit Weighting Model
Psych 466, Miyamoto, Aut '17

19. Next: Methods 2 – 4 Are Illustrated with an Apartment Choice Problem
See hnd02-2.p466.a17.pdf. (You can download this pdf from the course website; look at the Week 2 section.)
Psych 466, Miyamoto, Aut ‘17

20. Applying Linear Judgment Models 2 - 4 to an Apartment Choice Problem
Problem: You are trying to evaluate a series of apartments; which is best? which is second best? etc.
Psych 466,, Miyamoto, Aut '17

21. Data for 8 Hypothetical Apartments
Psych 466,, Miyamoto, Aut '17

22. Model 1: The Multiple Regression Model
Model 1 cannot be applied to these data!
Why? Because you don't know the value of the criterion (your degree of satisfaction with living in similar apartments) for a sample of apartments.
In order to apply the multiple regression model, you would have to live in similar apartments, and keep notes about rent, commute time, size and quality, plus a rating of how satisfied you were with living in each apartment. Conceivably, you could find data based on students who are similar to you and are living in similar apartments, but such data would probably be hard to find.
Psych 466, Miyamoto, Aut '17

23. Model 2: The Model of the Judge
Step The Judge must produce a rating of satisfaction for a sample of apartments.
Psych 466,, Miyamoto, Aut '17

24. Model 2: The Model of the Judge (Cont.)
Step Use a statistical program to compute a multiple regression that that predicts Judge based on the values for Rent, Commute, Size, and Quality.
Psych 466,, Miyamoto, Aut '17

25. Model 2: The Model of the Judge (Cont.)
Step 2.3. Compute the predicted score for every apartment in the set of possible choices. Step 2.4. Rank order the predicted satisfaction scores, from the lowest (1) to the highest (8).
Psych 466,, Miyamoto, Aut '17

26. Model 2: The Model of the Judge (Cont.)
Step 2.3. Compute the predicted score for every apartment in the set of possible choices. Step 2.4. Rank order the predicted satisfaction scores, from the lowest (1) to the highest (8).
Psych 466,, Miyamoto, Aut '17

27. Model 3: The SMART Model (the Importance Weighting Model)
Step Convert the data to z-scores.
Psych 466,, Miyamoto, Aut '17

28. Model 3: The SMART Model (Cont.)
Step Assign importance weights to the dimensions, Apartment, Rent, Commute and Size.
Assign positive weights to dimensions that contribute positively to a better outcome, and assign negative weights to dimensions that contribute negatively to a better outcome.
Psych 466,, Miyamoto, Aut '17

29. Model 3: The SMART Model (Cont.)
Step 3.3. Use the importance weights and the prediction equation (2) to compute the expected satisfaction of each apartment. Equation (2):
Psych 466,, Miyamoto, Aut '17

30. Model 3: The SMART Model (Cont.)
Step Rank order the predicted satisfaction scores, from the lowest (1) to the highest (8)
Psych 466,, Miyamoto, Aut '17

31. Method 4: Unit Weighting Model
Step 4.1. Convert all cues to z-scores. This produces a table that is identical to Table 4. Step 4.2. Give +1 weights to every dimension that contributes positively to the outcome (satisfaction). Give −1 weights to every dimension that contributes negatively (adversely) to the outout.
Psych 466,, Miyamoto, Aut '17

32. Method 4: Unit Weighting Model (Cont.)
Step Apply the Unit Weighting equation to the data.
Psych 466,, Miyamoto, Aut '17

33. End of Example
Psych 466,, Miyamoto, Aut '17

34. Only if you want to make a quantitative prediction
This table is Table 3 in ‘e:\p466\hnd.02-2a.p466.a13.docm’. The table was copied to an image and pasted into these slides.
Repeat this Table Without the Red Rectangles
Psych 466, Miyamoto, Aut '17

35. This table is Table 3 in ‘e:\p466\hnd. 02-2a. p466. a13. docm’
This table is Table 3 in ‘e:\p466\hnd.02-2a.p466.a13.docm’. The table was copied to an image and pasted into these slides.
Conclusions
Psych 466, Miyamoto, Aut '17

36. What We Have Learned from the Study of Linear Judgment Models
Human judges typically believe that they use complex judgment processes to make judgments from complex cues. (This may be true.)
The part of the human judgment process that validly predicts the criterion is well modeled by a simple linear model.
We don't need to use optimal regression models to outperform human judges.
We don't need to know the value of the criterion in order to create a model that outperforms human judges.
Evidence: Model of the judge, unit weighting and importance weighting models outperform the human judge.
Note: These methods work only because the human judge has some valid knowledge of the relationship between the cues and the criterion.
Sample Size Affects the Predictive Accuracy of Multiple Regression & MUD
Psych 466, Miyamoto, Aut '17

37. Sample Size Affects the Accuracy of Multiple Regression Model and Model of the Judge
This point is emphasized by Gigerenzer: If you are using a multiple regression model or the model of the judge, the predicted accuracy of the model is bad if the sample size is too small.
If the sample size is small, then the regression weights will tend to be inaccurate. (Technically, the variance of the regression weights greatly increases as the sample size gets smaller.)
If the sample size is small, the unit weighting model can be more accurate than multiple regression model or the model of the judge. (This point is emphasized by Gigerenzer.)
Psych 466, Miyamoto, Aut '17
Why Do Statistical Model Outperform Humans?

38. Why Do Statistical Models Outperform Human Judges?
Human judgment is affected by internal random variation; statistical model is not.
Human judgment is affected by vivid individual cases (anecdotes); statistical model is not.
Speculation: Human judge tries to fit information into a story; statistical model ignores story; it just adds up the evidence But is the human preference for stories bad?
General Discussion of Linear Judgment Models - END
Psych 466, Miyamoto, Aut '17

39. General Discussion of Linear Models
Why aren’t linear judgment models used more widely in practical decision making?
College or graduate admissions
NIH or NSF grant review committees
Political decisions like where to locate a prison; where to locate a homeless shelter;
The Denver bullet study
Are linear judgment methods dehumanizing, e.g., when choosing who will be admitted to a college?
END
Psych 466, Miyamoto, Aut '17

40. Psych 466, Miyamoto, Aut ‘17

41. Set Up for Instructor
Turn off your cell phone. Close web browsers if they are not needed.
Classroom Support Services (CSS), 35 Kane Hall,
If the display is odd, try setting your resolution to 1024 by 768
Run Powerpoint. For most reliable start up:
Start laptop & projector before connecting them together
If necessary, reboot the laptop
Psych 466, Miyamoto, Aut ‘17