1. Predictors: Psychological Assessments
Chapter 4
Predictors: Psychological Assessments
Psychology Applied to Work®

2. Learning Objectives
Identify the major types of reliability and what they measure.
Understand the major manifestations of validity and what they measure.
Know the major types of psychological tests categorized by content.
Explain the role of psychological testing in making assessments of people, including ethical issues and predictive accuracy.
Explain non-test predictors such as interviews, assessment centers, physical ability testing, work samples, biographical information, and letters of recommendation.
Understand the controversial methods of assessment.
Psychology Applied to Work®

3. Overview Predictor: Any variable used to forecast a criterion
Two psychometric criteria: reliability and validity
See Hogan’s Guide to Reliability and Validity
Psychology Applied to Work®

4. Reliability Consistency and stability of measurement Four types:
Test-retest Reliability: Coefficient of stability
Equivalent-Form Reliability: Coefficient of equivalence
Internal-Consistency Reliability: Homogeneous content
Split-half Reliability
Cronbach’s Alpha or Kuder-Richardson 20 (KR20)
Inter-Rater Reliability: Conspect reliability
Psychology Applied to Work®

5. Validity Convergent validity Divergent validity
Accuracy of measurement
Construct Validity – Single unitary concept
Continuous process of verifying the accuracy of inferences among concepts
Convergent validity
Divergent validity
Psychology Applied to Work®

6. Psychology Applied to Work®

7. Validity (cont’d) Validity Coefficient Criterion-Related Validity
Concurrent criterion-related validity
Predictive criterion-related validity
Validity Coefficient
Psychology Applied to Work®

8. Validity (cont’d) Items appear appropriate for purpose of assessment
Content Validity
Representative sample of subject domain
Subject Matter Experts
Similar type of "validity" - Face Validity
Items appear appropriate for purpose of assessment
Psychology Applied to Work®

9. Predictor Development
Measure construct
directly (e.g. mechanical comprehension)
sample of behavior (demonstrate on the job)
“Fix it”
Measure
current - Job interview – e.g. interpersonal skills
past behavior – letter of recommendation
(could also be done in an interview, e.g. “give me an example of when you….”)
Psychology Applied to Work®

10. The Purpose of Testing Tests have right/wrong answers, surveys don't
But both are considered “tests” by EEOC
Low-stakes vs. moderate-stakes v. high-stakes testing
Give an example of each
Psychology Applied to Work®

11. Test Content Intelligence Mechanical Aptitude Concept of g
How do you define “g”?
Single best predictor job performance
Multiple intellectual abilities
What are some?
What do Air Traffic Controllers need?
Academic vs. practical vs. creative intelligence
Mechanical Aptitude
Bennett-Mechanical Test
Psychology Applied to Work®

12. Test Content (cont’d) Personality No “right v. wrong” answers
Scale scores used to predict job success
Big 5 Personality Theory
EEOAC
Dark Triad
Machiavellianism, Narcissism, psychopathy
Concerns with faking
What can be done about it?
Which is a better measure of personality?
Self or observer? (Oh et al. 2011)
Psychology Applied to Work®

13. Test Content (cont’d) Integrity Overt (transparent) Personality-based
More predictive of counterproductive work behavior than job performance or turnover (Van Iddekinge et al. ‘13)
Why do you think it is so difficult to validate?
Psychology Applied to Work®

14. Test Content (cont’d) Situational Judgment (SJT)
Responses are not scored simply as right or wrong
So how are they scored?
Designed to reflect reality of making decisions in life
Think of one for your project job
Computerized Adaptive Testing (CAT)
Automated, difficulty level of questions pre-calibrated
Used by the military and other large volume assessments (SAT, GRE)
Psychology Applied to Work®

15. Test Content (cont’d) Testing on the Internet
The move from paper-and-pencil testing to online computer testing (“paper to pixels”)
What are some advantages and disadvantages?
Proctored v. unproctored web-based testing
Would you be more likely to want to use a cognitive ability test or personality inventory online, unproctored?
Psychology Applied to Work®

16. Testing in Retrospect Trends:
Tyranny of testing: critical decisions based on a single test
Inducements to cheat on employment tests
Some tests are useful in predicting job performance, others are not
As a class, moderately predictive; but could be due to poor criterion specification
Psychology Applied to Work®

17. Interviews Value Degree of Structure Situational Interviews
Universal in its appeal
Criterion-related validity
Serves other personnel functions achieved through its use
Degree of Structure
More structure increases validity and fairness
Situational Interviews
Situational (hypothetical) question
Psychology Applied to Work®

18. Work Samples High fidelity (replica) simulations
Effective in blue-collar jobs
Effective at predicting “can do,” but not potential
Psychology Applied to Work®

19. Situational Exercises
White collar counterpart of work samples;
In-basket test
Leaderless group discussion
Modest validity (could be due to poor criterion definition) and very costly
Low fidelity but could still be a reasonable compromise
Psychology Applied to Work®

20. Assessment Centers Five Characteristics Effectiveness
Managers intended candidates – selection, promotion, training
Assessed in group context
Performance is rated by assessors
Numerous performance dimensions assessed
Variety of group exercises and inventories over 1-3 days
Effectiveness
Select talented people (AC ratings correlate with advancement)
May be predictive because of common stereotypes held
Psychology Applied to Work®

21. Physical Abilities Testing
Three main constructs:
Strength
Explosive strength
Gross body coordination
Stamina (Endurance)
Why is it difficult to assess in a work sample?
Psychology Applied to Work®

22. Biographical Information
Past life experiences used to predict future behavior
F. Mael says to ask about past behavior, not intentions –
Why?
Usefulness
Typically high validity
Reveals consistent patterns of behavior in our lives
Often locates unique criterion variance
Legally defensible, but problematic
Problems
Fairness issues
Invasion of privacy
Fakable
Psychology Applied to Work®

23. Letters of Recommendation
Commonly used selection method but least valid: restricted range
Primary value if negative information about candidate is revealed
Psychology Applied to Work®

24. Drug Testing Substance abuse is major global problem
Two types of assessments
Screening test
Confirmation test
Problems of reliability
Practical issues
Cost savings
Issues of uniform drug testing
Some issues outside the scope of I/O psychology
Psychology Applied to Work®

25. Controversial Methods of Assessment
Polygraphy or Lie Detection
President Reagan 1988 ban on private-sector use
Increasing use in government and security agencies
Test of Emotional Intelligence
Highly controversial and still not well-established
Psychology Applied to Work®

26. Overview and Evaluation of Predictors
Four major evaluative standards
Validity: Predictive accuracy
Fairness: Differential predictive accuracy across different groups
Applicability: Interview highest applicability and most universally used
Cost: Subjective and arbitrary, but highly relevant to organizations
Psychology Applied to Work®

27. Overview and Evaluation of Predictors
Psychology Applied to Work®

28. Overview and Evaluation of Predictors
Predictors rarely meet all standards in practice
Three dimensions to judgments of job-related experience
Personal attributes
Perceived outcome of experience
Aspects of experience judged relevance and important
Psychology Applied to Work®