Predictors: Psychological Assessments Chapter 4 Predictors: Psychological Assessments Psychology Applied to Work®

Learning Objectives Identify the major types of reliability and what they measure. Understand the major manifestations of validity and what they measure. Know the major types of psychological tests categorized by content. Explain the role of psychological testing in making assessments of people, including ethical issues and predictive accuracy. Explain non-test predictors such as interviews, assessment centers, physical ability testing, work samples, biographical information, and letters of recommendation. Understand the controversial methods of assessment. Psychology Applied to Work®

Overview Predictor: Any variable used to forecast a criterion Two psychometric criteria: reliability and validity See Hogan’s Guide to Reliability and Validity Psychology Applied to Work®

Reliability Consistency and stability of measurement Four types: Test-retest Reliability: Coefficient of stability Equivalent-Form Reliability: Coefficient of equivalence Internal-Consistency Reliability: Homogeneous content Split-half Reliability Cronbach’s Alpha or Kuder-Richardson 20 (KR20) Inter-Rater Reliability: Conspect reliability Psychology Applied to Work®

Validity Convergent validity Divergent validity Accuracy of measurement Construct Validity – Single unitary concept Continuous process of verifying the accuracy of inferences among concepts Convergent validity Divergent validity Psychology Applied to Work®

Psychology Applied to Work®

Validity (cont’d) Validity Coefficient Criterion-Related Validity Concurrent criterion-related validity Predictive criterion-related validity Validity Coefficient Psychology Applied to Work®

Validity (cont’d) Items appear appropriate for purpose of assessment Content Validity Representative sample of subject domain Subject Matter Experts Similar type of "validity" - Face Validity Items appear appropriate for purpose of assessment Psychology Applied to Work®

Predictor Development Measure construct directly (e.g. mechanical comprehension) sample of behavior (demonstrate on the job) “Fix it” Measure current - Job interview – e.g. interpersonal skills past behavior – letter of recommendation (could also be done in an interview, e.g. “give me an example of when you….”) Psychology Applied to Work®

The Purpose of Testing Tests have right/wrong answers, surveys don't But both are considered “tests” by EEOC Low-stakes vs. moderate-stakes v. high-stakes testing Give an example of each Psychology Applied to Work®

Test Content Intelligence Mechanical Aptitude Concept of g How do you define “g”? Single best predictor job performance Multiple intellectual abilities What are some? What do Air Traffic Controllers need? Academic vs. practical vs. creative intelligence Mechanical Aptitude Bennett-Mechanical Test Psychology Applied to Work®

Test Content (cont’d) Personality No “right v. wrong” answers Scale scores used to predict job success Big 5 Personality Theory EEOAC Dark Triad Machiavellianism, Narcissism, psychopathy Concerns with faking What can be done about it? Which is a better measure of personality? Self or observer? (Oh et al. 2011) Psychology Applied to Work®

Test Content (cont’d) Integrity Overt (transparent) Personality-based More predictive of counterproductive work behavior than job performance or turnover (Van Iddekinge et al. ‘13) Why do you think it is so difficult to validate? Psychology Applied to Work®

Test Content (cont’d) Situational Judgment (SJT) Responses are not scored simply as right or wrong So how are they scored? Designed to reflect reality of making decisions in life Think of one for your project job Computerized Adaptive Testing (CAT) Automated, difficulty level of questions pre-calibrated Used by the military and other large volume assessments (SAT, GRE) Psychology Applied to Work®

Test Content (cont’d) Testing on the Internet The move from paper-and-pencil testing to online computer testing (“paper to pixels”) What are some advantages and disadvantages? Proctored v. unproctored web-based testing Would you be more likely to want to use a cognitive ability test or personality inventory online, unproctored? Psychology Applied to Work®

Testing in Retrospect Trends: Tyranny of testing: critical decisions based on a single test Inducements to cheat on employment tests Some tests are useful in predicting job performance, others are not As a class, moderately predictive; but could be due to poor criterion specification Psychology Applied to Work®

Interviews Value Degree of Structure Situational Interviews Universal in its appeal Criterion-related validity Serves other personnel functions achieved through its use Degree of Structure More structure increases validity and fairness Situational Interviews Situational (hypothetical) question Psychology Applied to Work®

Work Samples High fidelity (replica) simulations Effective in blue-collar jobs Effective at predicting “can do,” but not potential Psychology Applied to Work®

Situational Exercises White collar counterpart of work samples; In-basket test Leaderless group discussion Modest validity (could be due to poor criterion definition) and very costly Low fidelity but could still be a reasonable compromise Psychology Applied to Work®

Assessment Centers Five Characteristics Effectiveness Managers intended candidates – selection, promotion, training Assessed in group context Performance is rated by assessors Numerous performance dimensions assessed Variety of group exercises and inventories over 1-3 days Effectiveness Select talented people (AC ratings correlate with advancement) May be predictive because of common stereotypes held Psychology Applied to Work®

Physical Abilities Testing Three main constructs: Strength Explosive strength Gross body coordination Stamina (Endurance) Why is it difficult to assess in a work sample? Psychology Applied to Work®

Biographical Information Past life experiences used to predict future behavior F. Mael says to ask about past behavior, not intentions – Why? Usefulness Typically high validity Reveals consistent patterns of behavior in our lives Often locates unique criterion variance Legally defensible, but problematic Problems Fairness issues Invasion of privacy Fakable Psychology Applied to Work®

Letters of Recommendation Commonly used selection method but least valid: restricted range Primary value if negative information about candidate is revealed Psychology Applied to Work®

Drug Testing Substance abuse is major global problem Two types of assessments Screening test Confirmation test Problems of reliability Practical issues Cost savings Issues of uniform drug testing Some issues outside the scope of I/O psychology Psychology Applied to Work®

Controversial Methods of Assessment Polygraphy or Lie Detection President Reagan 1988 ban on private-sector use Increasing use in government and security agencies Test of Emotional Intelligence Highly controversial and still not well-established Psychology Applied to Work®

Overview and Evaluation of Predictors Four major evaluative standards Validity: Predictive accuracy Fairness: Differential predictive accuracy across different groups Applicability: Interview highest applicability and most universally used Cost: Subjective and arbitrary, but highly relevant to organizations Psychology Applied to Work®

Overview and Evaluation of Predictors Psychology Applied to Work®

Overview and Evaluation of Predictors Predictors rarely meet all standards in practice Three dimensions to judgments of job-related experience Personal attributes Perceived outcome of experience Aspects of experience judged relevance and important Psychology Applied to Work®