Combining Test Data MANA 4328 Dr. Jeanne Michalski

Selection Decisions  First, how to deal with multiple predictors?  Second, how to make a final decision?

Developing a Hiring System  OK, Enough Assessing:  Who Do We Hire??!!

Summary of Performance-Based Hiring  Understand job requirements and performance expectations  List competencies, KSAO’s that predict performance  Match attributes with selection tools  Choose/develop each tool effectively  Make performance-based decisions

Interpreting Test Scores  Norm-referenced scores  Test scores are compared to applicants or comparison group.  Raw scores should be converted to Z scores or percentiles  Use “rank ordering”  Criterion-referenced scores  Test scores indicate a degree of competency  NOT compared to other applicants  Typically scored as “qualified” vs. “not qualified”  Use “cut-off scores”

Setting Cutoff Scores  Based on the percentage of applicants you need to hire (yield ratio). “Thorndike’s predicted yield”  You need 5 warehouse clerks and expect 50 to apply. 5 / 50 =.10 (10%) means 90% of applicants rejected  Cutoff Score set at 90th percentile  Z score 1 = 84 th percentile  Based on a minimum proficiency score  Based on validation study linked to job analysis  Incorporates SEM (validity and reliability)

Selection Outcomes PREDICTION PERFORMANCE No PassPass Regression Line Cut Score 90% Percentile

Selection Outcomes PREDICTION High Performer Low Performer True Positive True Negative Type 2 Error False Positive Type 1 Error False Negative PERFORMANCE No HireHire

Selection Outcomes PREDICTION High Performer Low Performer PERFORMANCE UnqualifiedQualified Prediction Line Cut Score

Dealing With Multiple Predictors “Mechanical” techniques superior to judgment 1. Combine predictors  Compensatory or “test assessment approach” 2. Judge each independently  Multiple Hurdles / Multiple Cutoff 3. Profile Matching 4. Hybrid selection systems

Compensatory Methods Unit weighting P1 + P2 + P3 + P4 = Score Rational weighting (.10) P1 + (.30) P2 + (.40) P3 + (.20) P4 = Score Ranking RankP1 + RankP2 +RankP3 + RankP4 = Score Profile Matching D 2 = Σ (P(ideal) – P(applicant)) 2

Multiple Regression Approach  Predicted Job perf = a + b 1 x 1 + b 2 x 2 + b 3 x 3  x = predictors; b = optimal weight  Issues:  Compensatory: assumes high scores on one predictor compensate for low scores on another  Assumes linear relationship between predictor scores and job performance (i.e., “more is better”)

Multiple Cutoff Approach  Sets minimum scores on each predictor  Issues  Assumes non-linear relationship between predictors and job performance  Assumes predictors are non-compensatory  How do you set the cutoff scores?

Multiple Cutoff Approach  Sets minimum scores on each predictor  Issues  Assumes non-linear relationship between predictors and job performance  Assumes predictors are non-compensatory  How do you set the cutoff scores?  If applicant fails first cutoff, why continue?

Test 1Test 2 Interview Background Finalist Decision Reject Multiple Hurdle Model Fail Pass

Multiple Hurdle Model  Multiple Cutoff, but with sequential use of predictors  If applicant passes first hurdle, moves on to the next  May reduce costs, but also increases time

Profile Matching Approach  Emphasizes “ideal” level of KSA  e.g., too little attention to detail may produce sloppy work; too much may represent compulsiveness  Issues  Non-compensatory  Small errors in profile can add up to big mistake in overall score  Little evidence that it works better

Making Finalist Decisions  Top-Down Strategy  Maximizes efficiency, but may need to look at adverse impact issues  Banding Strategy  Creates “bands” of scores that are statistically equivalent (based on reliability)  Then hire from within bands either randomly or based on other factors (inc. diversity)

Banding  Grouping like test scores together  Function of test reliability  Standard Error of Measure  Band of + or – 2 SEM  95% Confidence interval  If the top score on a test is 95 and SEM is 2, then scores between 95 and 91 should be banded together.

Applicant Total Scores

Information Overload!!  Leads to:  Reverting to gut instincts  Mental Gymnastics

Combined Selection Model Selection Stage Selection TestDecision Model Applicants  Candidates Application BlankMinimum Qualification Hurdle Candidates  Finalists Four Ability Tests Work Sample Rational Weighting Hurdle Finalists  Offers Structured InterviewUnit Weighting Rank Order Offers  Hires Drug Screen Final Interview Hurdle

Alternative Approach  Rate each attribute on each tool  Desirable  Acceptable  Unacceptable  Develop a composite rating for each attribute  Combining scores from multiple assessors  Combining scores across different tools  A “judgmental synthesis” of data  Use composite ratings to make final decisions

List of Critical Attributes

Performance Attributes Matrix

Who Do You Hire??

Improving Ratings 1. Use rating system  Unacceptable  Did not demonstrate levels of attribute that would predict acceptable performance  Acceptable  Demonstrated levels that would predict acceptable performance  Desirable  Demonstrated levels that would predict exceptional performance

Categorical Decision Approach 1. Eliminate applicants with unacceptable qualifications 2. Then hire candidates with as many desirable ratings as possible 3. Finally, hire as needed from applicants with “acceptable” ratings  Optional: “weight” attributes by importance

Sample Decision Table

More Positions than Applicants

More Applicants than Positions

Selection  Top Down Selection (Rank) vs. Cutoff scores  Is the predictor linearly related to performance?  How reliable are the tests? 1. Top-down method – Rank order 2. Minimum cutoffs – Passing Scores

Final Decision  Random Selection  Ranking  Grouping  Role of Discretion or “Gut Feeling”

Summary of Performance-Based Hiring  Understand job requirements and performance expectations  List competencies, KSAO’s that predict performance  Match attributes with selection tools  Choose/develop each tool effectively  Make performance-based decisions