© PeopleFluent 2014 Testing the Test Presenter: Lisa Harpe, Ph.D. Peoplefluent Research Institute (PRI)

2 © 2014 PeopleFluent | Confidential Presentation Title | date Agenda Regulatory Climate What is a Test? What is Validity? Questions for a Third Party Test Vendor Questions for a Test User Documentation Checklists

3 © 2014 PeopleFluent | Confidential Presentation Title | date Regulatory Climate EEOC Experts on staff Charges Some well known cases in the mid-2000s: Dial Corp, Whirlpool, Fed-Ex Criminal Background Checks ADA violations OFCCP Experts on staff Settlements (in last 2 years) Leprino Foods GE Lincoln Electric

4 © 2014 PeopleFluent | Confidential Presentation Title | date UGESP Uniform Guidelines on Employment Selection Procedures (UGESP) (1978) Adverse impact (A.I) No need to show validity if no A.I. Justify, change, find alternative or eliminate if A.I. Justification Validity evidence = job related Business necessity

5 © 2014 PeopleFluent | Confidential Presentation Title | date UGESP Citations Section 60-3, Uniform Guidelines on Employee Selection Procedure (1978); 43 FR (August 25, 1978). Equal Employment Opportunity Commission: 29 CFR Part 1607 Department of Labor / Office of Federal Contract Compliance Programs: 41 CFR Part 60-3 Department of Justice: 28 CFR Civil Service Commission: 5 CFR (c)

6 © 2014 PeopleFluent | Confidential Presentation Title | date Resources Uniform Guidelines on Employment Selection Procedures (UGESP, 1978) Department of Labor’s (DOL) Testing and Assessment: an Employer’s Guide to Good Practices (1999) American Psychological Association’s (APA) Standards for Educational and Psychological Testing (2014) Society of Industrial Organizational Psychologist’s (SIOP) Principles for the Validation and Use of Personnel Selection Procedures (4 th edition, 2003)

7 © 2014 PeopleFluent | Confidential Presentation Title | date What is a Test? ‘Any instrument or procedure that samples behavior or performance’ (DOL Guide) ‘Any device or procedure that samples behavior which is scored in a standard process’ (APA Standards) ‘Any procedure used singly or in combination to make personnel decisions’ (‘selection procedure, test, assessment, and predictor’ = interchangeable terms) (SIOP Principles) vs. basic qualification questionnaire

8 © 2014 PeopleFluent | Confidential Presentation Title | date Types of Tests Cognitive ability tests Physical ability tests Achievement tests Personality tests Job knowledge tests Skills tests Honesty and integrity tests Biodata inventories Work samples and simulations Assessment centers

9 © 2014 PeopleFluent | Confidential Presentation Title | date Types of Tests Employment interviews Education and experience requirements (including licensing and certification) Recommendations and reference checks Criminal record check Medical examinations Drug and alcohol tests

10 © 2014 PeopleFluent | Confidential Presentation Title | date What is Validity? Principles advocates the Standards’ definition as “the degree to which accumulated evidence and theory support specific interpretations of test scores entailed by proposed uses of a test” (AERA et al., 1999, p. 184)

11 © 2014 PeopleFluent | Confidential Presentation Title | date Validity Validity depends on the purpose of the test Measure of extraversion vs. predictor of job success vs. predictor of training success Selection vs. promotion vs. placement Target population Validity depends on the use of the test results Cut score (Pass/Fail) Ranking Part of assessment battery

12 © 2014 PeopleFluent | Confidential Presentation Title | date Sources of Validity Evidence Content validity Can you show a direct link between the test items and job behaviors, work outcomes or KSAOs? Criterion related validity Can you show that the test is related to performance? Construct validity Are there multiple lines of evidence that whatever is being measured by the test is related to the job? Other Response Process Internal structure Consequential

13 © 2014 PeopleFluent | Confidential Presentation Title | date Content Validity Test Content Validity Work behaviors, activities, outcomes or KSAOs

14 © 2014 PeopleFluent | Confidential Presentation Title | date Content Validity Expert Judgment SME Incumbents Hiring manager HR Industrial Psychologists Define Important Work Behaviors, Activities and Outcomes Evaluate Test Content for Relevance

15 © 2014 PeopleFluent | Confidential Presentation Title | date Content Validity JobRepresentative Work Behavior Test Content Administrative Assistant Maintain supervisor’s calendar Microsoft Outlook Skills Assessment Administrative Assistant Word processingMicrosoft Word Skills Assessment Truck DriverDrive truckTruck Driving Test

16 © 2014 PeopleFluent | Confidential Presentation Title | date Criterion Related Validity Criterion Measures Performance ratings 6 month, annual Supervisor, peer, customer / client Productivity Sales revenue Turnover Absenteeism Multi-dimensional

17 © 2014 PeopleFluent | Confidential Presentation Title | date

18 © 2014 PeopleFluent | Confidential Presentation Title | date Criterion-related validity Predictive: Can test results tell us how well a person will perform in the job? Concurrent: Are test results related to job performance?

19 © 2014 PeopleFluent | Confidential Presentation Title | date Criterion-related Validity Correlation Coefficient (r)

20 © 2014 PeopleFluent | Confidential Presentation Title | date Criterion-related Validity Correlation Coefficient (r)

21 © 2014 PeopleFluent | Confidential Presentation Title | date Criterion-related Validity Correlation Coefficient (r)

22 © 2014 PeopleFluent | Confidential Presentation Title | date Question 1: Is the correlation statistically significant? Question 2: Is the correlation practically significant? Criterion-related Validity

23 © 2014 PeopleFluent | Confidential Presentation Title | date Criterion-related Validity Correlation coefficient value Interpretation above.35 very beneficial likely to be useful depends on circumstances below.11 unlikely to be useful Guidelines for Interpreting Correlation Coefficients (DOL guide)

24 © 2014 PeopleFluent | Confidential Presentation Title | date TestJob Performance R-sq =.75 Criterion-related Validity Coefficient of Determination (R-squared)

25 © 2014 PeopleFluent | Confidential Presentation Title | date Test Job Performance R-sq =.00 Criterion-related Validity Coefficient of Determination (R-squared)

26 © 2014 PeopleFluent | Confidential Presentation Title | date Test Job Performance R-sq =.2 Coefficient of Determination (R-squared) Criterion-related Validity

27 © 2014 PeopleFluent | Confidential Presentation Title | date

28 © 2014 PeopleFluent | Confidential Presentation Title | date What Influences the Validity Coefficient Validity of BOTH the test (predictor) and performance measure (criterion) Reliability of BOTH the test (predictor) and performance measure (criterion) Range restriction in the test (predictor) and / or performance measure (criterion) Sample size affects the significance of r

29 © 2014 PeopleFluent | Confidential Presentation Title | date Construct validity Construct = unobservable trait or characteristic Examples Extraversion Customer service orientation Intelligence Oral presentation skills Ability to trouble-shoot Leadership

30 © 2014 PeopleFluent | Confidential Presentation Title | date Construct Validity Predictor Measure (e.g., CallCenterPro) Criterion Construct Domain (e.g., Interpersonal Interactions) Criterion Measure (e.g., Supervisor Rating) Predictor Construct Domain (e.g., People Skills)

31 © 2014 PeopleFluent | Confidential Presentation Title | date Other Types of Validity Evidence Face Validity Response Process Internal Structure Consequential

32 © 2014 PeopleFluent | Confidential Presentation Title | date Job Analysis Why Do I Care About This? Analysis of Work (SIOP) 2 Purposes Develop predictors (tests / selection procedures) Develop criteria (performance measures) Approaches Questionnaires Observations Interviews Documentation Currency

33 © 2014 PeopleFluent | Confidential Presentation Title | date Reliability Reliability = Consistency or Stability Why Do I Care About This? Types: Test-retest Interrater reliability Internal consistency Alternate form

34 © 2014 PeopleFluent | Confidential Presentation Title | date Reliability Guidelines for Interpreting Reliability Coefficients (DOL) Reliability Coefficient Interpretation.90 and up excellent good adequate below.70 may have limited applicability

35 © 2014 PeopleFluent | Confidential Presentation Title | date Validity Generalization Meta-analysis Synthetic Validity Transportability Job similarity Job context similarity Criterion similarity Test use similarity Test taker similarity Test fairness

36 © 2014 PeopleFluent | Confidential Presentation Title | date Third Party Tests Sources of Validity Evidence Vendor manuals and tech reports Independent reviews Research literature Mental Measurements Yearbook (Buros Institute of Mental Measurements) Tests in Print (TIP) (Buros Institute of Mental Measurements)

37 © 2014 PeopleFluent | Confidential Presentation Title | date Questions for a Test Vendor What are your credentials and experience? Can you provide references? Do you have a technical report that describes the development and validation of the test? Will this report be updated over time? If so, how will you inform me of the updates? What costs are involved with using the test in my selection process? For which job(s) is the test valid? Can you show that the test is valid for my company and my job(s)?

38 © 2014 PeopleFluent | Confidential Presentation Title | date Questions for a Test Vendor Can you show that the test is reliable? Can you show that the test is not biased? Can you provide the results of adverse impact studies by race and gender? Can you conduct adverse impact studies for my job(s)? Can you provide information on test administration, scoring and interpretation? Will you evaluate the effectiveness of the test in my company? Will you make changes if warranted?

39 © 2014 PeopleFluent | Confidential Presentation Title | date Will the test by changed or updated? How will you inform me of any changes or updates? Can you document the validity of the test in accordance with the UGESP requirements? If the vendor can not provide an answer or does not know what you are talking about, run don’t walk! What to ask a test vendor

40 © 2014 PeopleFluent | Confidential Presentation Title | date Questions for the Test User What employment tests are used in the selection process? For which job(s)? Where in the process? Does the test have an adverse impact? Who developed this test? where did we get the test? Did we consider any alternatives? Where is the job analysis for this job?

41 © 2014 PeopleFluent | Confidential Presentation Title | date How are the test results used? Is the test reliable? Do we have evidence of the test’s validity? Does the evidence meet the UGESP and SIOP criteria for documentation? Questions for the Test User

42 © 2014 PeopleFluent | Confidential Presentation Title | date Homemade Tests See recommended process for work sample tests Tests found on the Internet Off the shelf tests ‘valid’ for a given industry Common Issues

43 © 2014 PeopleFluent | Confidential Presentation Title | date Establish a ‘test’ panel of subject matter experts (individuals who know the job and what the job requires) – 1-2 supervisors, 2-3 incumbents (the more the better) with more than 1 year experience, HR rep. Is test related to job? Identify and document what knowledge, skills or abilities (KSA) are measured by the test. Document that KSAs measured by test are required on the job Provide examples of tasks and duties representing the KSAs Indicate frequency of performing tasks or duties requiring the KSAs Indicate the importance of performing tasks or duties requiring the KSAs Are the KSAs needed on the first day of the job? RECOMMENDED PROCESS FOR WORK SAMPLE TEST

44 © 2014 PeopleFluent | Confidential Presentation Title | date How to score test? Experts define criteria as specifically as possible Panel of SMEs should score the test Panel should agree on definition of success Conduct an annual review of the test Relationship to performance (could include turnover) Adverse impact RECOMMENDED PROCESS FOR WORK SAMPLE TEST

45 © 2014 PeopleFluent | Confidential Presentation Title | date Other Use test results as input to facilitate the interview process and not as a defining step unless you have strong evidence to support a cut score Use samples of work required on first day of job Standardize administration and scoring and use of results Maintain records Test taker name Test taker race and gender Requisition test taker is being considered for Test date Test results – component, overall and any decision resulting from the test RECOMMENDED PROCESS FOR WORK SAMPLE TEST

46 © 2014 PeopleFluent | Confidential Presentation Title | date Thank You Please contact me with questions or comments (919) For more information Corporate Headquarters (877) (toll free) or (919)