1. ©2011 Cengage Learning. All Rights Reserved. May not be scanned, copied or duplicated, or posted to a publicly accessible website, in whole or in part. Reliability of Selection Measures CHAPTER 4 Part 2Foundations of Measurement for Human Resource Selection

2. What Is Reliability? ReliabilityReliability  The degree of dependability, consistency, or stability of scores on a measure (either predictors or criteria) used in selection research. Predictions concerning selection must be based on the results of “true measures” of differences that have:Predictions concerning selection must be based on the results of “true measures” of differences that have:  Consistency in results  Dependability of measurement  Stability over time 4–2 ©2011 Cengage Learning. All Rights Reserved. May not be scanned, copied or duplicated, or posted to a publicly accessible website, in whole or in part.

3. 4–3 FIGURE 4.1Summary of Hypothetical Test and Retest Results for 10 Applicants Taking the Programming Aptitude Test ©2011 Cengage Learning. All Rights Reserved. May not be scanned, copied or duplicated, or posted to a publicly accessible website, in whole or in part.

4. Errors of Measurement Errors of MeasurementErrors of Measurement  Factors that affect obtained scores but are not related to the characteristic, trait, or attribute being measured Obtained Score ComponentsObtained Score Components  X obtained = X true + X error  where –X obtained = obtained score for a person on a measure –X true = true score on the measure, that is, actual amount of the attribute measured that a person really possesses –X error = error score on the measure that is assumed to represent random fluctuations or chance factors. 4–4 ©2011 Cengage Learning. All Rights Reserved. May not be scanned, copied or duplicated, or posted to a publicly accessible website, in whole or in part.

5. Errors of Measurement True ScoreTrue Score  The mean or average score made by an individual on many different administrations of a measure if external and inter conditions were perfect Error ScoreError Score  The score made by individuals that is influenced by factors present at the time of measurement that distort individuals’ scores either over or under what they would have been on another measurement occasion 4–5 ©2011 Cengage Learning. All Rights Reserved. May not be scanned, copied or duplicated, or posted to a publicly accessible website, in whole or in part.

6. 4–6 ©2011 Cengage Learning. All Rights Reserved. May not be scanned, copied or duplicated, or posted to a publicly accessible website, in whole or in part. FIGURE 4.2Relationship Between Errors of Measurement and Reliability of a Selection Measure for Hypothetical Obtained and True Scores

7. TABLE 4.1Examples of Sources of Errors of Measurement Contributing to Unreliability in Human Resource Selection Measures 4–7 ©2011 Cengage Learning. All Rights Reserved. May not be scanned, copied or duplicated, or posted to a publicly accessible website, in whole or in part. NOTE: By use of the term selection measure, we mean both predictors and criteria.

8. 4–8 TABLE 4.1Examples of Sources of Errors of Measurement Contributing to Unreliability in Human Resource Selection Measures (cont’d) ©2011 Cengage Learning. All Rights Reserved. May not be scanned, copied or duplicated, or posted to a publicly accessible website, in whole or in part. NOTE: By use of the term selection measure, we mean both predictors and criteria.

9. Methods of Estimating Reliability Reliability CoefficientReliability Coefficient  An index that summarizes the estimated relationship between two sets of measures  The higher the coefficient, the higher the reliability estimate and the lower the error of measurement Principle Methods of Estimating ReliabilityPrinciple Methods of Estimating Reliability  Test-retest  Parallel or equivalent forms  Internal consistency  Interrater reliability estimates 4–9 ©2011 Cengage Learning. All Rights Reserved. May not be scanned, copied or duplicated, or posted to a publicly accessible website, in whole or in part.

10. Choosing a Method of Estimating Reliability How dependably can an individual be assessed with a measure at a given moment?How dependably can an individual be assessed with a measure at a given moment? How dependably will data collected by a measure today be representative of the same individual at a future time?How dependably will data collected by a measure today be representative of the same individual at a future time? How accurately will scores on a measure represent the true ability of an individual on the trait being sampled by a measure?How accurately will scores on a measure represent the true ability of an individual on the trait being sampled by a measure? When individuals are being rated by more than one rater, to what degree do evaluations vary from one rater to another? Or, to what extent is an individual’s score due to the rater rather than to the individual’s behavior or other characteristics being rated?When individuals are being rated by more than one rater, to what degree do evaluations vary from one rater to another? Or, to what extent is an individual’s score due to the rater rather than to the individual’s behavior or other characteristics being rated? 4–10 ©2011 Cengage Learning. All Rights Reserved. May not be scanned, copied or duplicated, or posted to a publicly accessible website, in whole or in part.

11. Test-Retest Reliability Estimates Test-Retest ReliabilityTest-Retest Reliability  The same measure is used to collect data from the same respondents at two different points in time  The reliability coefficient represents a coefficient of stability that indicates the extent to which the test can be generalized from one time period to the next  Factors that may affect test-retest reliability  Memory (overestimate)  Learning (underestimate) 4–11 ©2011 Cengage Learning. All Rights Reserved. May not be scanned, copied or duplicated, or posted to a publicly accessible website, in whole or in part.

12. 4–12 FIGURE 4.3Illustration of the Design for Estimating Test-retest Reliability ©2011 Cengage Learning. All Rights Reserved. May not be scanned, copied or duplicated, or posted to a publicly accessible website, in whole or in part. NOTE:Test-retest reliability for Time 1—Time 2 (Case A) = 0.94; test-retest reliability for Time 1—Time 2 (Case B) = 0.07.

13. Guidelines for Using Test-Retest Reliability Use test-retest reliability whenUse test-retest reliability when  the length of time between the two administrations is long enough to offset the effects of memory or practice  there is little reason to believe that memory will affect responses to a measure  it can be determined that nothing has occurred between the two testings that will affect responses  information is available on only a single item measure 4–13 ©2011 Cengage Learning. All Rights Reserved. May not be scanned, copied or duplicated, or posted to a publicly accessible website, in whole or in part.

14. Parallel or Equivalent Forms Reliability Estimates Parallel or Equivalent Forms StrategyParallel or Equivalent Forms Strategy  Administering two equivalent versions (forms with different items but assessing the same measure) of a measure to the same respondent group  Coefficient of equivalence—the computed reliability Pearson correlation estimate  Coefficient of equivalence and stability—the use of equivalent forms administered over time to account for the influence of random error to the test content (over equivalent forms) and transient error (across situations). 4–14 ©2011 Cengage Learning. All Rights Reserved. May not be scanned, copied or duplicated, or posted to a publicly accessible website, in whole or in part.

15. 4–15 ©2011 Cengage Learning. All Rights Reserved. May not be scanned, copied or duplicated, or posted to a publicly accessible website, in whole or in part. FIGURE 4.4Basic Measurement Procedures and Requirements for Developing Parallel Forms (A and B) of a Math Ability Test

16. Internal Consistency Reliability Estimates Internal Consistency Reliability EstimateInternal Consistency Reliability Estimate  An index of a measure that shows the extent to which all parts of a measure are similar in what they measure—that results are free from error due to the ways items or questions were phrased, interpreted by respondents, and so on Internal Consistency ProceduresInternal Consistency Procedures  Split-half reliability  Kuder-Richardson reliability  Cronbach’s coefficient alpha (α) reliability 4–16 ©2011 Cengage Learning. All Rights Reserved. May not be scanned, copied or duplicated, or posted to a publicly accessible website, in whole or in part.

17. 4–17 FIGURE 4.5 Representation of Odd-even (Day) Split-half Reliability Computed for a Job Performance Criterion Measure ©2011 Cengage Learning. All Rights Reserved. May not be scanned, copied or duplicated, or posted to a publicly accessible website, in whole or in part.

18. 4–18 ©2011 Cengage Learning. All Rights Reserved. May not be scanned, copied or duplicated, or posted to a publicly accessible website, in whole or in part. NOTE: 0 = incorrect response; 1 = correct response. FIGURE 4.6An Example of Data Used in Computing K-R 20 Reliability Coefficient

19. 4–19 FIGURE 4.7An Example of Applicant Data in Computing Coefficient Alpha (α) Reliability ©2011 Cengage Learning. All Rights Reserved. May not be scanned, copied or duplicated, or posted to a publicly accessible website, in whole or in part. NOTE: Applicants rate their behavior using the following rating scale: 1 = Strongly Disagree, 2 = Disagree, 3 = Neither Agree nor Disagree, 4 = Agree, and 5 = Strongly Agree. Case 1 coefficient alpha reliability = 0.83; Case 2 coefficient alpha reliability = 0.40.

20. Interrater Reliability Estimates Sources of Measurement ErrorSources of Measurement Error  What is being rated (e.g., employee behavior)  Who is doing the rating (rater characteristics) Procedures for Calculating Interrater ReliabilityProcedures for Calculating Interrater Reliability  Interrater Agreement  Percentage of rater agreement, Kendall’s coefficient of concordance ( W ), and Cohen’s kappa ( K )  Interclass Correlations  Pearson product-moment correlation ( r ) and Cohen’s weighted kappa ( K )  Intraclass Correlations 4–20 ©2011 Cengage Learning. All Rights Reserved. May not be scanned, copied or duplicated, or posted to a publicly accessible website, in whole or in part.

21. 4–21 FIGURE 4.8Example of a Research Design for Computing Intraclass Correlation to Assess Interrater Reliability of Employment Interviewers ©2011 Cengage Learning. All Rights Reserved. May not be scanned, copied or duplicated, or posted to a publicly accessible website, in whole or in part. NOTE: The numbers represent hypothetical ratings of interviewees given by each interviewer.

22. 4–22 TABLE 4.2Descriptive Summary of Major Methods for Estimating Reliability of Selection Measures ©2011 Cengage Learning. All Rights Reserved. May not be scanned, copied or duplicated, or posted to a publicly accessible website, in whole or in part.

23. 4–23 ©2011 Cengage Learning. All Rights Reserved. May not be scanned, copied or duplicated, or posted to a publicly accessible website, in whole or in part. TABLE 4.2Descriptive Summary of Major Methods for Estimating Reliability of Selection Measures (cont’d)

24. Interpreting Reliability Coefficients Reliability AnalysesReliability Analyses  help determine the dependability of data we will use in selection decision making  help in estimating the amount of error included in scores on any measure we choose to study What Does a Reliability Coefficient Mean?What Does a Reliability Coefficient Mean?  The extent (in percentage terms) to which individual differences in scores on a measure are due to “true” differences in the attribute measured and the extent to which they are due to chance errors 4–24 ©2011 Cengage Learning. All Rights Reserved. May not be scanned, copied or duplicated, or posted to a publicly accessible website, in whole or in part.

25. Reliability Coefficients Characteristics of Reliability CoefficientsCharacteristics of Reliability Coefficients  They are specific to the reliability estimation method and group on which they are calculated  They are a necessary but not a sufficient condition for validity  They are based on responses from a group of individuals  They are expressed by degree, and are determined ultimately by judgment 4–25 ©2011 Cengage Learning. All Rights Reserved. May not be scanned, copied or duplicated, or posted to a publicly accessible website, in whole or in part.

26. Factors Affecting Estimated Reliability Method of estimating reliability Stability Sample Individual differences among respondents Length of a measure Test question difficulty Homogeneity of content response format Administration and scoring 4–26 ©2011 Cengage Learning. All Rights Reserved. May not be scanned, copied or duplicated, or posted to a publicly accessible website, in whole or in part.

27. 4–27 FIGURE 4.9Rough Characterization of Reliability Methods in Terms of Generally Providing Upper- or Lower-bound Estimates ©2011 Cengage Learning. All Rights Reserved. May not be scanned, copied or duplicated, or posted to a publicly accessible website, in whole or in part.

28. 4–28 FIGURE 4.10Relation among Test Length, Probability of Measuring the True Score, and Test Reliability ©2011 Cengage Learning. All Rights Reserved. May not be scanned, copied or duplicated, or posted to a publicly accessible website, in whole or in part.

29. 4–29 FIGURE 4.11Illustration of the Relation Between Test Question Difficulty and Test Discriminability ©2011 Cengage Learning. All Rights Reserved. May not be scanned, copied or duplicated, or posted to a publicly accessible website, in whole or in part. Case 1: Item Difficulty = 0.10 (10% Get Test Question Correct) Case 2: Item Difficulty = 0.50 (50% Get Test Question Correct)

30. Standard Error of Measurement Standard Error of MeasurementStandard Error of Measurement  The estimated error in a particular individual’s score on the measure Calculating the Standard Error of MeasurementCalculating the Standard Error of Measurement 4–30 ©2011 Cengage Learning. All Rights Reserved. May not be scanned, copied or duplicated, or posted to a publicly accessible website, in whole or in part.

31. Standard Error of Measurement (cont’d) Uses of the Standard Error of MeasurementUses of the Standard Error of Measurement  Shows that scores are an approximation represented by a band or range of scores on a measure  Aids decision making in which only one individual is involved  Can determine whether scores for individuals differ significantly from one another  Helps establish confidence in scores obtained from different groups of respondents. 4–31 ©2011 Cengage Learning. All Rights Reserved. May not be scanned, copied or duplicated, or posted to a publicly accessible website, in whole or in part.

32. Standard Error of Measurement (cont’d) Guidelines for Interpreting Individual ScoresGuidelines for Interpreting Individual Scores  The difference between two individuals’ scores should not be considered significant unless the difference is at least twice the standard error of measurement of the measure.  Before the difference between scores of the same individual on two different measures should be treated as significant, the difference should be greater than twice the standard error of measurement of either measure. 4–32 ©2011 Cengage Learning. All Rights Reserved. May not be scanned, copied or duplicated, or posted to a publicly accessible website, in whole or in part.

33. Key Terms and Concepts Reliability Errors of measurement Obtained score True score Error score Reliability coefficient Test-retest Parallel or equivalent forms Internal consistency Interrater reliability estimates Internal consistency Split-half reliability Kuder-Richardson reliability Cronbach’s coefficient alpha (α) reliability Measurement error Interrater agreement Interclass agreement Intraclass agreement Reliability analyses Reliability coefficient Standard error of measurement 4–33 ©2011 Cengage Learning. All Rights Reserved. May not be scanned, copied or duplicated, or posted to a publicly accessible website, in whole or in part.