MGTO 324 Recruitment and Selections Validity II (Criterion Validity) Kin Fai Ellick Wong Ph.D. Department of Management of Organizations Hong Kong University of Science & Technology

Prologue A refreshment of some essential concepts –Three types of validity Content-related validity Construct-related validity Criterion-related validity Face validity??? –Some remarks Validation as a necessary step in test development Never be too rigorous about making distinctions among these three types of validity “All validation is one, and in a sense all is construct validation” (Cronbach, 1980)

Outline Criterion-related Validity Part 1: Defining Criterion Part 2: Assessing criterion validity Part 3: Applications Part 4: Incremental validity and multiple- regression analysis

Outline Criterion-related Validity Part 1: Defining Criterion Part 2: Assessing criterion validity Part 3: Applications Part 4: Incremental validity and multiple-regression analysis

Part I: Defining criterion What is criterion validity? –“Criterion-related evidence demonstrates that test scores are systematically related to one or more outcome criteria” (APA’s Standards, 1985, p. 11) –“Logically, the value of a criterion-related study depends on the relevance of the criterion measures that is used What is a criterion? –Usually it is a concrete behavior that could be directly observed In educational contexts: GPA for an academic screening test In organizational contexts: Sales Volume for a personnel screening test In social contexts: Divorce or not for a Marriage Satisfaction Test Sometimes, a criterion could be a psychological construct, e.g., job satisfaction, job commitment, creativity, etc.

Part I: Defining criterion So, what is the key difference between criterion- and construct-validity? –As Conbrach has argued, they are essentially the same Usually, we use construct validation for an early stage of test development Criterion validation usually refers to “How accurately can criterion performance be predicted from scores on the test?” –In this sense, criterion is a behavior (or sometimes a construct) that can be logically predicted by the test scores »In educational contexts: GPA for an academic screening test »In organizational contexts: Sales Volume for a personnel screening test »In social contexts: Divorce or not for a Marriage Satisfaction Test

Part I: Defining criterion Criterion-related validity –Collecting evidence to show that how well a test corresponds with a well-defined criterion –In other words, the appropriateness of using a test to predict another behavior (i.e., the criterion) –So, the criterion validity of a test can be revealed by the correlation between test scores and the corresponding criterion.

Part I: Defining criterion A quick test of your understanding –Test A is supposed to predict Criterion A The correlation between Test A and Criterion A is.38 –Test B is supposed to predict Criterion B The correlation between Test B and Criterion B is.74 –Test C is supposed to predict Criterion C The correlation between Test A and Criterion A is -.51 –Test D is supposed to predict Criterion D The correlation between Test B and Criterion B is -.88 –Please rank Tests A to D in descending order (from the highest validity to the lowest)

Part I: Defining criterion Criterion-related validity –Collecting evidence to show that how well a test corresponds with a well-defined criterion –In other words, the appropriateness of using a test to predict another behavior (i.e., the criterion) –So, the criterion validity of a test can be revealed by the correlation between test scores and the corresponding criterion.

Outline Criterion-related Validity Part 1: Defining Criterion Part 2: Assessing criterion validity Part 3: Applications Part 4: Incremental validity and multiple-regression analysis

Part II: Assessing criterion validity There are three types of criterion-related validity –Predictive validity Test scores are collected before the occurrence of the criterion –Concurrent validity Test scores and the criterion occur are simultaneously collected –Postdictive validity Test scores are collected after the occurrence of the criterion

Part II: Assessing criterion validity Predictive validity –present test scores are used to predict future performance or behaviors A-level results  university final GPA (Present)(Future) Test scores are collected before the occurrence of the criterion – Difficulties Attrition (fine-tuning) Time consuming

Part II: Assessing criterion validity Concurrent validity –The current test scores are used to predict the current performance or behaviors Food consumption  Health (Present) Learning ability test  Job performance(Present) –Why don’t we measure the criterion directly

Part II: Assessing criterion validity Postdictive validity –The current test scores are used to predict the past performance or behaviors Self esteem test  Childhood’s happiness (Present) (Past) Self esteem test  job satisfaction 5 years ago (Present) (Past) –Very rarely used

Part II: Assessing criterion validity Further Remarks about predictive validity –According to the APA’s Standards “Predictive studies are frequently, but not always, preferable to concurrent studies of selection tests for education or employment”; “concurrent evidence is usually preferable for acheivement tests, tests used for certification, diagnostic clinical tests, or for tests used as measures of a specified construct”

Part II: Assessing criterion validity Criterion-related validation –Using statistical methods –Relationship between the test and the criterion The goodness of prediction –Validity coefficient (r v )

Part II: Assessing criterion validity Criterion-related validation –Using statistical methods –Relationship between the test and the criterion The goodness of prediction –Validity coefficient (r v ) Correlation between the test score and the criterion Similar to reliability coefficient –Expressing the degree of validity Ranging from 0 to 1 –Usually around 0.3 – 0.4, seldom > 0.6 r 2 v represents the percentage of variation of the criterion explained by the regression equation of Y (criterion score) on X (test score)

Part II: Assessing criterion validity

Factors affecting criterion validity –Factor 1: Remember the formula pertaining to the relationship between reliability and validity? –Criterion validity depends on the reliability of the test as well as the reliability of the criterion

Part II: Assessing criterion validity Factors affecting criterion validity –Factor 2: The problem of range restriction –It occurs when either the criterion or the test has restricted range –Validity decreases as range of either variable decreases

Outline Criterion-related Validity Part 1: Defining Criterion Part 2: Assessing criterion validity Part 3: Applications Part 4: Incremental validity and multiple- regression analysis

Part III: Applications The application of the concept validity in personnel selection –Effects of validity on selection, or –By knowing the validity of a test, how can we recruit more successful employees? –See how the validity of a Extroversion scale is important in recruiting more successful salesman…

Part III: Applications Background information –Definition of a successful salesman Average sales volume per month > $10 million –Base rate The percentage of employees who would succeed if they are selected at random (i.e., without using Extroversion or tests’ results) 0.6 –Selection ratio The percentage of employees hired (i.e., hired employees / total applicants) 0.3 –Successful rate The percentage of employees who would succeed if they are selected based on the Extraversion scale

Part III: Applications Taylor & Russell (1939, Journal of Applied Psychology) A table that summaries how validity coefficient influences successful rate When the validity coefficient is 0 –Successful rate = base rate = 0.6 When the validity coefficient is 0.5 –Successful rate: 0.76 When the validity coefficient is 0.8 –Successful rate: 0.83 When the validity coefficient is 1 –Successful rate: 1

Outline Criterion-related Validity Part 1: Defining Criterion Part 2: Assessing criterion validity Part 3: Applications Part 4: Incremental validity and multiple- regression analysis

Part IV: Incremental validity and multiple regression analysis Incremental validity –In organization contexts, more tools will be used in addition to selection tests –So, evidence that a test is valid for particular inferences does not necessarily mean that the test is valuable Though a test may be reliable and valid, the decision to use it depends on the extent to which the test gives you more information than you could find if it were not used That is the unique information gained through using the test –For example, HKCEE results and IQ scores are highly correlated –Using HKCEE results per se might be enough »HKCEE results might have accounted for most variances explained by IQ »An IQ test becomes not useful, even though it is reliable and valid

Part IV: Incremental validity and multiple regression analysis How to assess incremental validity? –we can use multiple regression analysis It allows multiple predictors in the same regression (prediction) equation The unique effect of each predictor could be revealed by using this method I will show you the steps and interpretations of multiple regression analysis in the coming workshop

Cortina, et al. (2000). The incremental validity of interview scores over and above cognitive ability and conscientiousness scores. Personnel Psychology