“… the development of a valid test requires multiple procedures, which are employed at different stages of test construction … The validation process begins.

Slides:



Advertisements
Similar presentations
Agenda Levels of measurement Measurement reliability Measurement validity Some examples Need for Cognition Horn-honking.
Advertisements

Measurement Concepts Operational Definition: is the definition of a variable in terms of the actual procedures used by the researcher to measure and/or.
 Degree to which inferences made using data are justified or supported by evidence  Some types of validity ◦ Criterion-related ◦ Content ◦ Construct.
Cal State Northridge Psy 427 Andrew Ainsworth PhD
© 2006 The McGraw-Hill Companies, Inc. All rights reserved. McGraw-Hill Validity and Reliability Chapter Eight.
Part 4 Staffing Activities: Selection
VALIDITY AND RELIABILITY
MGT-491 QUANTITATIVE ANALYSIS AND RESEARCH FOR MANAGEMENT
General Information --- What is the purpose of the test? For what population is the designed? Is this population relevant to the people who will take your.
Managing Human Resources, 12e, by Bohlander/Snell/Sherman © 2001 South-Western/Thomson Learning 5-1 Managing Human Resources Managing Human Resources Bohlander.
Chapter 4A Validity and Test Development. Basic Concepts of Validity Validity must be built into the test from the outset rather than being limited to.
Chapter 4 Validity.
VALIDITY.
Developing a Hiring System Reliability of Measurement.
Concept of Reliability and Validity. Learning Objectives  Discuss the fundamentals of measurement  Understand the relationship between Reliability and.
OS 352 2/28/08 I. Exam I results next class. II. Selection A. Employment-at-will. B. Two types of discrimination. C. Defined and methods. D. Validation.
1 The Employment Interview. 2 One of the most widely used hiring tools in the public and private sector Used for selection and recruiting Involve an interaction.
PART TWO EMPLOYMENT Chapters 5-7.
Part 5 Staffing Activities: Employment
Validity of Selection. Objectives Define Validity Relation between Reliability and Validity Types of Validity Strategies.
Chapter 7 Correlational Research Gay, Mills, and Airasian
Chapter 7 Evaluating What a Test Really Measures
Classroom Assessment A Practical Guide for Educators by Craig A
Validity Lecture Overview Overview of the concept Different types of validity Threats to validity and strategies for handling them Examples of validity.
Understanding Validity for Teachers
Test Validity S-005. Validity of measurement Reliability refers to consistency –Are we getting something stable over time? –Internally consistent? Validity.
Measurement Concepts & Interpretation. Scores on tests can be interpreted: By comparing a client to a peer in the norm group to determine how different.
Mgmt Staffing Prof. Howard Miller. Staffing Function  Among several human resource functions  Benefits  Compensation  Safety  Labor Relations.
6-1 McGraw-Hill/IrwinCopyright © 2009 by The McGraw-Hill Companies, Inc. All Rights Reserved. fundamentals of Human Resource Management 3 rd edition by.
Torrington, Hall & Taylor, Human Resource Management 6e, © Pearson Education Limited 2005 Slide 7.1 Importance of Selection The search for the perfect.
Ch 6 Validity of Instrument
Instrumentation.
Foundations of Educational Measurement
McMillan Educational Research: Fundamentals for the Consumer, 6e © 2012 Pearson Education, Inc. All rights reserved. Educational Research: Fundamentals.
Technical Adequacy Session One Part Three.
Foundations of Recruitment and Selection I: Reliability and Validity
Standardization and Test Development Nisrin Alqatarneh MSc. Occupational therapy.
MGTO 324 Recruitment and Selections Validity II (Criterion Validity) Kin Fai Ellick Wong Ph.D. Department of Management of Organizations Hong Kong University.
Managing Human Resources, 12e, by Bohlander/Snell/Sherman © 2001 South-Western/Thomson Learning 5-1 Managing Human Resources Managing Human Resources Bohlander.
Validity Is the Test Appropriate, Useful, and Meaningful?
6. Evaluation of measuring tools: validity Psychometrics. 2012/13. Group A (English)
Part 5 Staffing Activities: Employment
Measurement Validity.
 Job evaluation is the process of systematically determining the relative worth of jobs to create a job structure for the organization  The evaluation.
6-1 McGraw-Hill/IrwinCopyright © 2009 by The McGraw-Hill Companies, Inc. All Rights Reserved. fundamentals of Human Resource Management 3 rd edition by.
~ Test Construction and Validation ~
Copyright © Allyn & Bacon 2008 Intelligent Consumer Chapter 14 This multimedia product and its contents are protected under copyright law. The following.
Validity and Item Analysis Chapter 4. Validity Concerns what the instrument measures and how well it does that task Not something an instrument has or.
Validity and Item Analysis Chapter 4.  Concerns what instrument measures and how well it does so  Not something instrument “has” or “does not have”
Experimental Research Methods in Language Learning Chapter 5 Validity in Experimental Research.
Uniform Guidelines on Employee Selection Uniform Framework for employment decisions -- apply only to selection procedures for employment decisions Discrimination.
Psychometrics. Goals of statistics Describe what is happening now –DESCRIPTIVE STATISTICS Determine what is probably happening or what might happen in.
Criteria for selection of a data collection instrument. 1.Practicality of the instrument: -Concerns its cost and appropriateness for the study population.
Measurement MANA 4328 Dr. Jeanne Michalski
©2011 Cengage Learning. All Rights Reserved. May not be scanned, copied or duplicated, or posted to a publicly accessible website, in whole or in part.
CHAPTER 6 Selecting Employees and Placing Them in Jobs
Test Validity “… the development of a valid test requires multiple procedures, which are employed at different stages of test construction … The validation.
Chapter 6 - Standardized Measurement and Assessment
VALIDITY, RELIABILITY & PRACTICALITY Prof. Rosynella Cardozo Prof. Jonathan Magdalena.
VALIDITY by Barli Tambunan/
Reliability and Validity in Research
Concept of Test Validity
Test Validity.
Evaluation of measuring tools: validity
Week 3 Class Discussion.
Reliability and Validity of Measurement
5 6 Selecting Employees C H A P T E R Training Employees
Job Analysis CHAPTER FOUR Screen graphics created by:
Cal State Northridge Psy 427 Andrew Ainsworth PhD
Chapter 8 VALIDITY AND RELIABILITY
Presentation transcript:

“… the development of a valid test requires multiple procedures, which are employed at different stages of test construction … The validation process begins with the formulation of detailed trait or construct definitions … Test items are then prepared to fit the construct definitions. Empirical item analyses follow … Other appropriate internal analyses may then be carried out including factor analyses of item clusters or subtests … The final stage includes validation and cross-validation of various scores and interpretive combinations of scores through statistical analyses against external, real-life criteria.” (Anastasi, 1986, p.3) Almost any information gathered in the process of developing or using a test is relevant to its validity … If we think of test validity in terms of understanding what a particular test measures, it should be apparent that virtually any empirical data obtained with the test represent a potential source of validity information.” (Anastasi, 1986, p.3) Test Validity

Types of Validity Content Validity [ the extent to which test items represent a domain] a) Subject Matter Expert Opinions (e.g., CVR statistic) b) Internal consistency reliability c) Correlation with other similar tests Content relevance Domain specification Content coverage Domain representativeness

1) Perform a job analysis Description of job tasks Rating of job tasks on various criteria Specification of KSAs Rating of KSAs on various criteria Link/connect tasks to KSAs From SIOP Principles: “The characterization of the work domain should be based on accurate and thorough information about the work including analysis of work behaviors and activities, responsibilities of the job incumbents, and/or the KSAOs prerequisite to effective to effective performance on the job. The researcher should indicate what important work behaviors, activities, and worker KSAOs are included in the domain, describe how the content of the domain is linked to the selection procedure, and explain why certain parts of the domain were or were not included in the selection procedure.” (p. 22) 2) Selection of SMEs From SIOP Principles: “ The success of the content-based study is closely related to the qualifications of the subject matter experts (SMEs) … The experts should have thorough knowledge of the work behaviors and activities, responsibilities of job incumbents, and the KSAOs prerequisite to effective to effective performance on the job” (p. 22) 3) Writing (or choosing) and evaluation of selection test items Steps in a Content Validation Effort

TASK -- KSA MATRIX To what extent is each KSA needed when performing each job task? 5 = Extremely necessary, the job task cannot be performed without the KSA 4 = Very necessary, the KSA is very helpful when performing the job task 3 = Moderately necessary, the KSA is moderately helpful when performing the job task 2 = Slightly necessary, the KSA is slightly helpful when performing the job task 1 = Not necessary, the KSA is not used when performing the job task KSAABCDEFGHIJKLMNOPQR Job Tasks

Sample Item Rating Form Connect each item to a KSA or two Rate difficulty of each item (5- point scale) relative to the level of KSA needed in the job)

Content Validity Issues Are the job activities and requirements stable across time? Does successful performance on the test require the same KSAs as successful performance on the job ? Is the type (or mode) of testing procedure the same as that required on the job? Do some KSAs not required on the job exist on the test?* (Stutts v. Freeman (1983) GATB test scores, dyslexia (manual job) Limited usefulness when abstract constructs are being measured (a small inferential leap is required between the test content and job requirements)*** From Anastasi (1986): “When tests are designed for use within special contexts, the relevant constructs are usually derived from content analysis of particular behavior domains” (p. 7). From SIOP Principles: “ When selection procedure content is linked to job content, content-oriented strategies are useful. When selection procedure content is less clearly linked to job content, other sources of validity evidence take precedence” (p. 23).

Section 1607.C(1) of the Uniform Guidelines “Appropriateness of content validity studies” A selection procedure based on inferences about mental processes cannot be supported solely or primarily on the basis of content validity. Thus, a content strategy is not appropriate for demonstrating the validity of selection procedures which purport to measure traits or constructs such as intelligence, aptitude, personality, common sense, judgment, leadership and spatial ability SIOP Principles stress a “unitarian” perspective where ANY EVIDENCE of validity supports inferences of job relatedness regardless of method used Guardians v. Civil Service (1980) – rank ordering of scores based on content evidence 1. Suitable job analysis 2. Reasonable competence in test construction 3. Test content related to job content 4. Test content representative of job content 5. Scoring systems selecting applicants who are better job performers

Predictive Correlation between test scores and performance scores after some time interval has passed (e.g., correlating test scores of applicants and subsequent performance scores collected 6 months to a year later) Range restriction issue on performance scores Time, cost, & pragmatic concerns Criterion-related Validity Concurrent Correlation between test scores and performance scores collected at the same time (e.g., correlating test scores with existing performance scores of employees) Motivation level Guessing, Faking Job experience factor Range restriction issue on performance scores Types of Validity (cont.)

Criterion-related Validity Issues A)Job Stability B) Reliable and relevant measure of job performance From SIOP Principles: “A relevant, reliable, and uncontaminated criterion(s) must be obtained or developed. Of these characteristics, the most important is relevance. A relevant criterion is one that reflects the relative standing of employees with respect to important work behavior(s) or outcome measure(s). If such a criterion measure does not exist or cannot be developed, use of a criterion-related validation strategy is not feasible (p. 14). C)Use of a representative sample of people and jobs D) Large sample (on predictor and criterion) From SIOP Principles: “A competent criterion-related validity study should be based on a sample that is reasonably representative of the work and candidate pool … A number of factors related to statistical power can influence the feasibility of a criterion-related study. Among these factors are the degree (and type) of range restriction in the predictor or the criterion, reliability of the criterion, and statistical power (p. 14)

Legal Issues and Criterion-related Validity Court focus on the content of measures as opposed to criterion validity evidence (relationship between test cores and job performance) Emphasis on the legal history of tests Criterion-validity emphasis versus concurrent validity designs Statistical significant relationships are not always acceptable (consideration of other factors such as test utility)

Reliability of both the criterion (job performance) and the predictor (test) Restriction of range (on both the test and job performance measure) Contamination of the criterion (e.g., measure of job performance is affected by other variables rather than one’s ability or knowledge) Factors Affecting the Validity Coefficient [correlation between a test and job performance] Standard error of estimate (validity coefficient):  y’ =  y 1 - r 2 xy  y = standard deviation of y (criterion) r 2 xy = correlation between x and y squared

Correction for Attenuation Observed validity coefficient  T =    x y xy 0 yy Criterion reliability Validity coefficient   = S S 1 1  of unrestricted sample  of restricted sample 1 -  +  2 S 2 1 S 2 1 (Predictor) 1 - (1 -  ) 2 S S (Criterion)  

Selection Ratio (SR) = n N # Job openings # Applicants Test Validity [Criterion-related]: The extent to which test scores correlate with job performance scores [Range is from 0 to 1.0] Test Utility Key Points

Proportion of “Successes” Expected Through the Use of Test of Given Validity and Given Selection Ratio, for Base Rate.60. (From Taylor & Russell, 1939, p. 576) Selection Ratio Validity Note: A full set of tables can be found I Taylor and Russell (1939) and in McCormick and Ilgen (1980, Appendix B). (SR)

Mean Standard Criterion Score of Accepted Cases in Relation to Test Validity and Selection Ratio (From Brown & Ghiselli, 1953, p. 342) Validity Coefficient Selection Ratio Selection Ratio Example

N s r xy SD y Z x – N T (C) # of applicants selected validity coefficient standard deviation of job performance in dollar terms average score on the selection procedure of those selected (standard score) number of applicants assessed cost of assessing each applicant Example of Brogden and Cronbach & Gleser Models

Types of Validity (cont.) Construct Validity [extent to which a test assesses the construct it intends to measure] Correlation between scores measuring a construct (e.g., anxiety) with one method (e.g., paper & pencil) with scores on the same construct using a different method (e.g., interview) [Convergent validation] Correlation between scores measuring a construct (e.g., anxiety) using one method (e.g., paper & pencil) with scores on a different construct (e.g., leadership) assessed with a different method (e.g., interview) [Discriminant validation] “Construct validation is indeed a never-ending process. However, that should not preclude using the test operationally to help solve practical problems and reach real-life decisions as soon as the available validity information has reached an acceptable level for a particular application. This level varies with the type of test and the way it will be used. Establishing this level requires informed professional judgment within the appropriate specialty of professional practice.” (Anastasi, p.4)

Method 1 (Paper & Pencil) Method 2 (Clinical Interview) Method 3 (Peer observation) Traits A B C Method 1 (Paper & Pencil) A (B oredom) B (Dep) C (Anxiety) Method 2 (Clinical Interview) A (B oredom) B (Dep) C (Anxiety) Method 3 (Peer observation) A (B oredom) B (Dep) C (Anxiety) Reliability Figures Mono-Trait; Hetero-Method Hetero-Trait; Mono Method Hetero-Trait; Hetero-Method

We suggest the Guidelines as a high-priority for revision because we believe the regulatory standards should consider contemporary scientific research and practice. Professional associations like SIOP, APA, AERA, and NCME have documented these advances in scholarly literature and in technical authorities like the Principles and Standards. Unfortunately, there are inconsistencies between the Guidelines and some scholarly literature related to validation research and the use of employee selection procedures, and between the Guidelines and other technical authorities. These inconsistencies create substantial ambiguity for employers that use employee selection procedures, as well as for federal agencies and the courts when determining whether a selection procedure is job-related. Consideration of contemporary research and scientifically supported recommendations will help clarify the standards for valid selection procedures. SIOP Position on Uniform Guidelines

PR Nightmare (e.g., UG goal is to prohibit discrimination in employment) Much content of UG are not related to scientific practices./research/ Why redo the entire guidelines? What would replace the UG? Who would be involved in doing this (e.g., stakeholders)? How about the competing interests of various groups? Case law from UG would still exist (SIOP ought to focus on impacting court decisions – position papers, lobbying efforts. litigation) “Search for alternatives” is part of the UG and has lead to many advancements in selection research Outtz Summary of Uniform Guidelines (UG)