III Choosing the Right Method Chapter 10 Assessing Via Tests p235 Paper & Pencil Work Sample Situational Judgment (SJT) Computer Adaptive chapter 10 Assessing via Tests1

Terminology Test – “..objective and standardized procedure for measuring a psychological construct using a sample of behavior” – What are some examples? – Does an interview qualify as a test? Construct – “…fairly well developed idea of a trait.” – Most KSAOs – Kerlinger’s definitions Kerlinger’s definitions Give an example of a construct and operational definition using Kerlinger’s definitions –that’s IO related chapter 10 Assessing via Tests2

More terms Standardization – “…controlling conditions and procedures so scores among different people are comparable.” – What makes an interview a test rather than just an interview? – Does a test require a passive or active response (behavior) or can it be either? chapter 10 Assessing via Tests3

NORM-REFERENCED AND DOMAIN- REFERENCED TESTING – What’ the difference between the two? (domain-referenced is also call criterion-referenced) – What are some examples (or situations) where the purpose would require one or the other? (thought question) Hint: when criticality of performance is important or where characteristics of the applicant pool may vary geographically or over time. – Which type requires statistical (correlational) and which type requires judgmental validation? chapter 10 Assessing via Tests4

TRADITIONAL COGNITIVE TESTS – Discrimination and generalization i.e. recognize or discover relationships Know, perceive, remember, understand, cognitive manipulation Problem solving, evaluation of ideas – Compare and contrast – IQ, GMA, Cognitive ability Also can be considered aptitude and/ or achievement e.g. WPT with NFL players (do you think scores are related to performance?) chapter 10 Assessing via Tests5

Cognitive Ability Off the shelf – WPT WPT – WGCTA WGCTA – DAT (Differential aptitude Test) DAT (Differential aptitude Test) – What is a job you would use each of them for? – What are the relative advantages and disadvantages for developing homemade tests vs. commercially available ones? chapter 10 Assessing via Tests6

PERFORMANCE TESTS – How can they be both criteria and / or predictors? Give examples. Work Samples and Simulations – “standardized abstraction of the work” – Give hypothetical examples of a high and low fidelity simulation – Developing Work Samples to measure “proficiency” Would all or some of the tasks be measures for (1) selection v. (2) criterion? Situational Judgments – Can be either multiple choice or video (higher fidelity) With incremental validity over GMA, Personality, job experience (Chan & Schmitt, 02) chapter 10 Assessing via Tests7

Non-Cognitive Performance – Physical Abilities See recent PTC newsletter on legal issues Sometimes its difficult to set valid cut scores – Fitness Testing a two edged sword for setting standards and not doing so. What is the legal conundrum here? Should fitness testing be required for some jobs on a daily basis? Should drug testing be performed for some jobs and not other? – Sensory & Psychomotor Proficiencies chapter 10 Assessing via Tests8

COMPUTERIZED TESTING – What is a stenographer? Where did they go? – What are some psychometric problems with mass testing across different geographical locations that CAT can help with? – How can IRT help? Computerized Adaptive Testing (CAT) – What’s the difference between linear testing and branching algorithms? chapter 10 Assessing via Tests9

ISSUES AND CONTROVERSIES Setting Cut Scores –rarely recommended but often necessary: Civil service Licenses & certification Cyclical hiring (needs forecasting estimates, e.g. teachers) Sequential (for multiple hurdle) – Predicted Yield Model (Thorndike, ‘49) Fluctuation of candidate qualifications varies availability of openings varies Depends upon accurate forecasting (history and research) – What are some consideration ? – Regression-Based Methods (see fig 10.2 p254) chapter 10 Assessing via Tests10

Translations of Psychomotor Instruments – Cross cultural testing a challenge: Different approaches to testing Test administration problems Score equivalence – Considerations: Use IRT to establish item equivalence Develop “global” measures (Schmitt, Kihm & Robie, 2000) – What is this global thing? chapter 10 Assessing via Tests11

Tests and Controversy – Do we need licensing tests to establish credentials? – Do we need educational proficiency exams? – We will always assess; either With tests (more objective) With subjective judgment chapter 10 Assessing via Tests12