18/2/2002J.Y.Y. Kwong1 L4 - Assessment Methods (II) SOC325 Behavior at work Ref: Spector, Ch. 2 (p.32-36)

18/2/2002J.Y.Y. Kwong2 Outline: 1. Different types of assessment methods for personnel selection Interviews Biographical information Work sample Psychological tests Assessment center 2. Assessing test reliability and validity

18/2/2002J.Y.Y. Kwong3 To have an accurate assessment of eye- sight, we need a good optician: How do you pick one? How to pick a good assessment test? What it takes to be a GOOD test?

18/2/2002J.Y.Y. Kwong4 A scientific way to evaluate a test: A good test should demonstrate: High reliability Evidence for its validity

18/2/2002J.Y.Y. Kwong5 168 lb Reliability A test is reliable when it gives consistent measurements across repeated observations. 170 lb 1 min. later You read 175 lb You read NO! I’m 168 lb only! He read Hey! It’s not reliable!

18/2/2002J.Y.Y. Kwong6 Reliability Test-retest reliability: A test has high test-retest reliability when it gives consistent measurement across times. 160 lb 1 min. later

18/2/2002J.Y.Y. Kwong7 Reliability Inter-rater reliability: A test has high inter-rater reliability when different users (raters) using it give consistent measurements of the same object.

18/2/2002J.Y.Y. Kwong8 Reliability Internal consistency reliability: For tests that use multiple items to give a measurement Each item should be consistently measuring the same variable Overall, I am satisfied with the teacher’s performance. v.s. 1. The teacher has structured the presentation well. 2. The teacher encouraged students’ participation. 3. The size of students’ desk was suitable for writing. 4. The teacher was responsive to students’ questions.

18/2/2002J.Y.Y. Kwong9 Reliability Internal consistency reliability: A test has high ICR when multiple items give similar or consistent measurement. 1. The teacher was structured the presentation well. 2. The teacher encouraged students’ participation. 3. The size of students’ desk was suitable for writing. 4. The teacher was responsive to students’ questions.

18/2/2002J.Y.Y. Kwong10 A good test has high reliability: Test-retest – across time (same user; same target) Inter-rater – across raters/users (same target; same time) Internal consistency reliability – across items (same target; same user; same time)

18/2/2002J.Y.Y. Kwong11 The concepts of validity TargetInstrument LengthIncome HealthRuler IntelligenceO-P scale Optimism Blood Pressure Social Status IQ test

18/2/2002J.Y.Y. Kwong12 Validity Is the tool appropriate to measure the KSAO that we are interested in? A test can be reliable but not valid! (Ruler – Optimism) A test is valid when there is evidence that the test score reflects what the test is supposed to measure How do we know whether a ruler or O-P scale really measures optimism? 3 types of evidence Face; Content; Criterion-related validity

18/2/2002J.Y.Y. Kwong13 Validity Face validity : the test looks like what it was designed to measure (subjective). 劃一個人, 一間屋, 一條路 測你的人際關係能力. O-P 我很少會感到沮喪 我很快便忘記不愉快的事情 面對困難時, 我仍能保持樂觀

18/2/2002J.Y.Y. Kwong14 Validity Content validity : the test adequately covers the content of the variable (subjective). 你有成為好男友的質素嗎 ? 別人曾讚頌你清潔整齊 你是個有耐性的聆聽者 你每天有梳理頭髮的習慣 你不挖鼻子 O-P 我很少會感到沮喪 我很快便忘記不愉快的事情 面對困難時, 我仍能保持樂觀

18/2/2002J.Y.Y. Kwong15 Validity Criterion-related validity : IQ test score relates to academic performance, speed of learning new words Standford Binet IQ test: Hi score more ‘A’ grades than low score 占卜 ( 有橫財 ) – 買六合彩 There is evidence for CR validity when the test score relates to another variable which is theoretically related (objective).

18/2/2002J.Y.Y. Kwong16 Validity Criterion-related validity : Examples O-P Hi score in O-P less likely to be depressed; low score more likely to be Psychological assessment scores should relate to job performance! Higher Ability tests score – higher performance rating

18/2/2002J.Y.Y. Kwong17 A good test has evidence for its validity: Face – the test “looks like” what it intends to measure (Picture drawing – interpersonal skills) Content – the content of the test has adequate coverage of the variable ( 整齊清潔 = 好男友？ ) Criterion-related – test score relates to another theoretically related variable (IQ – “A”; Ability test score – job performance)

18/2/2002J.Y.Y. Kwong18 Typical Criterion-related Validity Coefficients Ager =.00 Handwriting analysis (Graphology).00 Vocational interests tests.10 College grades.13 Reference checks.17 Experience.18 Personality tests.24 Assessment centers.25 Structured interviews.34 Biographical information blanks.35 Job knowledge tests.50 Cognitive and psychomotor ability tests.53

18/2/2002J.Y.Y. Kwong19 How to pick a good assessment test? Ask for information about the test’s Reliability Validity JA  Task & KSAOs  Pick some reliable and valid assessment tests  Now, we know how to assess the job applicants! NEXT, recruitment and selection