Reliability Definition: The stability or consistency of a test. Assumption: True score = obtained score +/- error. ~ Domain Sampling Model ~ Item Domain Test

Test-retest method [error is due to changes occurring due to the passage of time] Issues: Length of time between test administrations if crucial (generally, the longer the interval, the lower the reliability) Memory Stability of the construct being assessed Speed tests, sensory discrimination, psychomotor tests (possible fatigue factor) Test Test r

Parallel/Alternate Forms [error due to test content and perhaps passage of time] Two types: 1) Immediate (back-to-back administrations) 2) Delayed (a time interval between administrations) Issues : Need same number & type of items on each test Item difficulty must be the same on each test Variability of scores must be the same on each test Test Test r

KR-20 and Coefficient Alpha [error due to item similarity] KR-20 is used with scales that have right & wrong responses (e.g., achievement tests) Alpha is used for scales that have a range of response options where there are no right or wrong responses (e.g., 7-point Likert-type scales R tt = k Σ p i (1 – p i ) k – 1 σ y 2 # of items variance of test scores % of those getting the item correct KR-20 ά = k 1 – Σ σ i 2 k – 1 σ y 2 # of items variance of test scores variance of scores on each item Alpha

Factors Affecting Reliability 1) Variability of scores (generally, the more variability, the higher the reliability) 2) Number of items (the more questions, the higher the reliability) 3)Item difficulty (moderately difficult items lead to higher reliability, e.g., p-value of.40 to.60) 4) Homogeneity/similarity of item content (e.g., item x total score correlation; the more homogeneity, the higher the reliability) 5) Scale format/number of response options (the more options, the higher the reliability)

Content Validity [ the extent to which test items represent the domain] a) Subject Matter Expert Opinions (e.g., CVR statistic) b)Internal consistency reliability c) Correlation with other similar tests Types of Validity

Predictive Correlation between test scores and performance scores after some time interval has passed (e.g., correlating test scores of applicants and subsequent performance scores collected 6 months to a year later) Range restriction issue on performance scores Time, cost, & pragmatic concerns Criterion-related Validity Concurrent Correlation between test scores and performance scores collected at the same time (e.g., correlating test scores with existing performance scores of employees) Motivation level Guessing, Faking Job experience factor Range restriction issue on performance scores Types of Validity (cont.)

Application Blanks (Weighted Application Blanks, Biographical Information Blanks) Honesty (Integrity) Tests, Drug Tests Interviews (e.g., Situational) Aptitude Tests (e.g., Mechanical, Clerical) Work Samples Assessment Centers (Situational Exercises) Types of Selection Tests

Application Blanks Content of items (use of job analysis) Number of application blanks (one for each position or job category) Legal issues Image of organization (e.g., format, recruitment issue, perceived fairness of questions ) Accuracy of data Education (e.g., 21%), salary (22%), job title (24%), years worked (29%) College students willing to include one lie on AB (95%); 45% had done so

Frequency of Common Inappropriate Application Blank Questions Item Not Appropriate Worded Appropriately Not Asked Past salary98.9%0%1.1% Minimum acceptable salary72.7%0%27.2% Reference source59.1%0%40.9% Age54.5%37.5%8.0% Information about relatives50.0%10.2%39.8% Conviction records43.2%28.4% Health40.9%2.3%56.8% Military service30.7% 38.6% Marital status27.3%0%72.7% Who to notify in case of emergency25.0%43.2%31.8% Length of time in residence23.9%0%76.1% Physical description, photo19.3%0%80.7% Rent or own car or home18.2%0%81.8% Handicap17.0%6.8%76.2% Organizational membership15.9%21.6%12.5% Work schedule13.6%63.6%22.7%

~ Honesty Testing ~ Types of Questions: 1)Frequency and extent of theft (e.g., What percentage of people take more than $1.00 per week from their employer?) 2) Punitiveness toward theft (e.g., should a person be fired if caught stealing $5.00?) 3) Thoughts about theft (e.g., Have you ever thought about taking company merchandise without actually taking any?) 4) Perceived ease of theft (e.g., How easy would it be for a dishonest person to steal from an employer?) 5) Likelihood of detection (e.g., What percent of employee thieves are ever caught? Validity Issues: a) Correlations with polygraph results b) Future behavior (e.g., # days with cash shortage, discharges) c) Admissions of past theft d) Shrinkage reduction e) Contrasted groups (e.g., scores by criminals vs. general population scores)

Drug Testing What does a positive drug test score indicate? Those being tested Testing procedure Type of company Type of test Type of job Applicants, employees or both Random or for cause Public or private Sensitivity, Cross-reactivity (TLC, EIA/RIA, GCMS) Safety concerns or not Some Issues:

~ Reference Checks ~ (Exceptionally common technique; e.g., 95% usage by organizations) In-Person (e.g., interview) Costly, time consuming Used in jobs that involve the concern for risks (e.g., security, $) Can elicit different types of information (differences between in-person and written reference information) Mail (or ) Low return rate (e.g., 56 – 64%) Standardized questions, format Written record of responses Ensure confidentiality of responses (signed statement by applicant)

~ Telephone Checks ~ (More frequently used than written references) Allows follow-up or clarification of answers given Less resistance to giving certain types of information can be collected Quick process Important data can be gleaned from various verbal cues (e.g., pauses, hesitations, voice inflections, voice level, intonations) Relatively high return rate Better responsiveness, more interactive nature of the method More confidence in the identity of responder

Sample Biographical Information Blank Items During high school, how many times did you make the honor roll? How much freedom or independence did your parents allow you in grade school? How important did your favorite high school teachers stress discipline in the classroom? How many times did you change schools before you were sixteen years old? Compared to other people in high school, how many friends did you have? How old were you when you spent your first week (or more) away from your parents? How bothered are you if you a job is left undone? How often do you read craft and mechanics magazines? How quickly do you normally work? How well do you feel you can understand the feelings of others? How well do you tolerate performing routine tasks?

~ Employment Interview ~ Frequently used to make selection decisions (over 90%) Social exchange (interpersonal) process Search for information

COMMON PROBLEMS WITH THE “TRADITIONAL” INTERVIEW Variety of Interviewer Biases * 1 st Impressions * Expectancy Effect * Contrast Effect * Stereotype Matching Different Questions Asked to Applicants (Lack of standardization) Disagreement on the Desirability of Interview Responses Little Formal Interviewer Training Subjective (or no) Scoring System Interview Conducted and Scored by One Person Poor Reliability, Validity, and Job Relevancy (Open to Legal Challenge)

Overview of Situational Interview Process 1)Perform a Job Analysis Using the Critical Incident Technique 2)Place Critical Incidents into Relevant Job Dimensions (e.g., Safety, Responsibility, Interpersonal Skills) 3)Reword Critical Incidents Into Question Format Incident: The employee was married for a year and a half and used any excuse to stay home. One day the employee’s children got colds and no one was around to care for them. So, the employee didn’t show up for work and didn’t phone in. Question: Your two teenage children are home in bed sick with colds. No friends or relatives are available to watch them. Your shift starts in two hours. What would you do in this situation? 4)Decide on the desirability of responses [Think of how good, average, and mediocre workers would have answered such a question] _____ _____ _____ _____ _____ Stay home Phone in & Go in, they just explain the problem have colds 5) Conduct interviews in groups of two or more. Each interviewer scores applicant independently. A single score is given after group discussion

Verbal Reasoning..… is to water as eat is to ….. A. continue drive B. foot enemy C. drink food D. girl industry E. drink enemy..... is to night as breakfast is to ….. A. supper corner B. gentle morning C. door corner D. flow enjoy E. supper morning ….. is to one as second is to ….. A. two middle B. first fire C. queen hill D. first two E. rain fire Differential Aptitude Test (DAT )

Numerical Ability Add A B C. 16 D. 26 N. none of these Add A B C. 25 D. 59 N. none of these Differential Aptitude Test (DAT )

Abstract Reasoning PROBLEM FIGURES ANSWER FIGURES A B C D E Differential Aptitude Test (DAT )

Abstract Reasoning (cont.) PROBLEM FIGURES ANSWER FIGURES A B C D E Differential Aptitude Test (DAT )

Mechanical Reasoning Which weighs more? (If equal, mark C.) AB Differential Aptitude Test (DAT )

Space Relations A B C D Differential Aptitude Test (DAT )

Spelling A) man B) Gurl C) Catt D) dog Language Usage 1) I just / left / my friends / house. A B C D 2) Ain’t we / going to / the office / next week? A B C D 3) I went / to a ball / game with / Jane. A B C D Differential Aptitude Test (DAT )

Clerical Speed and Accuracy V. AB AC AD AE AF W. aA aB BA Ba Bb X. A7 7A B7 7B AB Y. Aa Ba bA BA bB Z. 3A 3B 33 B3 BB Differential Aptitude Test (DAT )

~ Work Sample Tests ~ (performing a piece, or sample task, of the job)

~ Assessment Center Process ~ Candidates Participate in Situational Exercises Sample Individual Exercises Interview Simulation Scheduling Exercise In-Basket Sample Group Exercises Leaderless Group Discussion Business Game Observed, discussed, and scored on various dimensions (e.g., communication, decisionmaking, planning/organizational skills) by trained raters Overall scores computed and ranked for personnel decisions Indiviudaul dimension scores used for developmental purposes

Ability Tests Types of Selection Tests (cont). Sensory (e.g., hearing, vision) Motor (e.g., dexterity, strength, agility) Cognitive (e.g., Intelligence) ADA concerns Reasonable accommodation Essential job duties

Wonderlic Personnel Test

Extraversion --- Outgoing, sociable Neuroticism (Emotional Stability): Depressed, anxious, worrisome, etc. Agreeableness: Flexible, forgiving Conscientiousness: Careful, thorough, persevering Openness to Experience: Curious, imaginative ~ “Big 5” Personality Factors ~  Overall, conscientiousness and extraversion are best predictors of managerial performance across jobs  Personality measures add to prediction above and beyond other commonly used measure such as cognitive ability

~ Myers-Briggs Type Indicator ~ Introversion Extraversion (Internally motivated; enjoys spending time(Action-oriented; motivated enjoys alone; prefers one-to-one communication)by outside world and social interactions) Sensing Intuitive (Desires concrete information; practical;(Imaginative, creative factual and detailed in orientation) improvises comfortable with ambiguity; focuses on contexts and connections) Thinking Feeling (Searches for facts and logic in a decision situation (Sensitive to other’s needs, Seeks Focus on tasks and work to be accomplished consensus, does not like conflict) Easily able to provide an objective and critical analysis) Judging Perceiving Plans before acting, breaks tasks down Is comfortable to acts without planning, into subsets, uses deadlines to accomplish tasks Enjoys working on multiple tasks at the same time, enjoys flexibility in work. Self-Report Inventories

When you go somewhere for a day, would you rather: a)plan what you will do and when, or b)just go In a large group, do you more often: a)introduce yourself, or b)get introduced Is it harder for you to adapt to: a) routine, or b)constant change Do you think it is a worse fault to be a) unsympathetic b)unreasonable Sample Myers-Briggs Items Literal ______________________ Figurative Forgive _____________________ Tolerate Impulse _____________________ Decision

Holly (7) Very Clear Clear Moderate Slight Slight Moderate Clear Very Clear Steve (10) Bill (5) Thinking --- Feeling F Frank (1) Betty (19) T Paul (2) Sally (2) Thinking Looks at the logical consequences of actions; Examines pros and cons of approaches objectively; Energized by critique and analysis; Focuses on tasks Desires to find a standard to apply in all situations; reasonable and fair Feeling Focuses on how others may be affected by decisions by mentally placing themselves into situations; Guided by personal values; Energized by appreciating and supporting others; Focuses on interactions Strives for harmony and positive interactions

Types of Selection Tests (cont). Personality Inventories Self-reports -- (e.g., Minnesota Multiphasic Personality Inventory ( MMPI ), California Psychological Inventory ( CPI), Myers-Briggs Type Indicator ( MBTI), Personality Inventory ( HPI), NEO PI-R NEO ( assesses the 5-Factor Model consisting of: Neuroticism, Extraversion Openness, Agreeableness Conscientiousness ) MMPI CPI MBTI HPI NEO (e.g., Thematic Apperception Test ( TAT ), Rorschach Inkblot Test (RIT) TATRIT Projective Techniques --

Rorschach Inkblot Test Ten cards which bilateral and symmetrical inkblots Scoring --- Location: the part of the blot used (e.g., use of the whole blot, common or unusual detail) Determinants: form, color, shading, and movement) Content (human figures, animal figures, anatomical diagrams, inanimate objects

TAT pictures that depict a variety of social and interpersonal situations. Participants are requested to write or tell a story about each picture to the examiner (e.g., what happened, what Ten pictures are gender-specific; the others can be used with either sex. Use: To uncover internal conflicts, dominant drives, interests, and motives. Specific motives include the need for achievement, need for power, the need for intimacy, and problem-solving abilities. Thematic Apperception Test