Department of Industrial Psychology  Faculty of Economic and Management Sciences Nadia Brits Supervisor: Prof. Deon Meiring ACSG Conference 16 March 2011 EVOLUTION OF THE CONSTRUCT-VALIDITY DEBATE Department of Industrial Psychology University of Stellenbosch

TODAY’S JOURNEY SACKETT & DREHER (1982)DESIGN FIXES CONSEQUENCES OF INVALID CONSTRUCTS DESIGN FIXES: TRIED & TESTED ABANDON DIMENSIONS DON’T TAKE AWAY MY DIMENSIONS EPIC OF THE CV DEBATE CRITICS OF LANCE (2008) ACs AT A CROSSROADS

Sufficient evidence for CV exists (Arthur et al., 2000; Thornton & Gibbons, 2008) The validity of ACs is questioned CONSTRUCT VALIDITY PUZZLE (Lievens, Chasteen, Day & Christiansen, 2006) CONSTRUCT-RELATED VALIDITY PARADOX (Arthur, Day & Woehr, 2008) SO-CALLED CONSTRUCT VALIDITY PROBLEM (Howard, 1997) BACKGROUND

ORIGINS OF THE CONSTRUCT-VALIDITY DEBATE Sackett & Dreher (1982) Expectations: CONVERGENT VALIDITY DISCRIMINANT VALIDITY ratings cluster according to DIMENSIONS, not exercises (which became known as EXERCISE EFFECT) Similar research followed

CONSEQUENCES OF INVALID CONSTRUCTS SELECTION of applicants based on AC performance ratings FEEDBACK based on AC results WASTE OF TIME AND MONEY Practitioners provide MISLEADING services to companies who appoint them to design and run ACs

DESIGN FIXES DEFINITION of dimensions NUMBER of dimensions TRANSPARENCY of dimensions to candidates Behavioural CHECKLISTS Type of SCORING METHOD ASSESSOR training TYPE of assessor (Lievens & Klimoski, 2001; Gaugler et al., 1987) EXPERIENCED assessors (Kolk et al. 2002; Thornton & Rupp, 2005).

DESIGN FIXES: TRIED AND TESTED Design fixes show LITTLE, INSIGNIFICANT IMPROVEMENT Controlling assessor variance has only a MARGINAL EFFECTS on construct-validity EXERCISE EFFECTS STILL DOMINATE What else can we do?

ABANDONING DIMENSIONS Move from dimensions-based ACs to task-based AC Why? Recurring exercise effects Exercise factors show positive correlations with external performance criteria

DON’T TAKE AWAY MY DIMENSIONS useless to learn a task that the participant might never encounter again Human performance is multidimensional cannot capture the full complexity of a real job (novel, non- repetitive tasks) Only dimensions will allow generalisation of AC results Research supports dimensions (Connelly et al., 2008; Melchers & Konig, 2008; Bowler & Woehr, 2006) knowledge about exercise-based ACs still lacking (Lievens, 2008).

RESULT : 3 LINES OF RESEARCH EXERCISES and DIMENSIONS explained same amount of variance – 34% (Lievens & Conway, 2001) EXERCISE EFFECT were larger than dimension effects – 52% (Lance et al., 2004) EXERCISES explained most variance – 33%, DIMENSIONS ALSO explained substantial amount - 22% (Bowler & Woehr, 2006)

EPIC OF THE HEATED DEBATE LANCE (2008): “ACs do not work they way they are supposed to” NOT SUFFICIENT EVIDENCE for 3 requirements of CV Supports the EXERCISE-BASED MODEL DISCARD DESIGN FIXES Exercise effects represent cross-situational specificity in candidate performance, not method bias...TRUE VARIANCE

COMMENTS/CRITICS TOWARDS LANCE (2008) ignores evidence of both dimension and exercises account for variance in AC performance (Howard, 2008) design fixes should continue to be investigated (Schleicher et al., 2008; Howard, 2008; Arthur, Day & Woehr, 2008; Melchers & König, 2008) Lance received support for candidates’ inconsistent performance....true variance individuals use set of stable skills and can adjust and adapt KSA’s according to the situation Some people perform better than others in a specific exercise

ACs AT A CROSSROADS Persistent EXERCISE EFFECTS performance variability: more situation-specific (57%) than situation-consistent (43%) (Hoeft and Schuler’s (2001) Lack of consensus on solutions and future directions Walter Mischel (1968) Consistency in behaviour ONLY when situational factors are acknowledged and taken into account TRAIT-ACTIVATION THEORY (TAT)

person-situation interaction to explain behaviour on the basis of responses to trait-relevant cues found in situations. SITUATION STRENGTH: strong vs weak situations SITUATION RELEVENCE: A situation is considered relevant to a trait if it provides cues for the expression of trait- relevant behaviour

APPROACHES TO IMPLEMENT TAT Adapting the CONTENT of the exercise Influencing the INSTRUCTIONS of each exercise to guide participants what type of behaviour to show training ROLEPLAYERS on how to interact in order to elicit certain behaviours from participants LARGE NUMBER OF SHORTER EXERCISES to obtains samples of performance on a number of independent tasks

Halaand and Christiansen (2002) found stronger convergence of AC ratings GAP IN RESEARCH about effectiveness of TAT

TO SUMMARISE SACKETT & DREHER (1982)DESIGN FIXES CONSEQUENCES OF INVALID CONSTRUCTS DESIGN FIXES: TRIED & TESTED ABANDON DIMENSIONS DON’T TAKE AWAY MY DIMENSIONS EPIC OF THE CV DEBATE CRITICS OF LANCE (2008) ACs AT A CROSSROADS