1. Exploring the Equivalence and Rater Bias in AC Ratings Prof Gert Roodt – Department of Industrial Psychology and People Management, University of Johannesburg Sandra Schlebusch – The Consultants ACSG Conference 17 – 19 March 2010

2. Presentation Overview  Background and Objectives of the Study  Research Method  Results  Discussion and Conclusions  Recommendations

3. BackgroundBackground Construct Validity has long been a Problem in ACs (Jones & Born, 2008) Perhaps the Mental Models that the Raters use are Part of the Problem However, other Factors that Influence Reliability Should not be Neglected

4. Background Continued To Increase Reliability Focus On all aspects of the Design Model (Schlebusch & Roodt, 2007):  Analysis  Design  Implementation o Context o Participants: o Process Owners (Simulation Administrator; Raters; Role-players)

5. Background Continued  Analysis (International Guidelines, 2009) o Competencies / Dimensions oAlso Characteristics of Dimensions (Jones & Born, 2008) o Situations o Trends/Issues in Organisation o Technology

6. Background Continued  Design of Simulations o Fidelity o Elicit Behaviour o Pilot

7. Background Continued  Implementation o Context:  Purpose o Participants o Simulation Administration (Potosky, 2008)  Instructions  Resources  Test Room Conditions

8. Background Continued  Raters  Background  Characteristics  “What are Raters Thinking About When Making Ratings?” (Jones & Born, 2008)

9. Sources of Rater Bias  Rater Differences (background; experience, etc.)  Rater Predisposition (attitude; ability; knowledge; skills, etc.)  Mental Models

10. Objective of the Study The Focus of this Study is on Equivalence and Rater Bias in AC Ratings More specifically on:  Regional Differences  Age Differences  Tenure Differences  Rater Differences

11. Participants (Ratees) Region Research Method Region FrequencyPercent Valid Percent Cumulative Percent Valid Western36834.8 Central53750.8 85.6 Eastern15214.4 100.0 Total1057100.0

12. Participants (Ratees) Age Research Method (cont.) Age (Recode) FrequencyPercent Valid Percent Cumulative Percent Valid 30 years or less 11510.912.3 31 - 40 years21720.523.335.6 41 - 50 years26825.428.764.3 51 years or older 33331.535.7100.0 Total93388.3100.0 MissingSystem12411.7 Total1057100.0

13. Participants (Ratees) Tenure Research Method (cont.) Years of Service (Recode) FrequencyPercent Valid Percent Cumulative Percent Valid 10 years or less 36334.338.9 11 - 20 years10610.011.450.3 21 - 30 years19618.521.071.3 31 years or more 26825.428.7100.0 Total93388.3100.0 MissingSystem12411.7 Total1057100.0

14. Research Method (cont.) Measurement: In-Basket Test Measuring Six Dimensions: Initiative; Information Gathering; Judgement; Providing Direction; Empowerment; Management Control Overall In-Basket Rating

15. Research Method (cont.) Procedure: Ratings were Conducted by 3 Raters on 1057 Ratees Observer (Rater) FrequencyPercentValid Percent Cumulative Percent Valid 137035.0 237835.8 70.8 330929.2 100.0 Total1057100.0

16. Initiative ResultsResults FrequencyPercentValid PercentCumulative Percent Valid 0 615.8 ND 96291.0 96.8 R 343.2 100.0 Total 1057100.0

17. Results (cont.) Initiative Reliability Statistics: Initiative Cronbach's Alpha N of Items.5564 Reliability Statistics: Initiative ObserverCronbach's AlphaN of Items 1.7854 2.6104 3.6214

18. Results (cont.) Information Gathering FrequencyPercent Valid Percent Cumulative Percent Valid 0615.8 ND99093.7 99.4 R6.6 100.0 Total1057100.0

19. Results (cont.) Information Gathering Reliability Statistics: Information Gathering Cronbach's AlphaN of Items.4853 Reliability Statistics: Information Gathering ObserverCronbach's AlphaN of Items 1.6033 2.3553 3.4533

20. Results (cont.) Judgement FrequencyPercentValid PercentCumulative Percent Valid 0625.9 ND54451.5 57.3 R34632.7 90.1 E1059.9 100.0 Total1057100.0

21. Results (cont.) Judgement Reliability Statistics: Judgement Cronbach's AlphaN of Items.8135 Reliability Statistics: Judgement ObserverCronbach's AlphaN of Items 1.9005 2.6705 3.7705

22. Results (cont.) Providing Direction FrequencyPercent Valid Percent Cumulative Percent Valid 0625.9 ND77673.4 79.3 R12912.2 91.5 E726.8 98.3 HE181.7 100.0 Total1057100.0

23. Results (cont.) Providing Direction Reliability Statistics: Providing direction Cronbach's AlphaN of Items.7455 Reliability Statistics: Providing direction ObserverCronbach's AlphaN of Items 1.7915 2.4785 3.7425

24. Results (cont.) Empowerment FrequencyPercent Valid Percent Cumulative Percent Valid 0625.9 ND54751.8 57.6 R25023.7 81.3 E16315.4 96.7 HE353.3 100.0 Total1057100.0

25. Results (cont.) Empowerment Reliability Statistics: Empowerment Cronbach's AlphaN of Items.7493 Reliability Statistics: Empowerment ObserverCronbach's AlphaN of Items 1.7723 2.7823 3.7643

26. Results (cont.) Control FrequencyPercent Valid Percent Cumulative Percent Valid 0615.8 ND81176.776.882.6 R12611.9 94.5 E383.6 98.1 HE201.9 100.0 Total105699.9100.0 MissingSystem1.1 Total1057100.0

27. Results (cont.) Control Reliability Statistics: Control Cronbach's AlphaN of Items.7485 Reliability Statistics: Control ObserverCronbach's AlphaN of Items 1.7885 2.6745 3.7575

28. Results (cont.) Overall In-Basket Rating Reliability Statistics: In-basket Cronbach's AlphaN of Items.7686 Reliability Statistics: In-basket ObserverCronbach's AlphaN of Items 1.8696 2.7136 3.6956

29. Results (cont.) Regional Differences Robust Tests of Equality of Means Statistic(a)df1df2Sig. InitiativeBrown-Forsythe11.5672631.991.000 Info GatheringBrown-Forsythe14.7552789.232.000 JudgementBrown-Forsythe12.0652625.270.000 Providing Direction Brown-Forsythe6.9902482.067.001 EmpowermentBrown-Forsythe9.2052566.078.000 ControlBrown-Forsythe3.7762484.448.024 In-BasketBrown-Forsythe10.8762621.425.000 a Asymptotically F distributed.

30. Results (cont.) Age Differences Robust Tests of Equality of Means Statistic(a)df1df2Sig. InitiativeBrown-Forsythe6.0023770.593.000 Info GatheringBrown-Forsythe6.4783913.407.000 JudgementBrown-Forsythe12.0743710.775.000 Providing Direction Brown-Forsythe4.1293725.823.006 EmpowermentBrown-Forsythe2.5713752.519.053 ControlBrown-Forsythe1.3233856.726.266 In-BasketBrown-Forsythe7.5323828.234.000 a Asymptotically F distributed.

31. Results (cont.)- tenure Tenure differences ANOVA Sum of SquaresdfMean SquareFSig. Initiative Between Groups.6733.2242.810.038 Within Groups74.197929.080 Total74.870932 Info Gathering Between Groups.4573.1523.073.027 Within Groups45.996929.050 Total46.452932 Judgement Between Groups8.43832.8135.197.001 Within Groups502.735929.541 Total511.173932 Providing Direction Between Groups2.5393.8461.597.189 Within Groups492.473929.530 Total495.012932 Empowerment Between Groups3.05631.0191.179.317 Within Groups802.629929.864 Total805.685932 Control Between Groups1.0673.356.794.498 Within Groups415.966928.448 Total417.033931 In-Basket Between Groups1.4163.4722.560.054 Within Groups171.258929.184 Total172.674932

32. Results (cont.) Rater Differences Robust Tests of Equality of Means Statistic(a)df1df2Sig. InitiativeBrown-Forsythe.80921001.085.446 Info GatheringBrown-Forsythe2.77221014.735.063 JudgementBrown-Forsythe6.83921041.068.001 Providing Direction Brown-Forsythe15.9922771.235.000 EmpowermentBrown-Forsythe24.9002843.165.000 ControlBrown-Forsythe3.7402837.540.024 In-BasketBrown-Forsythe7.7842945.444.000 a Asymptotically F distributed.

33. Results (cont.) Post Hoc Tests: Judgement Multiple Comparisons Dependent Variable: Judgement Dunnett T3 (I) Observer (J) Observer Mean Difference (I-J) Std. Error Sig. 95% Confidence Interval Lower Bound Upper Bound Lower Bound Upper Bound Lower Bound 1 2.145(*).056.029.01.28 3.203(*).058.001.06.34 2 1-.145(*).056.029-.28-.01 3.058.055.641-.07.19 3 1-.203(*).058.001-.34-.06 2-.058.055.641-.19.07 * The mean difference is significant at the.05 level.

34. Results (cont.)

35. Post Hoc Tests: Providing Direction Multiple Comparisons Dependent Variable: Providing Direction Dunnett T3 (I) Observer (J) Observer Mean Difference (I-J) Std. Error Sig. 95% Confidence Interval Lower Bound Upper Bound Lower Bound Upper Bound Lower Bound 1 2.143(*).045.005.03.25 3-.182(*).065.015-.34-.03 2 1-.143(*).045.005-.25-.03 3-.325(*).060.000-.47-.18 3 1.182(*).065.015.03.34 2.325(*).060.000.18.47 * The mean difference is significant at the.05 level.

36. Results (cont.)

37. Post Hoc Tests: Empowerment Multiple Comparisons Dependent Variable: Empowerment Dunnett T3 (I) Observer (J) Observer Mean Difference (I-J) Std. Error Sig. 95% Confidence Interval Lower Bound Upper Bound Lower Bound Upper Bound Lower Bound 1 2.023.059.971-.12.16 3-.432(*).076.000-.61-.25 2 1-.023.059.971-.16.12 3-.455(*).077.000-.64-.27 3 1.432(*).076.000.25.61 2.455(*).077.000.27.64 * The mean difference is significant at the.05 level.

38. Results (cont.)

39. Post Hoc Tests: Control Multiple Comparisons Dependent Variable: Control Dunnett T3 (I) Observer (J) Observer Mean Difference (I-J) Std. Error Sig. 95% Confidence Interval Lower Bound Upper Boun d Lower Bound Upper Bound Lower Bound 1 2.095.044.090-.01.20 3-.044.059.834-.18.10 2 1-.095.044.090-.20.01 3-.139(*).054.030-.27-.01 3 1.044.059.834-.10.18 2.139(*).054.030.01.27 * The mean difference is significant at the.05 level.

40. Results (cont.)

41. Post Hoc Tests: In-Basket Multiple Comparisons Dependent Variable: In-Basket Dunnett T3 (I) Observer (J) Observer Mean Difference (I-J) Std. Error Sig. 95% Confidence Interval Lower Bound Upper Bound Lower Bound Upper Bound Lower Bound 1 2.056.031.215-.02.13 3-.080.037.096-.17.01 2 1-.056.031.215-.13.02 3-.135(*).033.000-.21-.06 3 1.080.037.096-.01.17 2.135(*).033.000.06.21 * The mean difference is significant at the.05 level.

42. Results (cont.)

43. Initiative Info Gathering Judgement Providing Direction Empower ment ControlIn-Basket Initiative1.000 Info Gathering.813(**)1.000 Judgement.448(**).445(**)1.000 Providing Direction.554(**).506(**).493(**)1.000 Empower ment.441(**).428(**).479(**).469(**)1.000 Control.491(**).535(**).419(**).431(**).400(**)1.000 In-Basket.475(**).418(**).761(**).679(**).814(**).595(**)1.000 Non-Parametric Correlations

44. Clear Regional; Age and Tenure Differences Do Exist among Participants Possible Sources of the Differences:  Regional Administration of In-Basket  Thus Differences in Administration Medium (Potosky, 2008) o Different Administrators (Explaining Purpose; Giving Instructions; Answering Questions) o Different Resources o Different Test Room Conditions DiscussionDiscussion

45. Differences Between Participants Regionally:  English Language Ability (not tested)  Motivation to Participate in the Assessment (not tested)  Differences in Employee Selection Processes as well as Training Opportunities (Burroughs et al., 1973)  Simulation Fidelity (not tested) Discussion (cont.)

46. Clear Regional; Age and Tenure Differences Do Exist among Participants Supporting Findings by Burroughs et al. (1973)  Age does Significantly Influence AC Performance  Participants from Certain Departments Perform Better Discussion (cont.)

47. Appropriateness of In-Basket for Ratees  Level of Complexity  Situation Fidelity Recommendations:  Ensure Documented Evidence (Analysis Phase in Design Model)  Pilot In-Basket on Target Ratees (Design Phase of Design Model)  Shared Responsibility of Service Provider and Client Organisation Discussion (cont.)

48. Context in Which In-Basket Administered  Purpose Communicated Recommendations:  Ensure Participants (Ratees) and Process Owners Understand and Buy- into Purpose Discussion (cont.)

49. Consistent Simulation Administration:  Instructions Given Consistently  Interaction with Administrator  Appropriate Resources Available During Administration  Test Room Conditions Appropriate for Testing Recommendations:  Ensure All Administrators Trained  Standardise Test Room Conditions Discussion (cont.)

50. Rater Differences do Exist Possible Sources of Rater Differences:  Background (All from a Psychology Background, with Management Experience)  Characteristics such as Personality (Bartels & Doverspike)  Owing to Cognitive Load on Raters  Owing to Differences in Mental Models (Jones & Born, 2008) Discussion (cont.)

51. Possible Sources of Rater Differences (cont.):  Training o All Received Behaviour Oriented Rater Training o Frame of Reference Different Discussion (cont.)

52. Recommendations:  Frame of Reference Training on:  Dimensions,  Management-Leadership Behaviour,  Norms  Project Management  Personality Assessment of Raters  Sub-dimension Differences

53. Questions?Questions? ?

54. SummarySummary  Found Rater Bias  Need to Research the Source of the Bias  Recommend Frame of Reference Training, Project Management Communication of Purpose and Administrator Training