The Roles of Rater Goals and Ratee Performance Levels in the Distortion of Performance Ratings By Xiaoye May Wang Kin Fai Ellick wong, and Jessica y. y. kwong 2010 Chelsea Hutto

Overview 2 studies Peer rating and non-peer rating context Results found: raters use different rating tactics to achieve specific goals

Rating Inaccuracy One of the most important concerns in performance evaluation Traditionally thought to be a result of rating errors Assumes raters involuntarily make errors when rating, due to poor scale design or cognitive limitations 1. which helps in making administrative decisions such as promotion and salary increases

A More Current Viewpoint Considering whether raters are intentionally distorting ratings Suggest raters have goals in mind and provide ratings that are consistent which such goals AKA Goal based approach to performance evaluations

Current Viewpoint Not just a measurement process And not just a bunch of lazy raters! Must look at rater goals & rater characteristics Ratings may vary based on interaction of rater goals and rater performance Researchers suggest that PE isn’t just a measurement process but also a social and communication process Raters are active participants with the ability and motivation to distort ratings to attain certain goals Because the main effect of rater goals should be dependent on rater characteristics Due to the influence of goals on performance ratings can’t be examined apart from ratee characteristics

This particular study Wong and Kwong (2007) Examine the interaction between rater goals and performance levels on performance ratings Why examine performance level?? Never been formally tested & functionality of performance evaluation 1. 2. Decision was Based off previous work by & provide contributions that weren’t previously made due to limiations in the study 3. Such as major personnel decisions (promotion and salary increases

Current Research Purpose: extend previous research in a context that allows us to understand the effects of rater goals on rating scores for ratees with different levels of performance. Examined the four most common rater goals

Rater Goals Identification: ID strengths and weaknesses Harmony: Seeks to maintain group harmony and interpersonal relationships Fairness: Seeks to reflect the accurate contribution of each team member Motivating: Seeks to increase the future motivation of ratees Identification goal is used as a control because a typical performance appraisal often requires raters to identify ratees strengths and weaknesses for administrative decisions (promotions and appointments) and for developmental purposes (identifying training needs)

Goal and Performance Interaction Harmony: Fewer rewards = lower satisfaction and more antagonistic behaviors Low performers vs. high performers Hypothesis 1: Raters will exhibit rating inflation when they pursue a harmony goal (vs. ID goal); with the inflation being more pronounced as the performance level of the ratee decreases 2. Low performers – likely to have fewer rewards, which suggests raters will inflate ratings to reduce the likelihood of such results, but doing so will lead to dissatisfaction among the high performers so they will inflate both to account for the equity norm.

Goal and Performance Interaction Fairness: Based of equality and equity norms Equity vs. equality orientation Hypothesis 2: In the context of pursuing fairness (vs. identification), raters will inflate ratings for poorer performers and deflate ratings for better performers when they evaluate performers from the same working group; however, raters will have no such rating distortion when they evaluate performers as out-group members. Equality – rewards are evenly allocated to all members regardless of individual contributions Equity – rewards are given based on individual contributions Is based off whether the raters and ratees are from the same group – Use equity for out group and equality for in group members

Goal and Performance Interaction Motivating: Raters are likely to have different theories of motivation for low and high performers, thus raters believe that different types of performance feedback are required to motivate. Low vs. High performers H3: Raters will inflate ratings of poorer performers and deflate ratings of better performers when they have a motivating goal (vs. an identification goal). Low – importaint to maintain their positive self evaluation and self efficacy – so strong neg feedback is demotivating seen as a threat to the self, to reduce this rater inflates ratings for low. High – people are motivated to reduce discrepancy between performance standard and current level – increase in discrepancy increases motivation – better perf are less motivated – get lower ratings

Study One 103 Undergrad students in a HRM course Within participants design Groups were formed voluntarily Required to complete peer evaluations twice Mid-semester and End of semester

Materials 5 page questionnaire 4 sets of peer evaluations, asked to complete with a different goal in mind Rated on a 1-7 contribution scale

Analyses 2 separate analyses Hierarchical linear modeling (HLM) ANOVA 2 x 3 study design Goal Condition x Performance Level 2. Which used restricted maximum likelihood estimation and reported values based on the robust standard errors 3. To examine inflation and deflation of ratings in each of the performance levels (low, medium, and high)

Results for Harmony Goal Hypothesis 1: Significance found at mid-semester and end of semester between harmony goal and rating scores Interaction between harmony and ratees performance level was also significant Greater rating inflation as performance level decreased Inflation was greatest for low performers, followed by medium, and then high Raters didn’t deflate ratings for high performers, which could be due to different justice norms used. Inflation was stronger at mid-semester than end of semester. Results suggest raters tend to use equality norm for low performers.

Results for Fairness Goal Hypothesis 2: Significance was found between fairness and rating scores for mid-semester and end of semester Interaction between fairness and performance level was significant – indicates inflation varied across performance levels Effects of fairness was significant for low and medium, but not for high performers Results are consistent with H2, which predicts that if raters use the equality norm to represent fairness, they would inflate their ratings for low while deflating for high. Results reveal rating inflation for poor, but don’t show the expected deflation for good.

Results for Motivating Hypothesis 3: Significance found between motivating and rating scores at mid-semester and end of semester Interaction between motivating and performance level was significant Effect of motivating was significant for low and medium, but not for high performers 2. Which means raters inflated ratings differently across performance levels Raters increased for low and medium but didn’t distort at all for high – may be due to raters assuming high performers were already highly motivated and didn’t need to receive neg feedback to be externally motivated

Discussion Raters have different distortions for ratees with various performance levels under different goal conditions Study 1 supports general proposition that to achieve a specific goal, raters tailor the degree of rating distortion to ratees with different performance levels Raters tended to inflate more as performance levels decreased

Study 2 120 Undergrad students in an organizational behavior course Between subjects design Randomly assigned to one of the rater goal conditions 15 minute video and evaluated the performance of 6 team members on a 10 item scale Scale included items on innovation, cooperation, presentation skills, team work spirit, etc. Introduced to backgrounds of members then video, observe, and evaluate. Then asked to play the role of team manager of the 6 members and give ratings for each of them according to their assigned rating goals.

Analyses and Results Same as Study 1 Hypothesis 1: Not supported Interaction between rater goals and ratee performance was not significant under harmony goal condition Supported by both ANOVA and HLM

Results Hypothesis 2: Partially supported Interaction between fairness and ratee performance level was significant Effect was significant for high performers but not for medium or low Significant rating deflation for high performers

Results Hypothesis 3: Partially supported Interaction between motivating and performance level was significant Effect of motivating was significant for low but there was not significant distortion for medium or high performers

Differences between Studies Harmony condition: Found significant results in Study 1 but not Study 2 Inconsistency may be due to different relationships between raters and ratees. Goal may not have been as salient in Study 2 as in Study 1

Discussion for both studies Harmony: Raters deflated their ratings for all performers. Raters tried to reduce conflict by lowering the discriminability rather than inflating ratings Fairness: Raters who wanted to achieve a motivating goal inflated ratings for low but didn’t distort for high Results suggest people are more likely to adopt equality rule for low performers in peer rating context and high performers in the non-peer rating context

Discussion Results suggest rating accuracy is shaped by the ratees ability (performance level) and the raters motivation (goals) Rating errors are not only due to inadequate rating skill, but also a result of the intentions of the raters Demonstrated the importance of examining contextual effects on performance appraisal Suggests there is an alternative understanding of rating inaccuracy

Limitations Didn’t build on long term relationships (one semester and 45 minutes) Lack of politics that are present in an organization Study 1 was a single item measure – hindered exploration of performance dimensionality

Future Research Should explore individual differences in implicit theories of rating strategies Clearly stating the rating goal for the rater may be a solution to reduce rating discrepancies

Discussion Questions The authors primarily used their own research to base research design and decisions on, do you feel this could significantly affect the overall results of this study? In what ways could we measure rater motivation to reduce rating inaccuracy in the organization? Do you think the authors research design could be improved to more appropriately test the author’s hypotheses? If so, how?