1. The Roles of Rater Goals and Ratee Performance Levels in the Distortion of Performance Ratings
By Xiaoye May Wang
Kin Fai Ellick wong,
and
Jessica y. y. kwong
2010
Chelsea Hutto

2. Overview 2 studies Peer rating and non-peer rating context
Results found: raters use different rating tactics to achieve specific goals

3. Rating Inaccuracy
One of the most important concerns in performance evaluation
Traditionally thought to be a result of rating errors
Assumes raters involuntarily make errors when rating, due to poor scale design or cognitive limitations
1. which helps in making administrative decisions such as promotion and salary increases

4. A More Current Viewpoint
Considering whether raters are intentionally distorting ratings
Suggest raters have goals in mind and provide ratings that are consistent which such goals
AKA Goal based approach to performance evaluations

5. Current Viewpoint Not just a measurement process
And not just a bunch of lazy raters!
Must look at rater goals & rater characteristics
Ratings may vary based on interaction of rater goals and rater performance
Researchers suggest that PE isn’t just a measurement process but also a social and communication process
Raters are active participants with the ability and motivation to distort ratings to attain certain goals
Because the main effect of rater goals should be dependent on rater characteristics
Due to the influence of goals on performance ratings can’t be examined apart from ratee characteristics

6. This particular study Wong and Kwong (2007)
Examine the interaction between rater goals and performance levels on performance ratings
Why examine performance level??
Never been formally tested & functionality of performance evaluation 1.
2. Decision was Based off previous work by & provide contributions that weren’t previously made due to limiations in the study
3. Such as major personnel decisions (promotion and salary increases

7. Current Research
Purpose: extend previous research in a context that allows us to understand the effects of rater goals on rating scores for ratees with different levels of performance.
Examined the four most common rater goals

8. Rater Goals Identification: ID strengths and weaknesses
Harmony: Seeks to maintain group harmony and interpersonal relationships
Fairness: Seeks to reflect the accurate contribution of each team member
Motivating: Seeks to increase the future motivation of ratees
Identification goal is used as a control because a typical performance appraisal often requires raters to identify ratees strengths and weaknesses for administrative decisions (promotions and appointments) and for developmental purposes (identifying training needs)

9. Goal and Performance Interaction
Harmony: Fewer rewards = lower satisfaction and more antagonistic behaviors
Low performers vs. high performers
Hypothesis 1: Raters will exhibit rating inflation when they pursue a harmony goal (vs. ID goal); with the inflation being more pronounced as the performance level of the ratee decreases
2. Low performers – likely to have fewer rewards, which suggests raters will inflate ratings to reduce the likelihood of such results,
but doing so will lead to dissatisfaction among the high performers so they will inflate both to account for the equity norm.

10. Goal and Performance Interaction
Fairness: Based of equality and equity norms
Equity vs. equality orientation
Hypothesis 2: In the context of pursuing fairness (vs. identification), raters will inflate ratings for poorer performers and deflate ratings for better performers when they evaluate performers from the same working group; however, raters will have no such rating distortion when they evaluate performers as out-group members.
Equality – rewards are evenly allocated to all members regardless of individual contributions
Equity – rewards are given based on individual contributions
Is based off whether the raters and ratees are from the same group – Use equity for out group and equality for in group members

11. Goal and Performance Interaction
Motivating: Raters are likely to have different theories of motivation for low and high performers, thus raters believe that different types of performance feedback are required to motivate.
Low vs. High performers
H3: Raters will inflate ratings of poorer performers and deflate ratings of better performers when they have a motivating goal (vs. an identification goal).
Low – importaint to maintain their positive self evaluation and self efficacy – so strong neg feedback is demotivating seen as a threat to the self, to reduce this rater inflates ratings for low.
High – people are motivated to reduce discrepancy between performance standard and current level – increase in discrepancy increases motivation – better perf are less motivated – get lower ratings

12. Study One 103 Undergrad students in a HRM course
Within participants design
Groups were formed voluntarily
Required to complete peer evaluations twice
Mid-semester and End of semester

13. Materials 5 page questionnaire
4 sets of peer evaluations, asked to complete with a different goal in mind
Rated on a 1-7 contribution scale

14. Analyses 2 separate analyses Hierarchical linear modeling (HLM) ANOVA
2 x 3 study design
Goal Condition x Performance Level
2. Which used restricted maximum likelihood estimation and reported values based on the robust standard errors
3. To examine inflation and deflation of ratings in each of the performance levels (low, medium, and high)

15. Results for Harmony Goal
Hypothesis 1: Significance found at mid-semester and end of semester between harmony goal and rating scores
Interaction between harmony and ratees performance level was also significant
Greater rating inflation as performance level decreased
Inflation was greatest for low performers, followed by medium, and then high
Raters didn’t deflate ratings for high performers, which could be due to different justice norms used. Inflation was stronger at mid-semester than end of semester. Results suggest raters tend to use equality norm for low performers.

16. Results for Fairness Goal
Hypothesis 2: Significance was found between fairness and rating scores for mid-semester and end of semester
Interaction between fairness and performance level was significant – indicates inflation varied across performance levels
Effects of fairness was significant for low and medium, but not for high performers
Results are consistent with H2, which predicts that if raters use the equality norm to represent fairness, they would inflate their ratings for low while deflating for high. Results reveal rating inflation for poor, but don’t show the expected deflation for good.

17. Results for Motivating
Hypothesis 3: Significance found between motivating and rating scores at mid-semester and end of semester
Interaction between motivating and performance level was significant
Effect of motivating was significant for low and medium, but not for high performers
2. Which means raters inflated ratings differently across performance levels
Raters increased for low and medium but didn’t distort at all for high – may be due to raters assuming high performers were already highly motivated and didn’t need to receive neg feedback to be externally motivated

18. Discussion
Raters have different distortions for ratees with various performance levels under different goal conditions
Study 1 supports general proposition that to achieve a specific goal, raters tailor the degree of rating distortion to ratees with different performance levels
Raters tended to inflate more as performance levels decreased

19. Study 2 120 Undergrad students in an organizational behavior course
Between subjects design
Randomly assigned to one of the rater goal conditions
15 minute video and evaluated the performance of 6 team members on a 10 item scale
Scale included items on innovation, cooperation, presentation skills, team work spirit, etc.
Introduced to backgrounds of members then video, observe, and evaluate. Then asked to play the role of team manager of the 6 members and give ratings for each of them according to their assigned rating goals.

20. Analyses and Results Same as Study 1 Hypothesis 1: Not supported
Interaction between rater goals and ratee performance was not significant under harmony goal condition
Supported by both ANOVA and HLM

21. Results Hypothesis 2: Partially supported
Interaction between fairness and ratee performance level was significant
Effect was significant for high performers but not for medium or low
Significant rating deflation for high performers

22. Results Hypothesis 3: Partially supported
Interaction between motivating and performance level was significant
Effect of motivating was significant for low but there was not significant distortion for medium or high performers

23. Differences between Studies
Harmony condition: Found significant results in Study 1 but not Study 2
Inconsistency may be due to different relationships between raters and ratees.
Goal may not have been as salient in Study 2 as in Study 1

24. Discussion for both studies
Harmony: Raters deflated their ratings for all performers. Raters tried to reduce conflict by lowering the discriminability rather than inflating ratings
Fairness: Raters who wanted to achieve a motivating goal inflated ratings for low but didn’t distort for high
Results suggest people are more likely to adopt equality rule for low performers in peer rating context and high performers in the non-peer rating context

25. Discussion
Results suggest rating accuracy is shaped by the ratees ability (performance level) and the raters motivation (goals)
Rating errors are not only due to inadequate rating skill, but also a result of the intentions of the raters
Demonstrated the importance of examining contextual effects on performance appraisal
Suggests there is an alternative understanding of rating inaccuracy

26. Limitations
Didn’t build on long term relationships (one semester and 45 minutes)
Lack of politics that are present in an organization
Study 1 was a single item measure – hindered exploration of performance dimensionality

27. Future Research
Should explore individual differences in implicit theories of rating strategies
Clearly stating the rating goal for the rater may be a solution to reduce rating discrepancies

28. Discussion Questions
The authors primarily used their own research to base research design and decisions on, do you feel this could significantly affect the overall results of this study?
In what ways could we measure rater motivation to reduce rating inaccuracy in the organization?
Do you think the authors research design could be improved to more appropriately test the author’s hypotheses? If so, how?