Within- and Between-group Agreement in Supervisors Evaluative Behaviours: Do evaluative styles exist? Jan Noeverman Erasmus School of Economics Department of Business Economics Room W H P.O. Box DR Rotterdam The Netherlands Tel Fax

Purpose Address level of analysis issues around the concept of evaluative style; by exploring and testing within- and between-group agreement in perceived evaluative behaviours of supervisors within the context of a single organisation.

Research questions 1.Do differences in perceived evaluative behaviours across subordinates occur at the level of individual subordinates or do they reflect differences at a higher level, i.e. between-group differences? 2.Do subordinates who are evaluated by the same supervisor report differences in perceived evaluative behaviours (i.e., within-group differences), or do they report similar evaluative behaviours (i.e., within-group- agreement)? 3.If there are within-group differences in perceived evaluative behaviours, do these reflect differences that are intended by the supervisor?

Theoretical background: general Explicit consideration of levels-of-analysis issues in management accounting and control literature is rare; Level-of-analysis issues have received increasing attention in leadership and organizational behavior research; Precise articulation of the level of one's constructs is an important priority for all organizational scholars whether they propose single- or mixed-level theories (Klein, Dansereau, & Hall (1994, p.196)

Theoretical background: levels of analysis Three basic alternatives as to the appropriate level of analysis (if level of measurement is perception of individual subordinate as a member of a group): 1.group as a whole, i.e. responses from individuals are homogeneous within groups; 2.subordinates as a whole, i.e. responses are independent of groups, thus groups are irrelevant; 3.subordinates as parts within groups, i.e. responses are heterogeneous within groups.

Theoretical background: role of leaders in performance evaluation Supervisors usually have (some) discretion in how they apply elements of the performance measurement or incentive system. May lead to subjectivity in performance evaluation. Little attention paid to the use of discretion and its effects on subordinates behaviours and attitudes. Thus: investigate variability in the level of subjectivity used by a single supervisor in the performance evaluation of each of his subordinates.

Theoretical background : RAPM as an illustrative example Despite its problematic state, the RAPM literature is a good example of management accounting research in which consideration of levels-of-analysis issues is essential, given the focus on evaluative style as the focal construct of interest. Levels-of-analysis issues raised in earlier studies (Hopwood, 1972; 1973; 1974; Otley, 1978) have been completely ignored in later research.

Development of hypotheses Equity theory (e.g., Adams, 1963) would predict at least some general behavioural consistencies from supervisors… (Schriesheim, Cogliser, & Neider, 1995, pp ) Hypothesis 1: There are significant between-group differences in perceived evaluative behaviours of the supervisor (where groups are subordinates reporting to the same supervisor).

Development of hypotheses For example, expectancy theory (e.g., Vroom, 1964) would predict that leaders will behave differently toward subordinates in their work unit, depending upon how instrumental each is to furthering his or her desired goals or outcomes. (Schriesheim, Cogliser & Neider, 1995, p. 118) Hypothesis 2: There are significant differences in perceived evaluative behaviours of the supervisor within groups of subordinates reporting to the same supervisor.

Development of hypotheses Previous theoretical arguments are based on the implicit assumption that the supervisor actually has some discretion. If so, it is likely that each supervisor will use this discretion in a different way. Hypothesis 3: There will be significant differences in the level of within-group agreement between groups of subordinates. Hypothesis 4. Within- and between-group differences in perceived evaluative behaviours are related to differences in intended evaluative behaviours.

Research method: two empirical studies Study 1: –qualitative (exploratory) study; –interviews with two supervisors and three subordinates each in a Dutch service organization (postal service); –general on role of supervisors in performance evaluation. Study 2: –Quantitative (exploratory) study using WABA (Dansereau et al., 1984) and r WG (James, Demaree, & Wolf, 1984; 1993); –Focus on specific concept of role of supervisors in performance evaluation, i.e. feedback quality (Steelman, Levy, & Snell, 2004); –Perception of feedback quality provided by supervisor from 51 subordinates from an accounting firm reporting to eight supervisors.

Results study 1 Hypo- thesis FindingsSupported? 1Important differences in perception of evaluative behaviours between the two groups. + 2Differences in perceived evaluative behaviours in one group, not in the other. +- 3Differences in within-group variance of perceived evaluative behaviours between the two groups. + 4High congruence in intended (supervisors perception) and perceived evaluative behaviours. +

Results study 2 Hypo- thesis FindingsSupported? 1 WABA shows both within-group and between-group variance in perceived feedback quality. Within-group variance is practically, but not statistically larger than between-group variance No significant differences in within-group agreement coefficients of perceived feedback quality. - 4Could not be tested, because supervisor responses were not collected. n/a

Conclusion (I) Significant within- and between-group differences in perceived evaluative behaviours exist within single organisations. –Indicates that individual subordinate (whole) is appropriate level of analysis. Mixed evidence regarding differences in within-group variability. Some preliminary evidence that part of perceived within- group differences are intended by supervisors.

Conclusion (II) Evaluative styles as a characteristic of supervisors are not likely to consist. Better to refrain from using the term style, and use evaluatorship instead as an umbrella concept. Consideration of levels of analysis as in the current study is relevant for many other concepts included in management accounting research, and particularly research on performance evaluation (i.e., performance, task or environmental uncertainty).

Limitations Small sample sizes in both studies. Only level of analysis of one construct considered; no relationships between variables investigated. Dyad not considered explicitly when collecting the data. No consideration of why within-group and/or between- group variances in perception may arise. 16

Further research Extend the analyses to levels-of-analysis issues in relationships, both bivariate, multivariate, and moderated relationships. Appropriate level of analysis may be dependent on particular evaluative behaviour that is object of study. Consider dyad as the appropriate level of analysis, using dyadic data. Study development of evaluatorship in the context of dyadic relationships over time.