1. Potential Biases in Student Ratings as a Measure of Teaching Effectiveness Kam-Por Kwan EDU Tel: 2766 6287 E-mail: etkpkwan

2. Your beliefs about student ratings Answer the 10 questions on the worksheet by stating whether you think each of the statements is ‘True’ or ‘False’. (Click here for the questions in PDF format)here There is no right or wrong answer. The main purpose is to find out more about your own beliefs and views about student ratings of teaching.

3. Uses of student ratings As a measure of student perceptions or satisfaction about the teaching As a measure of teaching effectiveness of the instruction As a measure of teaching effectiveness of the teacher

4. Validity concerns about ratings Do ratings accurately reflect the perceptions and satisfaction of the students? Do ratings accurately reflect the teaching effectiveness of the instructional context? Do ratings accurately reflect the teaching effectiveness of the teacher?

5. Ratings as student perceptions Biases exist if the ratings fail to measure accurately what students really feel about the teaching u Little dispute in the validity of ratings for this use

6. Ratings as TE of the instruction Biases exist if ratings are influenced by factors unrelated to teaching effectiveness u Lower ratings for larger classes  no bias u Lower ratings for ‘better-looking’ teachers  existence of bias

7. Ratings as TE of the teacher Biases exist if ratings influenced by factors beyond the control of the instructor u Higher ratings for more emphatic teachers  possibly no bias u Higher ratings for teachers of smaller class  potential bias or unfairness

8. Criticisms of student ratings Students cannot make consistent judgments about teaching Student ratings are popularity contests Students cannot make accurate judgments until they have graduated Student ratings are unrelated to amount of learning Staff and students disagree on what constitutes good teaching

9. Some more criticisms Student ratings are influenced by u time and day of the teaching u class size u level of the course u rank of instructor u nature of the course: required or elective u difficulty level of course / assignments u expected grades u disciplinary differences

10. What research evidence is there? Over 70 years of research More than 2000 studies A huge body of research evidence Some of the criticisms are valid, some are not

11. Ratings are reasonably reliable Well-constructed student rating forms are highly reliable (alphas in the 0.8 / 0.9 range) Ratings are stable over time (r > 0.8) High correlation exists between ratings of same instructor and course (r = 0.7 to 0.89)

12. Ratings are reasonably valid Staff and students generally agree on the important dimensions of good teaching Student ratings are moderately correlated with achievement (r = about 0.5) Student ratings correlate moderately with alumni ratings, classroom observations, and self-evaluation by staff Distinguishable ‘profiles’ of teaching can be revealed from student ratings

13. Course and teacher effects Small associations are found between student ratings and the following factors: u class size (ratings > for smaller classes) u level of course (ratings > for higher-level courses) u nature (ratings > for elective courses) u discipline (languages & art > social sciences > engineering & science) u rank (ratings > for higher-rank teachers)

14. Course and teacher effects (2) Inconsistent / no associations are found between student ratings and: u gender of instructor u day and time of the course

15. Student factors Small associations found between student ratings and: u prior subject interest (ratings > for higher prior interest) u students’ major (ratings > for major) u perceived workload / difficulty (ratings > for higher workload / more difficult courses) u expected grades (ratings > for higher grade)

16. Overall effect About 15 to 20 percent of the variation in ratings can be explained by the combined effects of the background (course, teacher and student) variables

17. Ratings are basically unbiased The existence of course, teacher and student effects on ratings generally support rather than refute the validity of student ratings as a measure of teaching effectiveness In most cases, the effects are quite small

18. Ratings can be unfair Teacher evaluation based on raw student ratings can be ‘unfair’ to individual teachers because: u a lot of factors affecting teaching effectiveness are outside the control of the teacher (although the effects are quite small) u there are big differences in the context in which different teachers operate

19. Conclusion Student ratings are generally valid as a measure of student perceptions/ satisfaction Student ratings are reasonably valid as a measure of teaching effectiveness of instruction Student ratings can be unfair when they are used for making personnel decisions concerning individual teachers

20. Implications Student ratings are useful for u understanding student satisfaction and perceptions u improving teaching Student ratings must be interpreted and used cautiously and in context when used for making judgements and personnel decisions about individual teachers

21. References Aleamoni, L. M. (1987). Student rating myths versus research facts. Journal of Personnel Evaluation in Education, 1, 111-119. Feldman, K.A. (1996). Identifying exemplary teaching: using data from course and teacher evaluations. New Directions for Teaching & Learning, 65. Jossey-Bass Publishers. Marsh, H.W. (1987). Students’ evaluation of university teaching: research findings, methodological issues, and directions for future research. Int. J. Edu Res., 11, 305-329.

22. Answers to T/F Questions 1. F 2. F 3. F 4. T 5. F 6. F 7. T 8. F 9. T 10. T