1. The Efficacy of Student Evaluations of Teaching Effectiveness
Megan O’Neill
Assistant Professor
Department of Humanities
Presented to the Faculty Senate October 25, 2018

2. Claim: “The current practice of using student evaluations as summative measures to determine decisions for retention, promotion, and pay for faculty members is improper an depending on the circumstances could be argued to be illegal.” Hornstein, 2017

3. What are “Student Evaluations of Teaching” (SET)?
History
1970s- A formative measure of student opinions to give feedback to the individual instructor.
1980s- Adopted by administrations to evaluate “teaching effectiveness” due to ease of data collection.
1990s to Now- A few studies argue that there are small-to-moderate correlations between SET scores and student achievement. When reanalyzed for variations in sample size, little-to-no correlation exists (Uttl, White, and Gonzalez, 2016).

4. Current Literature on SET
Challenges to Reliability
While inter-item consistency exists, agreement between students is no greater than “chance,” indicating that students do not agree with what they are being asked to evaluate. No inter-rater reliability.
Patterns of reliability indicate that the tool provides information about students, not instructors (Clayson, 2017).
Challenges to Validity
Tool measures student satisfaction only
By what valid criteria are students able to determine instructor knowledge, pedagogical practices, etc?

5. Current Literature on SET
Challenges to Data Analysis
Statistical averages can not be drawn from categories that differ in quality, rather than quantity or magnitude (Stark & Freishat, 2014).
Challenges to Conclusions
Current data does not support the assumption that students learn more from higher rated instructors. (Uttl, White, and Gonzalez, 2016).

6. What Variables Effect SET Scores?
Research shows SET scores are effected by:
Discipline
Student interest and motivation
Class size
Class time (of day)
Classroom space
Gender
Race
Instructor Accent
Uttl, White, and Gonzalez, 2016

7. Gender & Race
In double blind studies of online teaching, women and people of color are rated lower than white men for the same teaching (Wagner, Rieger, and Voorvelt, 2016). When instructor effectivness is evaluated using a holistic methods, the data shows that less competent male instructors receive higher SET scores than more effective female instructors (Boring, 2016).

8. Best Practices & Recommendations
Drop omnibus items about “overall teaching effectiveness” and “value of the course” from SET. They are misleading.
Do not average or compare averages of student rating. Statistically unsound. Instead, report distribution, number of responders, and response rate.
Understand the limitations of student comments. While they are the authority on their own experiences, they are usually not well suited to evaluate pedagogy.
Avoid comparing teaching in courses of different types, levels, sizes, functions, or disciplines.
Use teaching portfolios as part of the review process.
Use classroom observations as part of milestone reviews.
Reconsider the role of SET in tenure and promotion decisions- the current model rewards instructors for not challenging students or disciplinary teaching paradigms.

9. Works Cited
Boring, A. (2017). Gender biases in student evaluations of teaching. Journal of Public Economics, 145, Clayson, D. (2018). Student evaluations of teaching and maters of reliability. Assessment and Evaluation in Higher Education, 43:4, Hornstein, H. (2017). Student evaluations of teaching are an inadequate assessment tool for evaluating faculty. Cogent Education, 4, 1-8. Stark, P.B., & Freishat, R. (2014). An evaluation of course evaluations. Retrieved from Uttl, B., White, C., & Gonzakez, D. (2017). Meta-analysis of faculty's teaching effectiveness: Student evaluation of teaching ratings and student learning are not related. Studies in Educational Evaluation, 54, Wagner, N., Rieger, M., & Voorvelt, K. (2016). Gender, ethnicity and teaching evaluations: Evidence from mixed teaching teams. Economics of Education Review, 54,