The Efficacy of Student Evaluations of Teaching Effectiveness Megan O’Neill Assistant Professor Department of Humanities Presented to the Faculty Senate October 25, 2018

Claim: “The current practice of using student evaluations as summative measures to determine decisions for retention, promotion, and pay for faculty members is improper an depending on the circumstances could be argued to be illegal.” Hornstein, 2017

What are “Student Evaluations of Teaching” (SET)? History 1970s- A formative measure of student opinions to give feedback to the individual instructor. 1980s- Adopted by administrations to evaluate “teaching effectiveness” due to ease of data collection. 1990s to Now- A few studies argue that there are small-to-moderate correlations between SET scores and student achievement. When reanalyzed for variations in sample size, little-to-no correlation exists (Uttl, White, and Gonzalez, 2016).

Current Literature on SET Challenges to Reliability While inter-item consistency exists, agreement between students is no greater than “chance,” indicating that students do not agree with what they are being asked to evaluate. No inter-rater reliability. Patterns of reliability indicate that the tool provides information about students, not instructors (Clayson, 2017). Challenges to Validity Tool measures student satisfaction only By what valid criteria are students able to determine instructor knowledge, pedagogical practices, etc?

Current Literature on SET Challenges to Data Analysis Statistical averages can not be drawn from categories that differ in quality, rather than quantity or magnitude (Stark & Freishat, 2014). Challenges to Conclusions Current data does not support the assumption that students learn more from higher rated instructors. (Uttl, White, and Gonzalez, 2016).

What Variables Effect SET Scores? Research shows SET scores are effected by: Discipline Student interest and motivation Class size Class time (of day) Classroom space Gender Race Instructor Accent Uttl, White, and Gonzalez, 2016

Gender & Race In double blind studies of online teaching, women and people of color are rated lower than white men for the same teaching (Wagner, Rieger, and Voorvelt, 2016). When instructor effectivness is evaluated using a holistic methods, the data shows that less competent male instructors receive higher SET scores than more effective female instructors (Boring, 2016).

Best Practices & Recommendations Drop omnibus items about “overall teaching effectiveness” and “value of the course” from SET. They are misleading. Do not average or compare averages of student rating. Statistically unsound. Instead, report distribution, number of responders, and response rate. Understand the limitations of student comments. While they are the authority on their own experiences, they are usually not well suited to evaluate pedagogy. Avoid comparing teaching in courses of different types, levels, sizes, functions, or disciplines. Use teaching portfolios as part of the review process. Use classroom observations as part of milestone reviews. Reconsider the role of SET in tenure and promotion decisions- the current model rewards instructors for not challenging students or disciplinary teaching paradigms.

Works Cited Boring, A. (2017). Gender biases in student evaluations of teaching. Journal of Public Economics, 145, 27-41. Clayson, D. (2018). Student evaluations of teaching and maters of reliability. Assessment and Evaluation in Higher Education, 43:4, 666-681. Hornstein, H. (2017). Student evaluations of teaching are an inadequate assessment tool for evaluating faculty. Cogent Education, 4, 1-8. Stark, P.B., & Freishat, R. (2014). An evaluation of course evaluations. Retrieved from https://www.stat.berkeley.edu/~stark/Preprints/teachEval14.pdf. Uttl, B., White, C., & Gonzakez, D. (2017). Meta-analysis of faculty's teaching effectiveness: Student evaluation of teaching ratings and student learning are not related. Studies in Educational Evaluation, 54, 22-42. Wagner, N., Rieger, M., & Voorvelt, K. (2016). Gender, ethnicity and teaching evaluations: Evidence from mixed teaching teams. Economics of Education Review, 54, 79-94.