1. "Evaluating Students' Evaluations of Teaching: Bias and Beyond"
Excellence in Teaching and Learning Committee (ETLC) Workshop
College of Liberal Arts & Social Sciences (CLASS)
Chris Davoli (Psychology) & David Kinney (Sociology)
November 2, 2018

2. Today: 1) Research regarding the sources and extent of biases in SET
2) Interdisciplinary discussion about how bias presents problems for SET at CMU
3) Possible solutions for reducing and/or eliminating biases in our SET
Today:

3. Sources of Bias Gender Class size Race/ethnicity
Students’ expected grade in course
Required vs. Elective course
Academic discipline
Physical appearance
Non-native English speaker
Class size
Intelligence/competency
Sexual orientation
Instructors perceived as “easy”
Personality
Entertainment
Age

4. Gender: The most-researched characteristic as a source of bias – The larger context
Women: 37.5 percent of all tenured faculty in American colleges & universities
Women of color: 8 percent
Full professors: Men outnumber women by more than two to one.
Majority of assistant professors, instructors, and lecturers are women. 
Gender gap in promotions: Sociology 85% of men and 78% of women.
Computer Science: 86% of men and 81% of women.
English departments: 86% of men and 80% of women (Weisshaar 2017).

5. Figure 2. Cumulative hazard plot of receiving tenure, by gender and field, over career time. “Years since starting” indicates the number of years spent as an assistant professor.
From: Publish and Perish? An Assessment of Gender Gaps in Promotion to Tenure in Academia
Soc Forces. 2017;96(2): doi: /sf/sox052
Soc Forces | © The Author Published by Oxford University Press on behalf of the University of North Carolina at Chapel Hill. All rights reserved. For permissions, please 5

6. GENDER PAY GAP
Female full professors - 90% Associate professors – 93% Assistant professors – 91% Lecturers – 88% of what men make.
Total full-time faculty:
MALE full professors: 26%
FEMALE full professors: 8.4%

9. Most recent and most rigorous study of gender bias
Anne Boring (2017)
23,001 evaluations of 379 instructors by 4,423 students in 6 mandatory first-year courses at a French university
French data: male students rank male instructors much higher than female instructors; but little difference among female students.
Men viewed as being more knowledgeable even though students learn as much from men as from women.

10. Gender bias – boring and associates (2016) continued
Four sections of an online course in a randomized, controlled, blind experiment at a U.S. university
When one professor taught the exact same online course under two different names, students gave lower rankings to the professor with the female name. 
Female students rated perceived male instructors higher than they rated perceived female instructors; but little difference by male students.

11. Mitchell & martin (2018): “gender bias in student evaluations”
Content analysis of comments on official course evaluations & from Rate My Professors
Students comment on female professors’ personality and appearance and are significantly more likely to call them “teachers” rather than “professors” indicating that students may have “less professional “respect” for their female professors (p. 652).
Personality mentioned 4.3% of the time for male professors vs. 15.6% for female profs.
Appearance: 0% for male professor and 10.6% for female professor.
Entertainment: 15.2% for male professor and 32.2% for female professor.
Quantitative SET findings: male received higher evaluations on all items in identical courses

12. KEY CONSEQUENCE OF GENDER BIAS IN S.E.T.S.
IN AN EFFORT TO INCREASE THEIR S.E.T. SCORES, FEMALE PROFESSORS MAY SPEND MORE TIME AND ENERGY ON COURSE PREPARATION AND GIVING ATTENTION TO STUDENTS
THIS IS AN “OPPORTUNITY” COST:
LESS TIME TO CONDUCT RESEARCH & PUBLISH
LESS TIME TO PROVIDE SERVICE TO THE DEPARTMENT, UNIVERSITY & COMMUNITY
MAY LEAD TO LOWERED CAREER AMBITIONS AND EVEN LEAVING ACADEME

13. FINAL WORDS ON GENDER BIAS
BIAS EXISTS IN BOTH FACE-TO-FACE & ONLINE CLASSES
USE OF BIASED SETs in EMPLOYMENT DECISIONS (PROMOTIONS etc.) IS DISCRIMINATORY AGAINST WOMEN
U.S. EQUAL PAY ACT OF 1963 PROHIBITS DETERMING WAGES ON THE BASIS OF SEX
TITLE VII OF THE CIVIL RIGHTS ACT OF 1964 DECREES THAT DISCRIMINATING PROMOTIONAL PRACTICES ON THE BASIS OF SEX ARE FORBIDDEN

14. Bias due to students’ expected grade?
Cho, Baek & Cho (2015): major university in South Korea
Natural experiment: due to computer technical problems, some students saw final grades before completing teaching evaluations
Students who know low grades in advance, rate teachers significantly lower than students who get low grades, but do not know in advance
And students who see their low grades and they are lower than they expected, rate teachers even lower: “retaliatory behavior” (p.178)
A discrepancy between grade received and grade expected affects SET indicating that “psychological and subjective” factors impact SET.

15. References
Boring, Anne “Gender Bias in Student Evaluations of Teaching.” Journal of Public Economics 145(January):27-41.
Cho, D., W. Baek & J. Cho “Why do good performing students highly rate their instructors? Evidence from a natural experiment.” Economics of Education Review 49:
Mitchell, K.M.W. & J. Martin “Gender Bias in Student Evaluations.” The Teacher: American Political Science Association. July:
Weisshaar, Katherine “Publish and Perish? An Assessment of Gender Gaps in Promotion to Tenure in Academia.” Social Forces 96(2):529–560.