1 of 14 Exploring race and gender differentials in student ratings of instructors: Lessons from a diverse liberal arts college Robert L. Moore, Hanna Song Spinosa, James D. Whitney Occidental College May 2014

2 of 14 Do the race and gender of instructors and the race and gender composition of their classes make a difference in the ratings instructors receive from their students? 1.The answer matters (1) social concerns, such as the persistence of discrimination (2) practical concerns, such as tenure and promotion 2.Previous research regarding gender: conflicting findings (1) no difference by gender (2) lower ratings for female instructors (3) “same-gender preferences” 3.Previous research regarding race: (1) hardly any (2) only one in economics literature (3) higher ratings for white instructors

3 of 14 What differentiates our study from previous research? 1.Explicit focus on both race and gender 2.More recent and largest dataset to date 3.Data sample with relatively high levels of diversity 4.Econometric techniques 5.Race and gender composition of enrolled students 6.Supplemental approaches including a Oaxaca decomposition and panel data estimates from a subset of class sections taught by the same instructor contemporaneously

4 of 14 What do we find in our study? 1.In several cases of student-instructor pairings (for example, white male student ratings of white female instructors, and so on), we find sizeable estimates of race and gender ratings differentials. 2.In only a few cases are the empirical estimates statistically significant after appropriate adjustments for clustering of the data. 3.Overall class-average student ratings in our dataset do not differ enough by instructor race and gender to warrant systematic ratings adjustments for tenure and promotion decisions, but do warrant a general attentiveness to particular teaching situations in which instructor and student demographics might matter. 4.Our clearest findings are cautionary observations regarding the challenges of related empirical research, including (1) clustering of the observations to adjust for heteroskedasticity, (2) demographic heterogeneity on the part of students as well as faculty, and (3) potential control variables that risk omitted variable bias if excluded 5.Much larger datasets, particularly datasets which span multiple institutions, or data which include the race and gender of individual student evaluators might yield more robust and consistent results than we have found in our study.

5 of 14 Data and methodology: Our dataset of Occidental College student evaluations is comparatively large and recent Includes all student evaluations that report an overall student rating of instructor, submitted for full-credit classes (counting for 4 or more units) with enrollments above 5 students StudyEvaluationsClassesInstructorsYearsInstitution Moore, Spinosa, and Whitney (2014) 74,0724, Occidental College Smith (2007), Smith and Hamilton (2011) 13, U of GA Centra and Guabatz (2000) 741 Multiple (20) Hamermesh and Parker (2005) >16, U of TX

6 of 14 Occidental is also noteworthy for the relatively high diversity of both its faculty and students Race and gender composition of faculty and students, Academic Year FacultyStudents PercentageRank* (259)PercentageRank* (263) MaleFemaleMaleFemaleMaleFemaleMaleFemale White39.7%30.1% %34.3% Black4.6%2.0% %3.2% Latino4.3%7.4%1245.8%8.1%1922 Native American 0.0% % 2737 Asian, Islanders 4.1%7.8%3796.9%9.8%712 Herfindahl Index148 * Among all US News national liberal arts colleges with available data.

7 of 14 Methodology 1.Like Smith (2007), Smith and Hamilton (2011): explicit consideration of race 2.Like Centra and Guabatz (2000): demographics of enrolled students 3.Like Hamermesh and Parker (2005): controls for non-demographic factors Basic structure of the regression equations we estimate: Q n =  +  X n +  Z n +  n 1.subscript n: a sample observation, by default a class average but in some specified cases an individual student evaluation. 2.Q: student rating of instructor (SRI), on a seven-point descending scale (“Overall, the instruction for this course was excellent.”)  X n : a vector summation of demographic variables for each observation (X n ) multiplied by their corresponding estimated coefficients (  ).  Z n : Analogous to  X n but with Z n denoting non-demographic control variables.  denotes a constant and  n a random error term for observation n.

8 of 14 Empirical results A: Student Rating of Instructor (SRI) differentials by instructor race and gender (1) Relative to White Male instructors, other race-gender instructor groups receive lower student ratings, dovetailing with previous research. (2) Most of the ratings differentials are comparatively small, averaging ratings points (-0.18 standard deviations), less than half as large as commonly reported in past studies. (3) Nearly all of the estimated differentials become statistically insignificant after adjusting the error terms for clustering of the data.

9 of 14 Results B. Estimating SRI differentials when student demographics vary across classes. Students in our sample are overrepresented in classes taught by instructors that match their own race and gender, and this can compress overall average SRIs by instructor.

10 of 14 After incorporating student demographics: (1) Within each student subgroup (except for Other Female students), estimated cross-group ratings are lower than own-group ratings in 23 of 25 cases. (2) The sizes of the estimated differentials are typically large compared to previously reported findings, averaging 0.63 ratings points (0.75 standard deviations), although only a few of the estimates are statistically significant.

11 of 14 Results C: Controlling for non-demographic factors that are likely to be associated with SRIs Hamermesh and Parker (2005) included two additional dummy control variables related to courses (Lower division, One-credit course) and four related to instructors (Female, Minority, Non-native English, Tenure track). We included more controls, as listed in the table to the right.

12 of 14 Results of incorporating non-demographic control variables: (1) a sharp reduction in the sizes and statistical significance of the diagonal entries (own-group differentials) to average 0.24 ratings points (0.28 s.d.), compared to 0.51 ratings points before, with none of the estimates statistically significant (2) a similarly sharp drop in the average size of the estimated ratings differentials within student groups vs. their respective own-group benchmark, from 0.63 ratings points before to 0.32 ratings points (0.38 standard deviations) now.

13 of 14 Except for the outlier case of Other Female students, the arrows in the chart below indicate substantial convergence of the own-group and sub-group ratings when non- demographic factors are included as control variables

14 of 14 Self-reported learning outcomes and, especially, detailed ratings of instructors sharply reduce the size and significance of the estimated coefficients for the base set of non- demographic control variables. (You might also be interested to note here the detailed ratings items that are most strongly associated with the overall rating of instruction: 1. clarity 2. intellectual enthusiasm 3. fulfilling course goals 4. organization 5. responsive to questions)