1. An Exploratory Analysis of Teaching Evaluation Data Michael D. Martinez Department of Political Science College of Liberal Arts and Sciences University of Florida martinez@ufl.edu August 17, 2011

2. Questions What does our teaching evaluation instrument actually tell us about our teaching? Are the items that students use to evaluate us actually measuring different things? Do the items in the teaching evaluation instrument actually produce a reliable scale? How much, on average, are teaching evaluations affected by level of instruction and size of class?

3. Teaching Evaluation form from a social science perspective Closed and open ended questions Questions Format AskedPublicly Visible Part I Q 1-7Closed-endedUF wideYes Part I Q 8-9Closed-endedCLAS onlyNo Part I Q 10Closed-endedUF wideYes Part I Q 11-15Closed-endedUF wideNo Part II Q 1-5Open-endedUF wideNo

4. Closed Ended Questions

5. Open ended questions

6. Data From CLAS Fall 1995 through Spring 2010 84163 sections Only includes Publicly Visible Data Excludes CLAS items Excludes “control” variables Q11-15 Excludes open-ended questions

7. What does our teaching evaluation instrument actually tell us about our teaching? Are the items that students use to evaluate us actually measuring different things? Probably Not Students act as though they develop an attitude about the class and rank it on almost all items based on that attitude. Do the items in the teaching evaluation instrument actually produce a reliable scale? Yes

8. Inter-item correlations (CLAS Data, Fall 1995 through Spring 2010) Q1Q2Q3Q4Q5Q6Q7Q10 Q1 1.000 Q2 0.8841.000 Q3 0.9000.8721.000 Q4 0.7770.7840.8081.000 Q5 0.7720.7860.8060.8561.000 Q6 0.8200.8840.8490.7920.8221.000 Q7 0.8740.9290.8830.8170.8380.9131.000 Q10 0.8580.8980.8700.8270.8540.8970.9161.000 Cronbach’s alpha = 0.978

9. How much, on average, are teaching evaluations affected by level of instruction and size of class? SOME, but less than might be expected. Q10 = a + b 1 Lower + b 2 Grad + b 3 Log Enrollment + e LowerGrad 1000-2000 level10 3000-4000 level00 6000 level01

10. How much, on average, are teaching evaluations affected by level of instruction and size of class? SOME, but less than might be expected. Q10 = a + b 1 Lower + b 2 Grad + b 3 Log Enrollment + e b 1 will be the average effect of teaching a lower division course, relative to an upper division course, controlling for the size of the class. b 2 will be the average effect of teaching a graduate course, relative to an upper division course, controlling for the size of the class. b 3 will be the average effect of the log of class size, controlling for the level of the class.

11. Regression of Instructor Evaluation (Q10) on Level of Course and Class size (log) CLAS Lower -.129 (.004) Graduate.121 (.007) Lg enroll -.094 (.003) Constant 4.621 (.008) R2R2.052 N of cases 84163 Entries are unstandardized regression coefficients, with standard errors in parentheses.

12. Regression of Instructor Evaluation (Q10) on Level of Course and Class size (log) CLASHumanitiesSoc and Beh Sci Phys and Math Sci Lower -.129 (.004) -.094 (.006) -.104 (.010) -.043 (.007) Graduate.121 (.007).095 (.014).074 (.013).273 (.014) Lg enroll -.094 (.003) -.122 (.005) -.078 (.005) -.061 (.004) Constant 4.621 (.008) 4.791 (.016) 4.612 (.017) 4.346 (.013) R2R2.052.048.049.030 N of cases 84163258341600540511

13. Expected Values: Humanities SizeLowerUpperGrad 104.414.514.60 454.234.324.42 1504.084.184.27 3004.004.094.19 Expected Values: Phys and Math SizeLowerUpperGrad 104.164.214.48 454.074.114.39 1504.004.044.31 3003.954.004.27

14. Expected Values: Soc and Beh SizeLowerUpperGrad 104.334.434.51 454.214.314.39 1504.124.224.30 3004.064.174.24 Expected Values: Political Sci SizeLowerUpperGrad 104.374.574.51 454.224.424.35 1504.094.294.23 3004.024.224.16

15. Morals of the Story We have a reliable teaching evaluation instrument which is definitely measuring something. Sections that are evaluated positively on one item tend to be evaluated positively on other items. Reliability isn’t validity. Response set could be a problem, but the cost of fixing it would be a disruption in the continuity of the data that we have. Like GRE scores, these scores should be regarded as a good measure, but not the only measure.

16. Morals of the Story Most variation in course evaluations is NOT accounted for by level of instruction or class size. Both class size and level of instruction matter, but should not be regarded as excuses for really low evaluations.

17. Darts and Laurels Laurel – Brian Amos, my graduate research assistant, for help with analyzing these data. Laurel – Dave Richardson and CLAS office, for making these data available to me. Dart – Academic Affairs, for not making these data publicly available in a usable form to everyone. Laurel – Academic Affairs, for (finally) creating a link to allow promotion candidates and award nominees to generate teaching evaluation reports in Word automatically with just a few clicks.

18. Darts and Laurels Dart – Academic Affairs, for not posting the CLAS-only items, and not posting the teaching evaluations of graduate students who taught their own courses. Laurel – Academic Affairs, for an otherwise much improved website showing evaluations Laurel – You, for patiently listening Thanks!