Review of SUNY Oneonta Course Evaluation Form Report and Recommendations from The Committee on Instruction: Part II October 4, 2010

Framework for Complete Re-design

Data Sources Peer-reviewed literature Surveys of teaching faculty –Spring 2009 and Fall 2009

Final Framework Based on Feldman’s (2007) summary of 23 empirical studies Advantages of using this framework: –it empirically discriminates which dimensions of teaching are the most important ones by examining: association with student achievement association with overall evaluations –offers reliable results (i.e., across many studies)

Item Pool Items on our final instrument were derived from three major sources: –existing course-evaluation instruments –peer-reviewed literature –surveys of teaching faculty

Criteria for Inclusion To be included on our instrument, each item needed to meet three criteria (see next slide):

Criteria for Inclusion 1)clearly relate to Feldman’s (2007) “high” or “moderate” (in importance) dimensions 2)be empirically associated with both high student achievement and high overall evaluations 3)be reasonably applicable to all of the following types of classes: freshman senior general education graduate distance-learning

Our New Instrument: Student Response to Faculty Instruction (SRFI)

SRFI Items 1.The instructor clearly explained his/her grading criteria, including how final grades in this course will be determined. 2.The instructor was clearly interested in the course material. 3.The instructor presented and explained ideas effectively.

SRFI Items, cont. 4.The instructor communicated the significance of the subject. 5.Throughout the course, the instructor made it clear what I should learn and accomplish. 6.The instructor was clearly interested in the learning of each student. 7.I would recommend this instructor to other students.

Pilot Testing

Fall classes 775 students responded Response rate: 81% Reliability estimate: Cronbach’s alpha:.93

Spring classes 638 students responded Response rate: 87% Reliability estimate: Cronbach’s alpha:.92

Validity Checks

Issue #1 Is students’ interest in the subject matter related to SRFI responses?

Correlations with Student Prior Interest Item text: “I was interested in the subject matter of this course even before it started.” Fall 2009 data: correlations ranged from.10 to.29 Spring 2010 data: correlations ranged from.09 to.25

Issue #1 Conclusion There is at best a weak relationship between prior interest in the subject matter and SRFI responses. This is very good news for gen. ed. instructors! They shouldn’t expect lower ratings on the SRFI simply because their course is gen. ed.

Issue #2 Are course grades related to SRFI responses?

Correlations with Course Grades Class-averages Fall 2009 data: correlations ranged from.01 to.23 (one correlation was negative: -.09) Spring 2010 data: correlations ranged from.02 to.23

Issue #2 Conclusion There is at best a weak association between course grades and student ratings on the SRFI. This means that “easy” courses don’t necessarily get high ratings from students.

Issue #3 Is class size related to SRFI responses?

Correlations with Class Size Course enrollment Fall 2009 data: class sizes ranged from 2 to 92 correlations ranged from.00 to -.36 Spring 2010 data: class sizes ranged from 2 to 60 correlations ranged from.02 to.12

Issue #3 Conclusion There is at best a weak association between class size and student ratings on the SRFI. This is good news for instructors of large classes; lower SRFI responses should not be expected simply because of class size.

Issue #4 Is course level related to SRFI responses?

Correlations with Course Level Fall 2009 data: 5 graduate level level level correlations ranged from -.01 to.22 Spring 2010 data: 2 graduate level level level correlations ranged from.06 to -.22

Issue #4 Conclusion There is at best a weak association between course level and student ratings on the SRFI. This is good news for instructors of first-year classes!

Committee Conclusions

SRFI’s Up We believe our pilot data provide satisfactory evidence of both the reliability and the validity of this instrument for all classes at SUNY Oneonta. There are no demographic questions on the SRFI. There will be space for free-response comments on the back of the form.

Validity Studies SPISRFI –Fall 2009 N = 775 –Spring 2010 N = 638

SPI < SRFI Advantages of the SRFI over the SPI: –demonstrated reliability and validity to the extent that we are able to measure validity without being invasive –shorter form requires less class time to administer –contributes to the College’s goal of sustainability 56 fewer pages of results for each renewal/tenure/promotion application

SPI/SRFI Comparison Item text: “The instructor presented and explained ideas effectively.”

Committee Recommendations

Package A: Recommendation Regarding the New Instrument (1 recommendation in this package)

Recommendation #1 The SRFI should replace the SPI, effective Spring 2011.

Package B: Recommendation Regarding Administration of the SRFI (1 recommendation in this package)

Recommendation #2 Departments should consider having a third party (not a class member) administer the SRFI. This person should verify that all forms correctly identify the course.

Package C: Recommendations Regarding Handling and Reporting of SRFI Results (6 recommendations in this package)

Recommendation #3 SRFI results should be returned to faculty only after grades have been recorded.

Recommendation #4 SRFI results should include: mean standard deviation frequencies

Recommendation #5 SRFI means should be reported to no more than one decimal place, and standard deviations to no more than two decimal places. Additional decimal places suggest a level of precision that does not exist.

Recommendation #6 SRFI results should be compared to those of the department and course level, never the college-wide average. 50% of SPI results are from 100-level courses.

Recommendation #7 SRFI results should be returned to faculty in a format that can be placed directly into dossiers without re-configuration or integration of comparative data.

Recommendation #8 Our committee’s published guidelines for interpreting SRFI data should be included as a cover sheet in all dossiers. Available on our web site.our web site

Package D: Recommendations Regarding Faculty Development to Improve Instruction (3 recommendations in this package)

Recommendation #9 Early-career faculty should be encouraged to consider administering anonymous surveys early in the semester. Areas of weakness can be identified and corrected prior to the end-of-semester SRFI.

Recommendation #10 Peer review should be strengthened, including the use of peer reviewers external to the discipline. See our web site for resources.our web site

Recommendation #11 Faculty should be encouraged to seek informal peer review to improve their teaching. This can be mutually beneficial!