1. Innovations in evaluating & valuing public health teaching: The challenge of course evaluations
Delia L. Lang, PhD, MPH
Elizabeth Reisinger Walker, PhD, MPH, MAT
Emory University, Rollins School of Public Health
Behavioral Sciences & Health Education

2. Background Student ratings: Utilized across disciplines
Inform undergraduate and graduate programs
Conflicting findings on reliability, validity, and use
Necessary but insufficient to address needs of multiple constituents
Background
Extensive literature documenting conflicting results regarding utilization of course evaluations (i.e. student ratings)

3. How are student ratings used?
Students
Planning course/ instructor selection
Faculty
Formative  course improvement
Administrators
Summative  performance assessment
Students: tend to make course selections based on student ratings of faculty alone rather than ratings related to course content or structure
Faculty: while some use info to modify courses and approach to teaching, many don’t find feedback useful, and don’t feel compelled to apply feedback to their pedagogy
Administrators: Performance assessment for purposes of hiring, promotion, documenting oversight of teaching for accrediting bodies

4. How are student ratings used?
Administrators
Summative  performance assessment
Faculty
Formative  course improvement
Students
Planning  course/instructor selection
Currently, the primary use of student ratings is for Summative purposes
Primary rationale among supporters: students are best positioned to offer their perceptions of instructors and to evaluate the degree to which they learn as a result of instructors’ teaching effectiveness
Dissenters argue that students often not motivated to thoughtfully provide ratings b/c any changes that may result wouldn’t benefit them, overwhelmed by demand to complete any ratings at a busy time of semester, While agreement exists that such assessments must be multi-dimensional to capture the many factors presumed to comprise effective teaching, no agreement exists on what these dimensions should be; for summative purposes, administrators focus on one global score
Concerns also arise from mixed results regarding reliability and validity of student rating instruments (face and construct-related). Other issues with student ratings include bias based on the gender or race/ethnicity of the instructor and influences from other factors unrelated to teaching effectiveness (e.g., academic discipline, class level, class size, meeting times, or required versus elective courses)

5. Reliability
Some reliable measures available; infrequently used
In-house instruments lack psychometric support
Validity
Cohen 1981 meta-analysis  r=.43
Uttl et al meta-analysis r=0
Teaching effectiveness & learning outcomes difficult to measure
Are student ratings reliable and valid indicators of teaching effectiveness?
literature suggested small-moderate correlations between teaching effectiveness and learning outcomes based on meta-analyses and literature reviews
Cohen 1981 meta-analysis  r=.43
Uttl et al meta-analysis r=0; calls previous findings ‘voodoo’ statistics
Mount Royal University, Canada – Studies in Educational Evaluation
Most instruments assess multiple dimensions of teaching effectiveness and student learning, but one global score used for summative purposes
Learning outcomes: typically measured in terms of grades, self-report statements on open ended questions
“…universities and colleges focused on student learning may need to give minimal or no weight to SET ratings….universities and colleges focused on students' perceptions or satisfaction rather than learning …may need to terminate all faculty members who do not exceed the average SET ratings of the department or the university… despite common sense objections that not every faculty member can be above the average.” (Uttl et al., 2017)

6. Other ways to assess teaching effectiveness
Student Evaluations & Feedback
Student Ratings
Student Group Instructional Diagnosis
Focus groups
Peer Feedback
Observations
Teaching Consultations
Instructor Reflections
Teaching Philosophy
Documentation of innovative and inclusive teaching
Description of pedagogical strategies
Teaching Portfolio
Professional development
Evidence of teaching effectiveness from a multidimensional perspective
Holistic Approach
Other ways to assess teaching effectiveness
External evaluation of teaching (peers/ pedagogy experts from university teaching centers): review, share, reflect on, and discuss teaching practices
Concerns: who performs review: peer? Expert? How is sensitive feedback delivered? should review results be used for formative or summative purposes? Does the peer review itself modify the teaching effectiveness for the observed session?
Student ratings: satisfaction with the course, perceived impact of teaching on learning
SGID - Alternative methods: Mixed methods, at mid-term or end of term: A faculty member not associated with the course facilitates a discussion with the students about the course’s strengths and weaknesses and then provides the instructor with a summary of the feedback and recommendations. Focus groups at the end of the semester provide in-depth information to supplement formal student ratings (Lang et al. 2017). Additionally, conducting qualitative interviews or focus groups with alumni can provide insight into longer-term impacts of teaching and learning
Concerns: time consuming – both on data collection and analysis end
Professional development: instructors documenting attendance at pedagogy workshops, teaching mentorship, evolving teaching philosophy, demonstration of innovative teaching, SoTL

7. Role of Public Health in assessing teaching effectiveness
Theoretical & methodological expertise
Faculty Support
SoTL in PH
Role of Public Health in assessing teaching effectiveness
PH has the theoretical and methodological expertise to develop innovative ways to measure teaching effectiveness and learning  think TEACHNG as INTERVENTION - a closer look at current instruments used to obtain student ratings may be warranted; however, efforts to collect data on other indicators of teaching effectiveness (as discussed on previous slide) is necessary;
Schools and Programs of PH undergoing major curricular changes to align with new CEPH criteria – opportune time to elevate how we assess learning and mastery of competencies – may require more than just asking students and faculty, perhaps include alumni and employers
Centers and Offices dedicated to teaching and learning, specifically housed within S & P of PH dedicated to academic support for faculty development in teaching and learning/ pedagogical scholarship and support of students; these units can lead efforts within their school/program to implement and evaluate innovative teaching strategies.