Innovations in evaluating & valuing public health teaching: The challenge of course evaluations Delia L. Lang, PhD, MPH Elizabeth Reisinger Walker, PhD, MPH, MAT Emory University, Rollins School of Public Health Behavioral Sciences & Health Education

Background Student ratings: Utilized across disciplines Inform undergraduate and graduate programs Conflicting findings on reliability, validity, and use Necessary but insufficient to address needs of multiple constituents Background Extensive literature documenting conflicting results regarding utilization of course evaluations (i.e. student ratings)

How are student ratings used? Students Planning course/ instructor selection Faculty Formative  course improvement Administrators Summative  performance assessment Students: tend to make course selections based on student ratings of faculty alone rather than ratings related to course content or structure Faculty: while some use info to modify courses and approach to teaching, many don’t find feedback useful, and don’t feel compelled to apply feedback to their pedagogy Administrators: Performance assessment for purposes of hiring, promotion, documenting oversight of teaching for accrediting bodies

How are student ratings used? Administrators Summative  performance assessment Faculty Formative  course improvement Students Planning  course/instructor selection Currently, the primary use of student ratings is for Summative purposes Primary rationale among supporters: students are best positioned to offer their perceptions of instructors and to evaluate the degree to which they learn as a result of instructors’ teaching effectiveness Dissenters argue that students often not motivated to thoughtfully provide ratings b/c any changes that may result wouldn’t benefit them, overwhelmed by demand to complete any ratings at a busy time of semester, While agreement exists that such assessments must be multi-dimensional to capture the many factors presumed to comprise effective teaching, no agreement exists on what these dimensions should be; for summative purposes, administrators focus on one global score Concerns also arise from mixed results regarding reliability and validity of student rating instruments (face and construct-related). Other issues with student ratings include bias based on the gender or race/ethnicity of the instructor and influences from other factors unrelated to teaching effectiveness (e.g., academic discipline, class level, class size, meeting times, or required versus elective courses)

Reliability Some reliable measures available; infrequently used In-house instruments lack psychometric support Validity Cohen 1981 meta-analysis  r=.43 Uttl et al. 2017 meta-analysis r=0 Teaching effectiveness & learning outcomes difficult to measure Are student ratings reliable and valid indicators of teaching effectiveness? 1975-1995 literature suggested small-moderate correlations between teaching effectiveness and learning outcomes based on meta-analyses and literature reviews Cohen 1981 meta-analysis  r=.43 Uttl et al. 2017 meta-analysis r=0; calls previous findings ‘voodoo’ statistics Mount Royal University, Canada – Studies in Educational Evaluation Most instruments assess multiple dimensions of teaching effectiveness and student learning, but one global score used for summative purposes Learning outcomes: typically measured in terms of grades, self-report statements on open ended questions “…universities and colleges focused on student learning may need to give minimal or no weight to SET ratings….universities and colleges focused on students' perceptions or satisfaction rather than learning …may need to terminate all faculty members who do not exceed the average SET ratings of the department or the university… despite common sense objections that not every faculty member can be above the average.” (Uttl et al., 2017)

Other ways to assess teaching effectiveness Student Evaluations & Feedback Student Ratings Student Group Instructional Diagnosis Focus groups Peer Feedback Observations Teaching Consultations Instructor Reflections Teaching Philosophy Documentation of innovative and inclusive teaching Description of pedagogical strategies Teaching Portfolio Professional development Evidence of teaching effectiveness from a multidimensional perspective Holistic Approach Other ways to assess teaching effectiveness External evaluation of teaching (peers/ pedagogy experts from university teaching centers): review, share, reflect on, and discuss teaching practices Concerns: who performs review: peer? Expert? How is sensitive feedback delivered? should review results be used for formative or summative purposes? Does the peer review itself modify the teaching effectiveness for the observed session? Student ratings: satisfaction with the course, perceived impact of teaching on learning SGID - Alternative methods: Mixed methods, at mid-term or end of term: A faculty member not associated with the course facilitates a discussion with the students about the course’s strengths and weaknesses and then provides the instructor with a summary of the feedback and recommendations. Focus groups at the end of the semester provide in-depth information to supplement formal student ratings (Lang et al. 2017). Additionally, conducting qualitative interviews or focus groups with alumni can provide insight into longer-term impacts of teaching and learning Concerns: time consuming – both on data collection and analysis end Professional development: instructors documenting attendance at pedagogy workshops, teaching mentorship, evolving teaching philosophy, demonstration of innovative teaching, SoTL

Role of Public Health in assessing teaching effectiveness Theoretical & methodological expertise Faculty Support SoTL in PH Role of Public Health in assessing teaching effectiveness PH has the theoretical and methodological expertise to develop innovative ways to measure teaching effectiveness and learning  think TEACHNG as INTERVENTION - a closer look at current instruments used to obtain student ratings may be warranted; however, efforts to collect data on other indicators of teaching effectiveness (as discussed on previous slide) is necessary; Schools and Programs of PH undergoing major curricular changes to align with new CEPH criteria – opportune time to elevate how we assess learning and mastery of competencies – may require more than just asking students and faculty, perhaps include alumni and employers Centers and Offices dedicated to teaching and learning, specifically housed within S & P of PH dedicated to academic support for faculty development in teaching and learning/ pedagogical scholarship and support of students; these units can lead efforts within their school/program to implement and evaluate innovative teaching strategies.