Student Evaluations of Teaching (SETs)

Slides:



Advertisements
Similar presentations
Assessment Assessment should be an integral part of a unit of work and should support student learning. Assessment is the process of identifying, gathering.
Advertisements

Using the Seven Principles as a Framework for the Evaluation of Student Ratings and Teaching Karl Wirth and Adrienne Christiansen Serie Center for Scholarship.
Designing Scoring Rubrics. What is a Rubric? Guidelines by which a product is judged Guidelines by which a product is judged Explain the standards for.
Using the IDEA Student Ratings System: An Introduction University of Saint Thomas Fall
IDEA What it is and How to Implement the System Texas A & M, February 2013 Shelley A. Chapman, PhD Senior Educational Consultant.
Faculty Evaluation Systems: Student Evaluations of Faculty What are we measuring? “The evaluation of teachers is a mark of a good college.” Ernest Boyer,
Susan Malone Mercer University.  “The unit has taken effective steps to eliminate bias in assessments and is working to establish the fairness, accuracy,
Update from the UNC General Education Council [presented to the UNC Board of Governors’ Educational Planning, Programs, and Policies Committee on February.
The Washington State University Critical Thinking Project Diane Kelly-Riley Kim Andersen Paul Smith Karen Weathermon Washington State University.
Office of Research, Evaluation, and Assessment April 19, 2008.
Standards and Guidelines for Quality Assurance in the European
 Description  The unit has a conceptual framework that defines how our programs prepare candidates to be well-rounded educators. Every course in the.
Implication of Gender and Perception of Self- Competence on Educational Aspiration among Graduates in Taiwan Wan-Chen Hsu and Chia- Hsun Chiang Presenter.
PDHPE K-6 Using the syllabus for consistency of assessment © 2006 Curriculum K-12 Directorate, NSW Department of Education and Training.
PROFESSIONAL DEVELOPMENT PLAN WORKSHOP. What is the Professional Development Plan? The Professional Development Plan is a directed planning and evaluation.
Creating a Teaching Dossier Shea Wang, Ph.D Interim Faculty Evaluation Coordinator Oct. 21, 2013.
EDU 385 Education Assessment in the Classroom
March 26-28, 2013 SINGAPORE CDIO Asian Regional Meeting and Workshop on Engineering Education and Policies for Regional Leaders Programme Evaluation (CDIO.
Main components of effective teaching. SRI validity A measuring instrument is valid if it measures what it is supposed to measure SRIs measure: Student.
Using Assessment Data Helen Thumann Department of Education.
University Teaching Symposium January 9,  Individual Development and Educational Assessment  Kansas State University  A diagnostic to.
“Developing Faculty Capabilities to Support Integrative Learning A Session Led by: L. Dee Fink, Ph.D. International Consultant in Higher Education AAC&U.
What Your Program Needs to Know about Learning Outcomes Assessment at UGA.
Stetson University welcomes: NCATE Board of Examiners.
Critiquing Quantitative Research.  A critical appraisal is careful evaluation of all aspects of a research study in order to assess the merits, limitations,
Then Now  Teaching as Art  Teachers and Teaching  Great teachers are born  How did I do?  Scholarship informs Teaching  Culture of Unexamined assumptions.
325K: COMBINED PRIORITY FOR PERSONNEL PREPARATION Webinar on the Annual Performance Report for Continuation Funding Office of Special Education Programs.
325D: PREPARATION OF LEADERSHIP PERSONNEL Webinar on Project Objective and Performance Measures for the Annual Performance Report for Continuation Funding.
FAIMER Assessing Teaching Excellence John Norcini, Ph.D.
Copyright © Springer Publishing Company, LLC. All Rights Reserved. BECOMING A SCHOLAR IN NURSING EDUCATION – Chapter 16 –
Standard 4: Faculty, Staff, & Students 1. Standard 4: Faculty, Staff, and Students Standard 4: Faculty, Staff, and Students (#82) INTENT STATEMENTS 4.1.
Good teaching for diverse learners
Designing Scoring Rubrics
Student Evaluations of Teaching
Classroom Assessments Checklists, Rating Scales, and Rubrics
Direct vs Indirect Assessment of Student Learning: An Introduction
Types of Tests.
Types of Tests.
A Pilot Study of the DAPTM Interview in the Online Environment
Eastern’s Assessment System
Faculty Evaluations: Tenured, Tenure-Track, and Associate
CRITICAL CORE: Straight Talk.
M-LANG project  Ref. n NO01-KA Interactive Exchange Workshop on how to use response systems and ICT tools for creating interactive learning.
Director of Policy Analysis and Research
Types of Tests.
Teaching Evaluations at TTU Using the IDEA Instrument
Elayne Colón and Tom Dana
ASSESSMENT OF STUDENT LEARNING
Tony Kirchner Developing a Quality Assurance Plan with the Teacher Work Sample as the Linchpin Tony Kirchner
Classroom Assessments Checklists, Rating Scales, and Rubrics
Using Data to Inform Instruction
Derek Herrmann & Ryan Smith University Assessment Services
Introduction to Student Achievement Objectives
The Efficacy of Student Evaluations of Teaching Effectiveness
Student Evaluations of Teaching (SETs)
Critically Evaluating an Assessment Task
Delia L. Lang, PhD, MPH Elizabeth Reisinger Walker, PhD, MPH, MAT
Assessments: Beyond the Claims
Links for Academic Learning: Planning An Alignment Study
Using Learning Gain Data as Evidence for Enhancement
Deconstructing Standard 2a Dr. Julie Reffel Valdosta State University
IDEA Student Ratings of Instruction
edTPA Coordinator, Illinois State University
Rubrics for evaluation
STUDENT GROWTH OBJECTIVES
Selecting Evidence to Demonstrate Teaching Effectiveness
TLQAA STANDARDS & TOOLS
ISU Teacher Education Quality Assurance System
The Teacher Work Sample: An Authentic Assessment
Presentation transcript:

Student Evaluations of Teaching (SETs) Review on the use of SETs, with recommendations for assessment of teaching faculty for employment decisions

SET questions revised in 2017 Previous questions had been used since 2011 Process included an in-depth review of existing research on SETs Committee found that SETs are of limited value for summative evaluations Summative = absolute evaluation of teaching effectiveness Set out to develop questions of greatest value for formative evaluations Formative = with a focus on improving effectiveness Questions based on the 7 Principles for Good Practice in Undergraduate Education (Chickering and Gamson 1987)

NEW SET questions I was encouraged to interact with the instructor regarding course content I was provided opportunities to cooperate with other classmates about course material I was informed of the instructor’s high expectations for my work in this course. I was provided with an evaluation of my academic progress at regular intervals during the semester. I was provided with ample opportunities to apply my learning in this course. I was prompted to think critically about the course material. I was provided an environment that supported my learning.

Committee Report on Use of SETs Review of existing literature on utility of SETs Review expert advice on how SETs should be used for evaluations of teaching effectiveness Recommendations on the how teachers should be evaluated at Auburn University

Existing literature on SETs Students are not qualified to evaluate faculty for teaching effectiveness Evidence for the validity of SETs for measuring individual teaching effectiveness is weak (at best) Many sources of bias in SET scores exist

Students not qualified to evaluate… Effectiveness of a teacher Numerous skills that define teaching effectiveness Knowledge, methods, course design, use of technology, course materials, grading, etc. SETs are used to gather the collective views of a group of students about their experience

SETs as measures of effectiveness Numerous meta-analyses of the relationship between SET scores and student learning Cohen (1981, 1982, 1983); Feldman (1987); McCallum (1984); Clayson (2009). Correlation coefficient (r) from 0.13 to 0.44

r = 0.5

SETs as measures of effectiveness Numerous meta-analyses of the relationship between SET scores and student learning Cohen (1981, 1982, 1983); Feldman (1987); McCallum (1984); Clayson (2009). Correlation coefficient (r) from 0.13 to 0.44 If r = 0.44, 80% of variation in SET due to something else Valid instruments for measuring students learning in aggregate.

Small sample bias in previous studies Uttl et al. 2017 Previous studies did not consider effect of small sample bias Studies with small sample sizes need stronger effects to get significance p-values Studies without significant p-values don’t get published

Feldman 1989 Clayson 2009

Small sample bias in previous studies Uttl et al. 2017 Previous studies did not consider effect of small sample bias Re-analysis of previous data Adjusted r from 0.05 to 0.27 Analysis of new data Adjusted r from -0.02 to -0.04

Research on Bias in SETs Numerous potential sources Faculty rank Faculty gender Student gender Faculty expressiveness Student motivation Required course or not Expected grade Level of course Class size Academic discipline Student workload

Global questions “The instructor’s overall effectiveness was.…” Particularly inappropriate for evaluating instructors Low reliability Not fair to evaluate based on a single score Use all scores Violates standards related to personnel evaluation

Implications SETs are not valid instruments for measuring effectiveness of individual teachers. SETs should not be used in a summative manner for evaluating teachers.

Recommendations SETs should not be used in summative manner SETs could be used in a formative manner Regardless of absolute scores, do scores improve over time? Indicative of a teacher that takes student learning seriously

Recommendations Number of evaluations if used No Global Questions! Minimum 6 to 8 courses total Courses with fewer than 10 raters excluded For summative employment decisions: 70% response rate minimum For formative considerations: 30% response rate acceptable No Global Questions! Use all scores or means from scores

Recommendations No comparisons with other faculty Prone to bias due to gender, race, other characteristics

SETs MUST be used with other metrics Peer review of course material Peer review of course instruction Review of metrics of teaching effort by supervisor (course policies followed; grades submitted on time) Review by expert outside sources (Biggio) Exit and alumni ratings

Other metrics of teaching performance Employer ratings Teaching scholarship Teaching awards Learning outcome measures Teaching portfolios Self-evaluation of teaching Generation of quantitative rubrics for any