The role of course evals in faculty reviews: One perspective John Petraitis CAFE March 25, 2011

Summative vs. Formative Summative: – Course evals as ends – 1 st page of IDEA report Formative/Developmental – Course evals as means – Remaining pages of IDEA

Logic of IDEA One size does not fit all – Allows faculty to identify key objectives – E.g., exploration of personal value, vs. creativity vs. understanding principles Advertise the warts – Reliability and representativeness

Three Key Pieces 12-item Faculty Info Form (FIF) 12-item Faculty Info Form (FIF) 36-item Student Reaction to Instruction Form (SRIF) 36-item Student Reaction to Instruction Form (SRIF) Multi-page Report

FIF 12 learning objectives 12 learning objectives

FIF Which of the 12 objectives do I chose? Which of the 12 objectives do I chose? – Depends on course – Depends on instructor – Depends on section – Depends on dept Recipes for disaster – Choosing all of them – Choosing none of them (by not filling out FIF)

SRIF Two kinds of Q’s Two kinds of Q’s – 36 questions linked to 12 learning objectives. – Extraneous influences, e.g., Discipline Student motivation Class size

Marriage of FIF &SRIF Weighting of students responses (on SRIF) for results/report Weighting of students responses (on SRIF) for results/report M = 0 M = 0 I = 1 I = 1 E = 2 E = 2

The Report (page 1) Reliability and Representativeness – Reliability number (not percent) Based on number (not percent) of respondents Would adding a few more respondents potentially alter the results dramatically? – Representativeness percent (not number) Based on percent (not number) of respondents 65% response rate is considered representative

The Report (page 1) Reliability and Representativeness – Reliability number (not percent) Based on number (not percent) of respondents Would adding a few more respondents potentially alter the results dramatically? – Representativeness percent (not number) Based on percent (not number) of respondents 65% response rate is considered representative

The Report (page 1) Reliability and Representativeness – Representativeness is key – If representative, reviewers could use page 1 as summative and formative – If not representative, reviewer might use if it helps the candidate

Averages: – Section A: “I” and “E” objectives Adjustments to IDEA Ratings – Discipline (national comparisons) ‘level the playing field’ – “Extraneous Influences” on student ratings are used to adjust scores or ‘level the playing field’ SRIF Item #39: SRIF Item #39: I really wanted to take this course regardless of who taught it SRIF Item #43: SRIF Item #43: As a rule, I put forth more effort than other students on academic work Class size The Report (page 1)

(food for thought) Improving Teaching Effectiveness (food for thought) The Report (p. 2-3)

IDEA and your review file Do your own formative evaluation Do your own formative evaluation – What can you learn from pages 2-3? – Show your interest in professional development – Do it even if IDEA is not reliable or not representative Do your own summative evaluation Do your own summative evaluation – Make it easy. Make it clear.

Interpreting IDEA results Reliability – “Unreliable” if small # students responded (even in low- enrollment courses) – Would results fluctuate significantly if a few more people completed it? – IDEA’s standard is 10 respondents or 75% response rate. – Unreliable does not mean not useful – it just means that it cannot be part of IDEA’s data base for comparative purposes small classes w/ high response rates Consider high response rates in small classes as reliable information – FOR that class, if not for normative comparisons.