“if student ratings are part of the data used in personnel decisions, one must have convincing evidence that they add valid evidence of teaching effectiveness” (Wilbert McKeachie, 1996, American Council of Learned Societies p. 3). UNCW Ad Hoc Committee “Use of SPOTs for RTP” Report to Faculty Senate, January 18, 2011

Charge to Committee To investigate and report as to what empirical research has found regarding the validity of SETEs (Student Evaluations of Teaching Effectiveness) in measuring teaching effectiveness (if they are not valid, or of low validity should UNCW be using SETEs in RTP, etc?). What do UNCW and UNC documents actually say about using SETEs for purposes of RTP? If SETEs are not valid measures of teaching effectiveness, what should be used to measure teaching effectiveness? Provide information about how other universities have attempted to resolve these issues. Develop recommendations for the UNCW Faculty Senate to discuss and possibly vote.

Committee Membership and Process Members (Past and Present RTP chairs, etc.) Craig Galbraith, (Management, past UNCW RTP Committee member and chairperson, ); John Fischetti ( Education; current UNCW RTP Committee member and chairperson, 2010-present); Barry Wray (Information Systems and Operations Management); John Taggart (Environmental Studies), Regina Felix (Foreign Language and Literatures), Susan Roberts (Clinical Research, Nursing), Yaw Chang (Mathematics and Statistics; former member of UNCW RTP Committee) Bruce McKinney (Communications; President, UNCW Faculty Senate) – ex officio, non voting member Process Bi-weekly meetings, rely on “scholarly” research (not opinions), meeting with student government representatives

Charge 1 Are SPOTs a Valid Measure of Teaching Effectiveness? – NO (particularly Global Measure) Studies indicate between 6% and 20% of teaching effectiveness captured by SETEs Almost all early studies were TAs in Freshman courses, with little control over course – can not be generalized to most university courses Newer studies indicate SETEs can be easily manipulated by instructors (such as giving chocolates, grades, etc.) or negatively biased (by a single comment, etc.) Newer studies indicate SETEs really measure “likability” and “attractiveness” Newer studies indicate high SETEs actually associated with lower student learning

Source: 1,800 students in 116 classes (87 instructors) using standardized learning outcome tests; Galbraith, Merrill & Kline, Research in Higher Education (forthcoming), 2011

Charge 2 UNC Policy (section ) Some student evaluation MUST be used Doesn’t specify weighting Warns against comparing SETEs with other faculty, departments, university States that that SETEs are most effectively used to compare changes over time (Specific sections of UNC Policy are provided in report) UNCW Policy “SPOT results are appropriately used for annual merit evaluation summaries, consideration for salary raises, RTP, and post- tenure-review decisions” “it is strongly suggested that peer and student evaluations be given similar emphasis in personnel recommendations.“ UNCW Reality SPOTs, Q16 are used extensively, and sometimes almost exclusively, for RTP SPOTS are typically compared across departments and university (quintiles) in reports

Charges 3 and 4 3: What can be used? Faculty mixed about peer evaluations, but generally positive about validity As SETEs became more dominant, peer evaluations became fewer Few “validity” studies published on peer evaluation 4: Universities are all over the board with SETEs U. of Minnesota (no global question, direct peer evaluations are weighted more) Southeast Missouri State Uses IDEA instrument Each department drives questions SETEs can not count more than 33% of measure of teaching effectiveness (based on IDEA Center’s recommendations)

Charge 5: Recommendations Recommendation 1: Eliminate the global question (Q16 from UNCW SPOTs) Recommendation 2: For RTP, Q1 to Q15 are not reported with any department, school, or university averages, quintiles, or categorical statements such as above average, average, etc. Recommendation 3: Q1 to Q15 be reported for RTP only as a comparison over time for that particular instructor. Recommendation 4: That some questions on the SETEs be tailored to specific departmental missions and expectations. Recommendation 5: Quantified SETEs (UNCW) cannot be weighted more than 25% to 33% as an assessment for teaching effectiveness for purposes of RTP. Recommendation 6: That the qualitative/written comments from student evaluations be more systematically administered and reported in order for them to be used more effectively in RTP decision.

Charge 5: Recommendations Recommendation 7: That the individual faculty member has a right to formally comment, explain, or respond to student evaluations (either quantitative or qualitative/written comments) for purposes of RTP, and that these comments, explanations, or responses be formally included as part of the SETE reporting process. Recommendation 8: That the process of peer evaluation of teaching be more uniform between departments and Schools within UNCW in order for them to be used more effectively in RTP decision. Recommendation 9: That peer evaluations be weighted at least equal to student evaluation of teaching (quantified question and student written comment section) for purposes of RTP Recommendation 10: That UNCW investigate using the IDEA Center’s evaluation system for student evaluation ( However, no global question should be used, and any quantified SETE process should never be weighted more than 25% to 35% as an indicator of faculty teaching effectiveness