Ritual or reality: do student evaluations have any effect on teacher thinking and practices? Presentation at the Australasian Higher Education Evaluation Forum (AHEEF), Christchurch, November, 2010 Sarah Stein, Jo Kennedy, Dorothy Spiller, Trudy Harris, Stuart Terry & Lynley Deaker

The Project Unlocking the Impact of Tertiary Teachers’ Perceptions of Student Evaluations of Teaching Ako Aotearoa (National Centre for Tertiary Teaching Excellence) National Project Fund grant ($150k plus $20k extension)

Project Partners From Left: Lynley, Sarah, Trudy, Dorothy, Jo & Stuart Otago Polytechnic Stuart Terry University of Waikato Dorothy Spiller Trudy Harris University of Otago Sarah Stein Jo Kennedy Lynley Deaker

Research question How do the current evaluation processes and practices influence teachers’ thinking and behaviours in relation to student learning at all stages of the teaching and learning cycle?

Objectives 1.Explore tertiary teachers’ perceptions about student evaluations and identify how those perceptions impact on their thinking and practices. 2.Explore how tertiary teachers make use of information obtained from student evaluations at all stages of the teaching and learning cycle. 3.Compare teacher thinking and behaviours around evaluations across three participating institutions. 4.Make recommendations about evaluation processes so that evaluations can be optimally used for teaching development and enhancing student learning.

Where we are up to... Literature Review – complete Questionnaire - complete, data analysis continues Interviews - underway

Literature Review: Questions gap between acceptance and use to inform practice using multiple sources of evaluation-how could it help teacher emotions and receiving student feedback relationship between teaching beliefs and using evaluations to inform teaching

Literature Review: Questions What interpretation guidance and PD support would help in the use of evaluations to inform teaching? What institutional support and incentives would help to integrate student evaluations and professional development? What are the perceptions about the ownership of the student evaluation process?

Progress to date: the questionnaire Developed a questionnaire for teaching staff with some variation in language to contextualise to each institution Piloted with a small sample from Otago and Waikato Universities Online for 3 weeks from 22 April 2010 and sent to 2445 teaching staff. Received 1065 responses – or 45%

Progress to date: interviews 20 interviews are being carried out at each institution due for completion in early 2011 Interviewees selected from the questionnaire respondents who volunteered – a range selected to explore stories behind the questionnaire responses

Next Steps – data analysis analysing questionnaire data using a range of processes looking at comparisons and variances between and within the groups (ongoing) coding of the qualitative data into themes – links with questions that emerged from the Literature Review this analysis has informed the selection of interviewees and the questions to be asked during the interview

Early Data Analysis Questionnaire data

Teacher perceptions of student evaluation data & influences on practice Q9 – To what extent do your reasons for using student evaluations influence your teaching decisions? ( E.g., decisions about learning outcomes, teaching strategies, assessment, etc. 1=A great deal, 5=Not at all) Q11 – To what extent does your institution’s use of student evaluation data influence your teaching decisions? (E.g., decisions about learning outcomes, teaching strategies, assessment, etc. 1=A great deal, 5=Not at all) Q13 – My course design refinements are influenced by student evaluation results: ( 1=A great deal, 5=Not at all) Q15 – My willingness to try new teaching approaches is constrained by the possible negative effects on my student evaluations. ( 1=A great deal, 5=Not at all )

Q9 – To what extent do your reasons for using student evaluations influence your teaching decisions? For example: decisions about learning outcomes, teaching strategies, assessment, etc. 1=A great deal, 5=Not at all Otago Polytechnic differs significantly from the two Universities for Q9 (P-value=0.044) Q9 – Teaching decisions influenced by reasons for doing student evaluations

q10 Comment Data - Theme Coding – all Institutions

Q 10 comments (reasons for influencing decisions) 1a I believe we must value how students interpret our teaching methods; process and especially how they value feedback with regard to their progress on a particular course or module. Students evolve intellectually and have a global perspective of where they fit in the world and change with it. We must be aware of those changes and expectations of our students globally; academically and culturally.

Q 10 – comments (reasons for influencing decisions) 1b Not everything is popular! Being confident (but not arrogant) as a teacher is important as well as being reflective. I continually reflect on my teaching.

Q 10 – comments (reasons against influencing decisions) 2a It is my perception that the evaluations are largely a measure of popularity rather than of learning/teaching. I believe that the students are generally not in a position at the time the evaluation is set to comment on their learning outcomes or my teaching.

Q 10 – comments (reasons against influencing decisions) 2b After 15 years I can tell if a course has worked or not. I am currently revising a whole course because I or it was not relevant to the students. I can see that in their faces; attentiveness and attendence/ participation.I don't need a survey to tell me that. Although it did confirm it.

Q9 & Years Teaching in the Tertiary Sector Q9 – To what extent do your reasons for using student evaluations influence your teaching decisions? For example: decisions about learning outcomes, teaching strategies, assessment, etc. 1=A great deal, 5=Not at all There was a significant difference within the groups based on the number of years experience in tertiary teaching (P-value=0.028)

Q11 – Institutional use of student evaluations as an influence on teaching decisions All 3 institutions are significantly different from each other (P-value=0.000), with Otago Polytechnic staff the most influenced. Q11 – To what extent does your institution’s use of student evaluation data influence your teaching decisions? E.g., : decisions about learning outcomes, teaching strategies, assessment, etc. 1=A great deal, 5=Not at all

Q13 - Course/paper design refinements influenced by student evaluations Both Universities are significantly different to Otago Polytechnic staff (P-value=0.004), but not from each other. Polytechnic staff have more of a perception that student evaluation results influence course/paper design. Q13 – My course design refinements are influenced by student evaluation results: 1=A great deal, 5=Not at all

Q15 - Teachers willingness to try new teaching methods constrained by evaluations There is a significant difference between Otago University and the other two institutions regarding teacher perceptions that student evaluation results constrain their willingness to try new teaching methods (P-value=0.000) Q15 – My willingness to try new teaching approaches is constrained by the possible negative effects on my student evaluations. 1=A great deal, 5=Not at all

Q4 – teachers’ activity after they receive results The categories that were most frequently used were: ▫ Q4(b) Read the open question comments made by the students - 86% rated 1 or 2 ▫ Q4(a) Spend time going over the data and responses – 80% rated 1 or 2 ▫ Q4(g) Actively look for feedback about teaching and assessment – 72% rated 1 or 2 ▫ Q4(d) Compare the data with previous evaluations – 57% rated 1 or 2 ▫ Q4(e) Discuss the results with colleagues/teaching team – 55% rated 1 or 2 Least used were : ▫ Q4(f) Seek assistance with interpreting the results from others – 19% rated 1 or 2 with 55% rating 4 or 5 ▫ Q4(c) Provide students with feedback on the results – 15% rated 1&2 with 62% rating 4 or 5

Q4 – teachers’ activity after they receive results Q4 – When you receive the results from your appraisals/evaluations do you… 1=always, 5=never There is a significant difference between institutions (F=21.210, p=0.000). The difference is between the two universities and the Otago Polytechnic.

And now…. Watch this space….