1. Student Evaluations of teaching: Do they matter?
Dorothy Spiller
Teaching Development Unit
University of Waikato, Aotearoa/ New Zealand

2. Mihi
E nga mana , e nga reo To those gathered here from diverse backgrounds E rau rangatira ma To my esteemed audience Tenei te mihi ki a koutou I greet you

3. Mihi Ka nui te hari me te koa I’m so happy
Ki te hoki mai ki tenei whenua
I’m so happy to return to this land
Ki taku ukaipo, ki taku whenua tapu
The land that provided me with nurturing and sustenance, the land of my birth

4. Mihi
Ka nui hoki te mihi ki nga iwi o tenei whenua I also wish to express my thanks to the people of this land Ki aku matua me aku tamariki To my parents and my children Mo nga taonga kua takoha mai ki ahau For the gifts that have been given to me No reira, tena koutou, tena tatou katoa Greetings once again to all of you/us who are gathered here

5. Do student evaluations matter?
For who?
To who?
For what?
Why?
When?
How?

6. The Quest What do teachers think about them?
Why do they have these views?
How do teachers engage with them?
How can we enhance student evaluation processes to optimise their potential to improve teaching and learning?

7. The Story The Beginnings The Quest The Equipment The Landscape
Re-designing the Landscape
The Holy Grail
Do the role play, shed the light on typical encounters when staff talk about appraisal

8. The Story – Once upon a time...
How did it all begin?
Do the role play, shed the light on typical encounters when staff talk about appraisal
Our work, institutions uses etc – is it useful, is it doing what we want?

9. The Story - Myth or Reality?
The institution uses this to judge me
People manipulate appraisals
Students are not in a position to make judgements
The instrument is invalid
I know my discipline and what they need to know
Are the questions relevant?
I can't try anything new because of the risk to my promotion
It is a popularity contest
These are the typical narratives that we encounter, and which we found reported by academics in the literature.
We need to find out is this what academics really think, and what impact these perceptions have on the process, and teaching and learning behaviours

10. The Story – The Quest
What do teachers really think? How does it make a difference to them, their teaching and their learners?
We need to find out is this what academics really think, and what impact these perceptions have on the process, and teaching and learning behaviours

11. The Story – The Quest
How do current evaluation processes and practices influence teachers’ thinking and behaviours in relation to student learning at all stages of the teaching and learning cycle?
What are the perceptions that tertiary teachers hold about student evaluations?
What factors affect these views?
How do tertiary teachers engage with student evaluations?

12. The Story - The Equipment
Ako Aotearoa funded NPF research
Two year project
Three institutions – Otago University, Otago Polytech and the University of Waikato
Researchers were Academic Developers and Evaluations personnel from these institutions

13. The Story - Count and find out...
Survey – online, run over three weeks with weekly reminders
Participants – goal to reach as broad a sample of the academic community for trends rather than in depth analysis
Institution
Total Staff
Number of responses
Response rate
University of Otago (OU)
1443
670
46%
University of Waikato (WU)
663
242
37%
Otago Polytechnic (OP)
320
153
48%
Total
2426
1065
44%

14. The Story - Ask and ye shall find...
Interviews – Semi-structured. Questions developed on the basis of themes that emerged from questionnaire data
Participants – Called for volunteers – selected based on demographics to get a range of disciplines and years’ experience
Literature review - to sharpen interview questions
Environmental scan of institutions, around appraisal practice, mission statements, teaching policies

15. The Story - Sense making
Survey – Quantitative analysis of summative questions and thematic analysis of associated comments (interpretivist paradigm)
Interviews –Thematic analysis

16. views?
Discoveries
What are teachers’ perceptions of student evaluations?
Why do they hold these views?

17. Discoveries: Appraisals Landscape
Widespread recognition of the worth of collecting this data : 73%
Main uses and limitations of appraisal:
Use
Responses
Informs teacher/course development
19%
Helps identify students’ learning needs
Limitations
Shortcomings in the current appraisal system
14%
Quality of responses by students
10%
Use for quality/summative processes 7%

18. The Story - Teachers’ Narratives
Appraisals are okay, but – but – but (mistrust of students’ judgement, of the instrument and the institution)
Read what students say
Appraisals can help me with my courses
I feel strongly about this
The timing is hopeless
The process ends here
I work alone
We don’t talk about them

19. The Story: Teachers’ Narratives
Beliefs about teaching and learning
Teacher manipulation
Status of teaching
Research focus
Need for multiple sources of
evaluation

20. The story – Emotional Terrain
Excited; cleansed; pissed off; crushed; feeling vulnerable; focus on negative; terrified; disappointed
OU – 10 indicated some form of emotional reaction positive and negative. Language suggests emotional intensity
WU – 4 general emotional significance and 3 directly negative
OP – 6 openly discussed feelings of vulnerability.

21. Discussion: join the quest!
What perceptions of evaluations have you encountered?
What are the reasons given for these perceptions?

22. The Story – Appraisals landscape
How do the respondents engage with appraisal data?
For WU (92%) and OU (95%) - ranked first to get feedback on students’ learning experiences
OP: Because it is required
Interpreting and responding to the data
For all institutions the majority of participants rank reading the open question/comments as their first response, and high numbers of participants spend time going over the data, look for feedback on teaching and compare with previous results.
However percentages are far lower when looking at active engagement and discussion with colleagues and students. This is most noticeable in the universities as opposed to Otago Polytech

23. Survey of the landscape
Use 1. Read the open question/comments made by students 2. Spend time going over the data and responses 3.= Actively looking for feedback about teaching and assessment Compare the data with previous evaluations 5. Discuss the results with colleagues/teaching teams 6. Provide students with feedback on the results 7. Seeking assistance with interpreting the results from others

24. The Story – Appraisals Landscape
How do teachers feed the outcomes back to students?
One of the lowest ranked use of appraisal data was to provide feedback to students. The universities were significantly lower than the polytech in this area.
The most preferred way of communicating with students was through paper outlines
The most commonly cited reason for not feeding back to students was the timing of appraisal

25. How do teachers use evaluations for professional development?
47% discussed with colleagues –mainly on course and paper refinements
Institutional use for promotion inhibits professional development
Improvements for professional development include flexibility in the instrument and the use of other evaluation methods

26. Stories from the interviews
Across the board there is a small percentage of staff who have systematic and deliberate strategies for feedback to students although a number have informal approaches and/or report to the next cohort
Timing of evaluations is cited as a major problems in terms of feedback to students
Discussion with colleagues is very limited, with a fairly strong culture of privacy around evaluations
More engagement with feedback at the polytechnic

27. Join the journey: discussion
To what extent do you consider staff at your institution engage with formal evaluations?
How do staff provide feedback to students on their response to student evaluations?
What strategies can we use to enhance engagement with evaluations feedback for professional development and student learning?

28. The story – the holy grail
1. That institutions ensure that there is clear alignment between their vision/policy statements concerning the auditing and developmental purposes of student evaluation systems and their implementation.
2. That institutions implement a professional development policy that includes explicit support for the education of staff and students about the purposes of student evaluation for curriculum and teaching, and the institutional intents and purposes of its student evaluation system.

29. The Holy Grail
That those who administer student evaluation systems recognise the variety of staff perceptions about student evaluations and provide communication, support and resources that address teacher expectation and needs, without compromising institutional intents and purposes.
That institutions ensure expectations about teacher and student roles and responsibilities in evaluations are unambiguous, and connections among performance, evaluation and rewards are clearly understood.

30. The Holy Grail
That teachers/faculties/departments schools and institutions embed within the evaluation policies and practices the notion that a well-rounded representation of teaching and courses is more likely to be achieved by drawing on multiple forms of evaluation data.
That professional development and course enhancement are firmly ensconced as the foundation and foci of student evaluation processes and practices.

31. Policies, perspectives, practices
Do policies highlight and support the developmental as well as the auditing purposes of student evaluation?
Do evaluation instruments and processes fit the needs and context of different programmes and levels?
Are evaluation questions informed by current research on teaching and learning?
Is there a culture that promotes conversation around teaching, learning and evaluation?
Is the evaluation process an integral part of the entire teaching and learning process?

32. Policies, perspectives, practices.
Does the institution provide support and guidance in relation to the interpretation of evaluation feedback?
Is there an institutional requirement for teachers to communicate with students about their responses to student feedback?
Are students provided with education about evaluations and their use?
Does the institution include evaluations education in staff professional development requirements?
Are staff required to demonstrate their response to evaluations feedback and undertake appropriate PD?

33. The case for mid-semester evaluations
Closing the loop
Feedback
Dialogue
Student engagement
Integral to teaching and learning process

34. Closing the loop
Student feedback
Analysis
Action
Feedback to students

35. The End or the beginning?