1. Student comments are just the start
Dr Liz Austen
Student Engagement, Evaluation and Research
@lizaustenbooth @SHU_StEER
#NJN2016

2. Wonderful, knowledgeable lecturers with plenty of

3. Wonderful, knowledgeable lecturers with plenty of industry experience

4. The classrooms are and

5. The classrooms are clean and warm.

6. Chaotic

7. Chaotic timetabling.

8. I would like to feel like I was getting better

9. I would like to feel like I was getting better value for money.

10. Benefits.....
more likely to influence change and enhance student experience
more effective at engaging staff in discussions around student feedback
focus on what students say ' in their own words' rather than responses to closed questions
qualitative comments shift the focus from student as consumer (metrics of satisfaction) to student as learner (context of learning)

11. Challenges.....
lacks the leverage in comparison to quantitative scores (Buckley 2012), esp in TEF agenda
often underused, analysed non-systematically, treated as an aside, (Stewart 2015, Grebennikov & Shah 2013, Zaitseva 2013) ...and limited published discussion
little systematic analysis of how quant and qual compare
privacy/confidentiality/ aggregated data - attempts to safeguard
focus on the negative (although comments are mainly positive)
generic and unhelpful? (inclination to quantify?)
focus on why rather than what.... for example the use of language can show better relationships with staff (Stewart 2015)
need for robust analysis by expert researchers

12. Methodological issues....
complexity
analysis ('thousands of relatively brief, disjointed, anonymous and highly polarised comments' (Zaitseva 2013, p. 227)
interpretation (semantics vs content 'the teachers were’ and ‘the teaching was')
coding ("some lecturers are poor" - negative or positive?)
the need to triangulate not isolate - with what?
resource intensive esp if comparing trends over time
trustworthiness (analysis e.g. bias) & authenticity (subjectivity e.g. contradiction)

13. Activity You will be provided with a set of qualitative comments:-
In groups, analyse your qualitative comments
Based on these findings, develop an Action Plan which includes
one example of data triangulation
one course level intervention aimed to improve the student experience
one request for collaborative working with STEER
Compare your findings in a large group discussion

14. What else could you do? More data collection?
Who analyses and when?
How should institutional researchers and course teams work together to interpret the comments?
Support to triangulate?
retention statistics, employability data, degree classifications, module grades, internal institutional survey results, module evaluation questionnaire data, equality data, external examiner reports, withdrawal surveys (Buckley 2012), UKES, Student Voice Report
More data collection?
focus groups, interviews, level 4 & 5 - note survey permissions
students as partners approach
'listening events' (Northumbria)
Student Survey Results Workshop (Lincoln)
NSS Staff/Student Conference (Bangor)
Appreciative Inquiry Workshops (SHU)

15. References
Alhija, F.N.-A., and Fresko, B. (2009). Student evaluation of instruction: What can be learned from students’ written comments? Studies in Educational Evaluation, 35, 37–44.
Buckley, A., (2012) Making it count: Reflecting on the National Student Survey in the process of enhancement. York: Higher Education Academy.
Grebennikov, L. and Shah, M., (2013). Student voice: using qualitative feedback from students to enhance their university experience. Teaching in Higher Education, 18(6), pp
Langan, A.M., Scott, N., Partington, S. and Oczujda, A., (2015). Coherence between text comments and the quantitative ratings in the UK’s National Student Survey. Journal of Further and Higher Education, pp.1-14.
Robinson, L., & Sykes, A., (2014) Listening to Students’ Views on NSS Data for Quality Enhancement, Health and Social Care Education, 3:1, 35-40
Stewart, M., (2015). The language of praise and criticism in a student evaluation survey. Studies in Educational Evaluation, 45, pp.1-9.
Zaitseva, E., Milsom C., & Stewart, M., (2013) Connecting the dots: using concept maps for interpreting student satisfaction, Quality in Higher Education, 19:2,
Kovacs, S., Grant, L. and Hyland, F., (2010). A study of the use of the National Student Survey to enhance the student experience in education departments. ESCalate, University of Bristol.