Student comments are just the start Dr Liz Austen Student Engagement, Evaluation and Research @lizaustenbooth @SHU_StEER #NJN2016

Wonderful, knowledgeable lecturers with plenty of

Wonderful, knowledgeable lecturers with plenty of industry experience

The classrooms are and

The classrooms are clean and warm.

Chaotic

Chaotic timetabling.

I would like to feel like I was getting better

I would like to feel like I was getting better value for money.

Benefits..... more likely to influence change and enhance student experience more effective at engaging staff in discussions around student feedback focus on what students say ' in their own words' rather than responses to closed questions qualitative comments shift the focus from student as consumer (metrics of satisfaction) to student as learner (context of learning)

Challenges..... lacks the leverage in comparison to quantitative scores (Buckley 2012), esp in TEF agenda often underused, analysed non-systematically, treated as an aside, (Stewart 2015, Grebennikov & Shah 2013, Zaitseva 2013) ...and limited published discussion little systematic analysis of how quant and qual compare privacy/confidentiality/ aggregated data - attempts to safeguard focus on the negative (although comments are mainly positive) generic and unhelpful? (inclination to quantify?) focus on why rather than what.... for example the use of language can show better relationships with staff (Stewart 2015) need for robust analysis by expert researchers

Methodological issues.... complexity analysis ('thousands of relatively brief, disjointed, anonymous and highly polarised comments' (Zaitseva 2013, p. 227) interpretation (semantics vs content 'the teachers were’ and ‘the teaching was') coding ("some lecturers are poor" - negative or positive?) the need to triangulate not isolate - with what? resource intensive esp if comparing trends over time trustworthiness (analysis e.g. bias) & authenticity (subjectivity e.g. contradiction)

Activity You will be provided with a set of qualitative comments:- In groups, analyse your qualitative comments Based on these findings, develop an Action Plan which includes one example of data triangulation one course level intervention aimed to improve the student experience one request for collaborative working with STEER Compare your findings in a large group discussion

What else could you do? More data collection? Who analyses and when? How should institutional researchers and course teams work together to interpret the comments? Support to triangulate? retention statistics, employability data, degree classifications, module grades, internal institutional survey results, module evaluation questionnaire data, equality data, external examiner reports, withdrawal surveys (Buckley 2012), UKES, Student Voice Report More data collection? focus groups, interviews, level 4 & 5 - note survey permissions students as partners approach 'listening events' (Northumbria) Student Survey Results Workshop (Lincoln) NSS Staff/Student Conference (Bangor) Appreciative Inquiry Workshops (SHU)

References Alhija, F.N.-A., and Fresko, B. (2009). Student evaluation of instruction: What can be learned from students’ written comments? Studies in Educational Evaluation, 35, 37–44.   Buckley, A., (2012) Making it count: Reflecting on the National Student Survey in the process of enhancement. York: Higher Education Academy. Grebennikov, L. and Shah, M., (2013). Student voice: using qualitative feedback from students to enhance their university experience. Teaching in Higher Education, 18(6), pp.606-618. Langan, A.M., Scott, N., Partington, S. and Oczujda, A., (2015). Coherence between text comments and the quantitative ratings in the UK’s National Student Survey. Journal of Further and Higher Education, pp.1-14. Robinson, L., & Sykes, A., (2014) Listening to Students’ Views on NSS Data for Quality Enhancement, Health and Social Care Education, 3:1, 35-40 Stewart, M., (2015). The language of praise and criticism in a student evaluation survey. Studies in Educational Evaluation, 45, pp.1-9. Zaitseva, E., Milsom C., & Stewart, M., (2013) Connecting the dots: using concept maps for interpreting student satisfaction, Quality in Higher Education, 19:2, 225-247 Kovacs, S., Grant, L. and Hyland, F., (2010). A study of the use of the National Student Survey to enhance the student experience in education departments. ESCalate, University of Bristol. https://www.heacademy.ac.uk/sites/default/files/8077.pdf