Maximising student engagement in learning gain measurement Dr Liz Austen, Nathaniel Pickering and Alan Donnelly Sheffield Hallam University

www.blogs.shu.ac.uk/steer @SHU_StEER Institutional research and evaluation Student experience data Student engagement Student communications Brief introduction to STEER www.blogs.shu.ac.uk/steer @SHU_StEER

NMMLG (Office for Students) Interim evaluation highlighted a number of issues with the data collection process and a low response rate for the longitudinal sample. The evaluation of the NMMLG project, led by Sheffield Hallam University, will continue in the autumn, working with providers and others involved in the project and focusing on improving our understanding of students’ perceptions of learning gain. Brief overview of our role in LG (only use text from OFS announcement) Discuss need to fully understand student engagement/motivation/support/impact (wellbeing) www.blogs.shu.ac.uk/steer @SHU_StEER

www.blogs.shu.ac.uk/steer @SHU_StEER Student engagement LEGACY analysis indicates that to be effective and ensure engagement, learning gain needs to be embedded in the curriculum or as part of developing a set of capabilities and skills that students recognise as necessary. Define student engagement in relation to LG measures www.blogs.shu.ac.uk/steer @SHU_StEER

Survey Design Checklist 1. Access to survey 2. Level of student support 3. Incentives 4. Optimal length of survey 5. Timing and possible information fatigue 6. Question wording and ordering 7. Survey question formats 8. Reporting and debriefing Do we really need another survey? institutional approaches to surveys (using surveys as description of quantitative LG measurement) - referred to as exercise, test etc - VOLUME - REPETITION (of methods and theme) - EXPLOITATION - some HEIs have a blanket ban without a clear business case brief overview of SDC - developed by STEER and used as a basis for evaluation of NMMLG (CAN WE SAY THIS) can be used more widely than LG - handout to follow at the end of the session - contact us for more info and if you want to use please feedback to tell us how Click to animate - all discussion require the questions "do we really need another survey?" to be discussed and debated

Survey Design Checklist Criterion Sector Evidence LG Evidence Incentives - response rates improve when providing a clear narrative for how scheme incentives are administered at the outset - immediate and guaranteed reward alongside longer-term 'lottery' opportunities is optimum Fan and Yan (2010), Cobanoglu and Cobanoglu (2003), Porter and Whitcomb (2003), Singer and Ye (2013) - Retail voucher payment - Charitable donation - Prize draw (two prizes of £500 and five prizes of - £100) - Printing credit - Drink vouchers - Results/personal development - None SDC - each criterion was reviewed to draw on sector evidence to cite best practice. All areas can be applied to LG methodologies. literature review to build checklist - linked to evidence - linked to context (LG - but could be satisfaction, experience, engagement, emplyability)

How can we maximise student engagement in learning gain measurement? Apply your knowledge of LG best practice to the criteria on your handout. Discuss with the person next to you and design the best approach to maximise student engagement (one criterion per pair/group). Provide handouts each with one of the eight criteria for effective survey design (if you really have to do one). In pairs, discuss how you would operationalise this aspect of methodological design to maximise student engagement. www.blogs.shu.ac.uk/steer @SHU_StEER

Reflections and next steps Thoughts on best practice? Handout print version of SDC - Creative Commons license attribute to STEER @ SHU Publication to follow ACCESS - Response rates for students are usually much higher if surveys are embedded within their own programmes of study, rather than seen as additional or institutional. LENGTH - Studies show that an optimal time for students to engage in any survey is approximately 13 minutes. Handout print version of SDC - Creative Commons license attribute to STEER @ SHU take reflections from the audience? (Time) www.blogs.shu.ac.uk/steer @SHU_StEER

www.blogs.shu.ac.uk/steer @SHU_StEER