Weaving narrative and numbers to create a pattern of student success Professor Vicky Gunn (Head of Learning and Teaching, Glasgow School of Art) Professor Karl Leydecker (Vice- Principal (Learning and Teaching), University of Dundee Caroline Turnbull (Quality Enhancement Manager, QAA)

National Approaches: Scotland’s Quality Enhancement Framework Enhancement-Led Institutional Review Enhancement Themes Student engagement Public information Institution-Led Review Evaluating the quality of learning & teaching through an enhancement-led approach since 2003 Partnership of agencies In moving through this slide discuss student engagement in each aspect of the circle. ELIR: students on review teams students involved in institutional preparations including institutional teams, contributing to the development of RAs and the identification of contextualised themes and working through subsequent action plans ILR: Students on ILR panels Students and Institutions working in partnership to deliver training to support student panel members/committee members Students involved in prep work within schools/subject areas Student engagement: Every ELIR team have a student reviewer Student representation at every level in their institutions Students participate in every ELIR – i.e. spend time in meetings with the ELIR team Support for student representation – sparqs provide support and development for institutions and students Information on the SE through national, institutional and longitudinal student experience surveys. Then go on to discuss the Enhancement Themes QAAS, US, NUS(Scotland) and SFC Search for Impact of the Enhancement Theme on YouTube (or click here)

The Enhancement Themes Key element of the QEF Distinctive aspect of Scottish HE Theme chosen and delivered by sector in collaboration with QAA Scotland Fosters partnership and collegiality Allows us to achieve things collectively that we might not be able to do individually Focus on developing strategies, shaping policies, innovating practice 14 Years of The Enhancement Theme Assessment and Integrative assessment (2003-04) Responding to student needs (2003-04) Employability (2004-06) Flexible delivery (2004-06) First year: engagement & empowerment (2005-08) Research-teaching linkages: enhancing graduate attributes (2006-08) Graduates for the 21st century (2008-11) Developing and supporting the curriculum (2011-14) Student Transitions (2014-17) Evidence for Enhancement: improving the student experience (2017-20)

Why is this a compelling theme now? Evidence for Enhancement Why is this a compelling theme now? Vision for Evidence for Enhancement Help Scottish sector enhance student experience by improving how it uses data and evidence Broad view of student experience and attention to student engagement across all activity Demonstrate the impact of the Theme for our students and other key stakeholders Exponential increase in the availability and visibility of data and evidence Universities changing practice and policy at all levels through use of new data systems This ‘data revolution’ is affecting everyone and the system as a whole Aligns with Government priorities and developments elsewhere

Optimising the use of existing evidence Student engagement 19 Institutions pursuing projects across the following areas: The nature of evidence Approaches to evaluation Staff upskilling and empowerment Learning analytics and dashboard development Quality assurance processes Student characteristics Student success, retention and employability Student engagement and belonging Learning space and curriculum development Digital technology, learning, teaching and assessment Measuring beyond metrics Abertay University Learning analytics University of Strathclyde Creative disciplines Glasgow School of Art Distance learning Queen Margaret University Supporting Programme Leaders Edinburgh Napier University Graduate employment University of Dundee Optimising the use of existing evidence Student engagement Student demographics, retention, and attainment

Evidence for Enhancement at University of Dundee New approach of using evidence for enhancement to improve the student experience started in 2013 The aims were (and remain): a) to raise the status and profile of learning and teaching at the University and raise ambition b) to use evidence as a way of motivating staff to bring about cultural change c) to use evidence to identify areas of strength to celebrate and areas to focus on d) to build a partnership approach with students by freely sharing and discussing the data with them e) to build confidence that deliberate actions could bring about positive improvements Initial focus was on the National Student Survey

Evidence for Enhancement at University of Dundee Up to 2013 National Student Survey performance was a) patchy over the years (84, 88, 87, 86, 90, 88% overall satisfaction 2008-13) surprisingly poor when looked at in detail below the level of overall satisfaction: 57th in 2017 in the Sunday Times for student satisfaction based on average of the themes plus overall satisfaction, and 77th in the Complete University Guide (which omits learning resources) part of the reason for quite a steep decline in UK University rankings up to 2013/14 poorly understood inside the University, with lack of awareness of increased competitiveness amongst English universities charging 9K fees basically not seen as that important, reflected in a low 65% response rate by students

Evidence for Enhancement at University of Dundee Actions taken Results out quickly to all staff in a form that can be easily digested Also disseminate results to student representatives directly Importance of benchmarking Transparency: celebrate excellent results and don’t hide poor results Light touch monitoring of School response (‘Hand the work back’) Organisation of annual NSS forum to celebrate success and share good practice Set target response rate and implemented a plan to achieve it, raising the profile of NSS with staff and students (high response rate increases the legitimacy of the NSS in eyes of staff and students)

Part of a wider series of initiatives Academic promotions system that removes binary divide between research and teaching Learning and Teaching Development Forums to share excellent practice Annual monitoring changed from just paper based exercise to a strategic dialogue Joint University / Student Union report on assessment and feedback Implemented TESTA to address very poor assessment and feedback scores Signed first of annual student partnership agreements in autumn 2013, strengthening further the already very close relations with DUSA, the Student Representative Council and School Presidents Improving student satisfaction now a joint responsibility of University and DUSA as part of the partnership agreement

Student Partnership Agreement One of the important things is that they contain KPIs that are owned jointly by the University and DUSA, eg recently: Improvements in NSS scores for student voice Improvements in NSS scores for optional question on Advisers of Studies Increased engagement with the University’s reading list management system Improved student engagement with DUSA elections Development of School Student Partnership Action Plans

Evidence for Enhancement at University of Dundee Evidence for Enhancement approach extended to other areas: widening access retention and progression rates of graduate level employment

Evidence for Enhancement sector wide Current Enhancement Theme is exploring how the same approach can be employed at sector level One key sector-level strand of work involves working with Institutional Planning Teams Initial focus was on an analysis of the Scottish sector’s NSS results as a whole Currently working on analysis of a selection of subjects Results can then be used to inform future sector level work under the aegis of the Theme and more broadly

Scoping the Creative Arts Territory in the Scottish Context Teaching Enhancements, Evidence Development, and the Interconnections with the Cultural Ecology of Scotland Collaborative authorship Glasgow School of Art, Scottish Royal Conservatoire, Edinburgh College of Art (University of Edinburgh), Queen Margaret University, Napier University, University of the West of Scotland, Duncan of Jordanstone College of Art (Dundee University)

Responding to the metrics march How can creative arts higher educators come to terms with, at the same time as challenging, the new metrics regime in a way that ultimately focuses on the impact of our learning and teaching regimes? Can the notion of the ‘cultural ecology’ help?

Creative Industries Strategy (UK) Where are our students in each of these arenas whilst they are studying and afterwards? Creative Industries Strategy (UK)

The creative arts’ existential crisis: Innovation and impact indicators How can creative arts students be brought together to generate new outcomes from the intermingling of analogue and digital wisdoms? Can this be done at the same time as ensuring innovations in each of these categories are also valued in their own right?

What the current metrics evidence means for creative arts: Heads of subjects = ‘creative producers’: Greater confidence in engagement with metrics and experimental data regarding student outcomes as evidence of impact. Work with reflective analysis rubrics that underpin annual discussions with programme leads, identifying the creative ‘legacy’ of the educational provision. Develop, define, engage with indicators of impact – this is where regional collaborations come in

Across our disciplines and institutions Impact- We are increasingly being asked not just to assert and describe excellent teaching and learning but also demonstrate the impact of it: Amongst our students Across our disciplines and institutions Beyond into our wider communities – Disciplinary-Professionally (typically in terms of research and dissemination as well as leadership) Wider influence (typically in terms of community engagement through widening access agendas, equality and diversity agendas, cultural ecology agendas, creative production initiatives).

How do we measure success? Indicators: What is evidence? Education Society Arts H.E. Wellbeing Culture Economy Innovation How do we measure success? Reach and significance of our learning and teaching?

Key conclusions regarding outcomes’ metrics (transferable to other disciplines?): Avoid fixating on current Graduate Outcomes measurements. They’re only a small part of the evidence. Understand how to work with the impact of our learning and teaching regimes in terms of reach and significance rather than description. Play a much more active role in the identification and closing of skills gaps in the range of policy ‘areas’ into which our students go and from this be open to: Offering innovative approaches to enabling graduate apprenticeships that might actually benefit us as a sector; Playing a significant role in identifying suitable input and outcome measures across the course of the learner journey that mitigate current metrics evidence.

This is less about whizzy learning analytics’ tools being used within programmes and more about getting diverse policy agendas to interface around their definitions of outcomes far more.