Welcome 1

Approaches to Self-Evaluation in Scottish Higher Education Institutions Paddy Maher Emeritus Professor, University of Highlands and Islands and QAA consultant 2

1.Institution-led quality reviews 2.QAA Enhancement-Led Institutional Review 3.Public information 4.Student engagement 5.Enhancement Themes Quality Enhancement Framework 3

A QAA Scotland project in Academic Year which aims to: Look at the mechanisms that HE Institutions use to evaluate learning and teaching practices and identify effective features Encourage institutions to reflect on and share experience of their approaches to self-evaluation Institutional approaches to self- evaluation (IASE) project 4

Institutional Review: ELIR ( or internal) Periodic Review Annual Monitoring Scope: monitoring and review processes 5

Literature survey Consultation Identification of factors facilitating effectiveness Dissemination of findings Methods 6

Evaluation is a growth area but little about internal reviews – ‘The limited number of papers on internal QA demonstrate how the quality debate has been dominated by the activities of external agencies.’ [Harvey and Williams, 2010 ] Lack of trust in top-down, externally-driven QA models – ’...20 years of QA has seen the systematic misalignment of quality culture and academic culture. Furthermore the situation is getting worse......(with notable exceptions, such as Scotland)...’ [Harvey, 2010] The wider evaluation literature 7

‘A modern university? – ‘...a series of separate schools and departments held together by a central heating system.’ [attributed to Robert Hutchins] Differing social practices – ‘.... quality and evaluative approaches which ignore the social practices of those involved risk alienating staff and introducing unsustainable initiatives.’ [Bamber, 2011] Developing trust: recognising diversity 8

Developing trust: ‘Staff and students are our university’s biggest asset, and a deep trust in their willingness to be professional and committed should be the corner stone of the quality system.’ Aligning academic and quality cultures: ’Monitoring of the quality...should be framed as a scholarly approach to professional learning and development, not as a control system...internal evaluations should again be framed as institutional research, i.e. efforts to understand our university and learn to develop it.’ Insights from Sweden [Eva Åkesson V-C, Uppsala University, HEIR Conference, 2012] 9

Consultation about evaluation processes and their effectiveness Interviews – Representatives of 8 Scottish HE Institutions Survey questionnaire – Completed by 30 respondents from 13 Scottish HE Institutions Groups consulted: – ELIR Reviewers – Scottish HE Enhancement Committee – Universities Scotland Teaching Quality Forum – sparqs (student participation in quality scotland) – QAA Scotland 10

Annual Monitoring AM ‘routine’ but taking on greater significance: e.g., ‘A little more conversation’ – A monitoring group (varying in composition) meets the subject team to discuss an annual report (varying in size) – Timing and nature of meetings vary but feedback is certain and rapid Data sets: improved quality and reliability Positive developments in student feedback: e.g., – Internal online surveys – Strategic approach to external surveys, e.g., NSS – Better loop closing: ‘You said, we did’ 11

Periodic Review – staff engagement Engage staff by demonstrating the academic value of review – e.g., staff contribute more to review agenda; better linkage with PSRB accreditations; develop skills of reflection; review panel membership as professional development; etc. Use annual monitoring reports more as the QA base to reduce new paperwork and allow the review to concentrate on evaluation and enhancement Make better use of externals’ subject/theme expertise in the enhancement element of the review 12

Student engagement Quoted as the most positive and effective feature of quality processes; for example: – Students’ enthusiastic involvement in reviews – Valued role of sparqs in training student reps – More effective Staff Student Liaison Committees, – etc., etc. Challenges? – Low survey response rates – How to engage students from harder-to reach categories: e.g., DLs Several respondents would like to learn more from other institutions’ approaches to student engagement 13

Preparing for ELIR 2013 cf 2008 (Good Practice in Reflective Analysis) More positive view of opportunity for institutional reflection – ‘The ELIR provided an opportunity to step back and reflect on the many activities and processes that underpin quality enhancement thereby ensuring a holistic review of our approach’ Characteristics of a ‘good’ RA: 2013 view very similar to 2008 – Use for updating commentary on Good Practice in Reflective Analysis Average length of RA ~doubled between ELIR 1 and ELIR 2 – Advance Information Set in ELIR 3 may reduce RA length again Similar preparation approaches across HEIs: – ELIR steering group with one or a few authors and gathering evidence and opinion from working groups Desire for greater sharing of experience of ELIR preparation 14

Initial conclusions Stability and change – Evolving but relatively stable QEF gives HEIs reassurance to develop processes that suit their requirements Vigour and rigour – Dynamic sector with ‘continual improvement process’ – Monitoring and review rigorous and meeting SFC guidelines Assurance and enhancement appear to be well-balanced – Respondents: institution-led evaluation leads to enhancement Effectiveness of self-evaluation processes? 15

Monitoring and review are more likely to be effective when... TRUSTED  Giving confidence to and developing trust of external and internal stakeholders: balancing assurance and enhancement  Answering the ‘so what?’ question  Based on teamwork and well supported by senior managers  Involving open discussions between reviewers and reviewed  Engaging a wide spectrum of students as reviewers and providers of evidence, and fully utilising sparqs expertise and support 16

Monitoring and review are more likely to be effective when... ALIGNING QUALITY AND ACADEMIC CULTURES  Flexible enough to recognise differences between subjects in culture and practice  Framed as a scholarly approach to professional learning and development and/or institutional research KEPT SIMPLE AND FOLLOWED THROUGH  ‘What’s working? What’s not? What needs to change?’  Leading to enhancement plans with SMART targets and closing loops  Monitoring, review and ELIR preparation are linked, and are themselves regularly reviewed 17

What’s next? Updating ‘Good practice in Reflective Analysis’ Project report Dissemination workshop 18

19