Developing a revised approach to considering subject level NSS data – a Durham example Richard Harrison Head of the Academic Support Office 15 April 2011

∂ Outline  Durham’s NSS results and aspirations  Consideration of results  Review of the existing approach  Revised approach 2010 onwards  Impact of revised approach

∂ Durham’s NSS results, Teaching Assessment and Feedback Academic Support Organisation and Management Learning Resources86 87 Personal Development Overall Satisfaction Response rate

∂ Durham’s NSS aspirations ‘To be in the top 5 UK multi-subject universities for student satisfaction, undergraduate qualifications on entry and undergraduate completion rates’ University Strategy, Education key targets:

∂ Durham’s comparative NSS performance, 2009 and All HEIs Excluding specialist institutions All HEIs Excluding specialist institutions Question 22 – Overall Satisfaction20 th 15 th 24 th 19 th Times Good University Guide23 rd 17 th 18 th 14 th Complete University Guide12 th 8 th 16 th 13 th

∂ Follow-up, : institutional  From NSS 2005: high-level summary of outcomes  From NSS 2007: Institution-level quantitative analysis of results for Questions 1 to 2  From NSS 2008: Institution-level qualitative analysis of answers to free text questions

∂ Follow-up, : subject level  Departmental action plans developed in Michaelmas Term o All questions with a mean below 3, or fewer than 70% agree o All questions where the mean fell by 0.5 or greater, or mean or % agreed fell for two consecutive years  Departmental action plans submitted to chairs of Faculty Education Committees for consideration  Faculty-level overviews of departmental results and action plans submitted to University Education Committee each January

∂ Why did we feel this wasn’t working?  Levelling off of results at institutional level  Lack of departmental engagement  Personal experience of academic staff that even when implemented well the University’s approach was not proving effective  Cumulative conclusions of institution-level statistical analyses of NSS 2007 to 2009

∂ Conclusion of institution-level statistical analyses NSS 2007: ‘Analysis of variability between departments suggests that there is a single underlying factor (effectively “overall experience”); that factor links the different subgroups of questions which otherwise vary largely independently from each other’ Report on analysis of NSS data, January 2008

∂ Conclusion of institution-level statistical analyses NSS 2008: ‘One interesting feature [of the comparator analysis] is that the shape of the profile for an institution generally changes relatively little but moves up and down from year to year. This corresponds to the observation, made last year and earlier in this report, that for Durham, the main difference between departments and across years is that scores on questions all tend to go up and down together, suggesting that effectively one overall factor (quality of experience) is being measured.’ Report on analysis of NSS data, January 2009

∂ Conclusion of institution-level statistical analyses NSS 2009: ‘the shape of the profile for many institutions generally changes relatively little but moves up and down from year to year … the main difference between departments and across years is that scores on questions all tend to go up and down together, suggesting that effectively one overall factor (quality of experience) is being measured. The difference in profile between institutions shows that there do seem to be real differences between institutions in aspects of perceived experience.’ Report on analysis of NSS data, January 2009

∂ Developing a revised approach ‘there comes a point where modernity begins to parody itself, pursuing answers without any sense of the original questions, proliferating devices for achieving ever greater “efficiency” in education as in other spheres’ N Blake, P Smeyers, R Smith and P Standish, Thinking Again: Education After Postmodernism (London, 1998), p.1

∂ Developing a revised approach ‘incremental improvements on specific NSS questions do not necessarily affect overall satisfaction, while improvements in overall satisfaction correlate with improvements across all other questions. It is therefore clearly a more effective use of time and energy to focus on improving overall student satisfaction as such, rather than working to raise scores on other individual NSS questions.’ Faculty Overview Report of NSS 2009

∂ Developing a revised approach Faculty NSS Manifesto:  Core principles underpinning all NSS-related activities o Communicate better with students o Manage obstacles and flashpoints o Focus on staff-student contact and high NSS impact activities  Faculty commitments to support departments in meeting these principles  Departmental commitments to undertake certain activities

∂ The revised approach  Faculty away-day to discuss key themes from NSS results  Development and agreement of a Faculty Manifesto  Consideration of Faculty Manifesto for approval by University Education Committee  Ongoing discussions between faculties and departments throughout the year  Reflective, backward looking commentary in the annual review report

∂ Impact of new approach NSS results 2009 and 2010: National ranking Qu. 22 Increase in score 2009 to 2010 National ranking 2009 National ranking 2010 Astrology+5%38/8814/84 Campanology+15%68/8513/86 Widgetology+6%44/5926/51

∂ Impact of new approach  Increased engagement from academic departments o Quality of discussion and engagement with issues o Genuinely strategic faculty level strategies o Ground-up staff development initiatives

∂ Questions ?