1. Mining NSS data for enhancement An example from Art & Design Mantz Yorke mantzyorke@mantzyorke.plus.com Higher Education Academy Conference “Higher Education Surveys for Enhancement” Nottingham, 18 May 2010

2. This presentation illustrates one way in which NSS data have been used in the interests of enhancement It ought to be adaptable to other situations

3. Art & Design makes a substantial contribution to the national economy. It is a national success story. Yet when it is viewed through the lens of the NSS, it does not come out well in comparison with other subject areas (save in respect of Assessment & Feedback). Question 1: Why? The paradox of Art & Design

4. Judged by NSS scores, some institutions do much better than others. Question 2: Why? Variation in the provision of Art & Design

5. I am critical of the NSS, on various conceptual and technical grounds. It is problematic as a basis for informing the public. It is however more useful in its ancillary role as a prompt to enhancing the student experience. Even if it’s a gross ‘measure’, we can still get messages from it. Institutions are ill-advised to ‘chase the ratings’: rather, they need to look underneath the scores to identify points at which enhancement-oriented activity might most profitably be targeted. Clearing some (substantial) brushwood

6. The NSS is part of a political agenda: it is here to stay (for the time being, at least – but the focus of interest is beginning to switch to student engagement, following the use of NSSE in the US and AUSSE in Australia). So we have to live with it, even if we don’t like it. Politics

7. Art & Design and the NSS

8. Teaching Asst & Fdback Acad Sup Org & Man Learn Res Pers Dev OvS Selected subject comparisons, NSS 2009 (New universities and colleges in England and Wales) Timetable Changes Organisation Explaining Interesting Enthusiastic Stimulating Prompt Detailed Clarified Specialist equipment NB Means are unweighted

9. Teaching Asst & Fdback Acad Sup Org & Man Learn Res Pers Dev OvS Selected subject comparisons, NSS 2009 (New universities and colleges in England and Wales) Timetable Changes Organisation Explaining Interesting Enthusiastic Stimulating Prompt Detailed Clarified Specialist equipment NB Means are unweighted

10. GLAD, the HE Academy ADM Subject Centre and the HEAD Trust combined in supporting a study of the reasons underlying the relatively poor performance of Art & Design since the inception of the NSS. The study had quantitative and qualitative aspects: only the former is represented in this presentation. (See the HEA-ADM Subject Centre website for the report of the study.) The NSS and Art & Design – a case study

11. Disaggregating to institutions with studio-based provision Design Studies is used as an illustration: similar results are found for Fine Art and Cine & Photo Studies

12. Specialist A&D institution Some institutions do (consistently) better than others No ‘specialist institution effect’ No particular size effect

13. Digging deeper into the NSS data Three examples

14. Prompt Detailed Clarified Timetable Changes Organisation Explaining Interesting Enthusiastic Stimulating Specialist equipment Design Studies, selected comparisons (NSS 2009) Teaching Asst & Fdback Acad Sup Org & Man Learn Res Pers Dev OvSNovel problems Clear criteria

15. Fine Art: selected comparisons (NSS 2009) Teaching Asst & Fdback Acad Sup Org & Man Learn Res Pers Dev OvS Prompt Detailed Clarified Timetable Changes Organisation Explaining Interesting Enthusiastic Stimulating Specialist equipment Novel problems

16. English: selected comparisons (NSS 2009) Teaching Asst & Fdback Acad Sup Org & Man Learn Res Pers Dev OvS Prompt Detailed Clarified Timetable Changes Organisation Explaining Interesting Enthusiastic Stimulating Specialist equipment Novel problems

17. Some possible determinants of the NSS outcomes Specific institutional and/or departmental causes (e.g. relocation, staffing difficulties). Programme leadership and/or management. Pedagogic approach in Art & Design. Curricular positioning of Art & Design in (some) multi-disciplinary institutions. Balance of FT and PT staff. Student expectations v. experiences: ‘The Deal’. A&D students having to be more self-organising than many others as regards booking spaces and resources; self-organising may cut across formal timetabling.

18. Some methodological points The illustrations are based on raw data. ‘Residuals’ make allowances for various institutional factors, such as entry profile, but their interpretation and presentation to others require statistical sophistication. The raw data may be rougher than one would like, because they are affected by local considerations and the limits of the NSS as an instrument. One year’s NSS scores may therefore be unreliable. Nevertheless, rough and ready data may be ‘good enough’ to act as prompts to enhancement. Warning Shaping the data into a usable form is a bit of a slog.

19. In conclusion NSS data show relative strengths and weaknesses. The ‘quality literature’ asserts that there’s almost always something that can be improved, by doing things better or doing things differently. The important thing is to identify those aspects of provision where enhancement activity is most likely to pay off, and to focus on them. There are unlikely to be many quick fixes.