1. Evidence for Enhancement: Making sense of surveys
Dr Alex Buckley
University of Strathclyde
21 May 2019

2. Introduction The NSS inspires polarized views Reality is boring
‘The NSS tells us everything we need to know about learning and teaching…’
‘The NSS can’t tell us anything interesting at all…’
Reality is boring
It’s a great idea to ask students about their experiences
NSS is just one way of doing that
Like any way of doing that, it has strengths and weaknesses
The weaknesses need to be kept in mind, but it is still very useful
Can we take the statistical limitations of the NSS into account and nevertheless make use of the data? YES!
By using confidence intervals
Practical guidance / technical information is in the Factsheet
(Qualitative data is key)

3. Confidence intervals
Confidence intervals are publicly available from OfS website
Represent the range in which you can be 95% confident that the ‘real’ % agreement score falls: the ‘margin of error’

4. Confidence intervals
Confidence intervals are publicly available from OfS website
Represent the range in which you can be 95% confident that the ‘real’ % agreement score falls: the ‘margin of error’
On average, confidence intervals are 31%
Average responses per institution at JACS3 subject level (106 subjects) is 58

5. Confidence intervals & statistical significance
How do we make sense of comparisons of NSS results?
One answer: see whether the comparison is statistically significant
A difference between two scores is statistically significant if we can be confident that it represents a genuine difference in students’ perceptions and isn’t due to chance
(This is a simplification…)

6. Confidence intervals & statistical significance
Caveats…
Statistical significance is just one way of interpreting the data
Significance testing should be used alongside other information
Binary significant/not significant distinction is just information that can support interpretation
It’s not all that matters

7. Comparing NSS results Different ways you can compare your NSS results:
Comparing your results from this year to your results from last year
Comparing your results to the sector average (for your subject)

8. 1. Comparing results from different years

9. 1. Comparing results from different years
On average, 2% of year-to-year comparisons are statistically significant
NSS scores are stable, reliable, but that means it takes a lot to get a significant change
There are big differences between questions and subjects
Nursing, course organisation (Q15): 26%
E.g. Italian studies: 0% for all questions
Communication skills (Q20): 0.7%
But statistical significance is just one ingredient…

10. 1. Comparing results from different years
On average, it’s unlikely that a score for a particular question will be statistically significantly different from last year
(So the confidence intervals are really important to help find out what changes are meaningful)
Getting data on the impact of enhancement is key. So what can you do?
NSS might be helpful if you have a large cohort, and you’re interested in timeliness of feedback, course organisation
Maximise your response rate
Use your own questionnaire
More specific response options, e.g. NSS questions with a 10-point scale
More specific questions, e.g. ‘What proportion of your feedback was provided in three weeks?’

11. 2. Comparing results to the sector average
Compare against the aggregate score for your subject
Sector average aren’t publicly available, they are provided with internal institutional data (just ask your surveys team)
The sector averages consist of so many responses that the confidence intervals can be treated as zero
Allows creation of an institutional ‘heatmap’

12. 2. Comparing results to the sector average

13. Conclusion
Statistical significance testing is a good way of cutting through the noise
But it’s just one element of interpretation, it shouldn’t be used in isolation
The NSS is a high-level quantitative census survey, and has predictable strengths and weaknesses
If you teach Nursing and you’re interested in the organisation of the course (Q15), your NSS results can tell you quite a lot
If you teach Archaeology and you’re interested in how the course has supported the synthesis of ideas (Q6), your NSS results will tell you much less
For smaller cohorts, qualitative data will be particularly appropriate

14. Further info
(2018) ‘How to use the NSS (responsibly)’, Annual Enhancement Themes Conference, June 2018, Glasgow. Available at: use-the-national-student-survey---responsibly-(paper).pdf
(2018) ‘What proportion of NSS comparisons are statistically significant?’, Society for Research in Higher Education Annual Conference, December 2018, Newport. Available at:
(2018) ‘How much are your NSS results really telling you?’, WonkHE, March Available at:
(2012) Making it count: Reflecting on the NSS in the process of enhancement (York, Higher Education Academy). Available at: