1. Welcome to today’s teaching and learning conversation with Prof
Welcome to today’s teaching and learning conversation with Prof. Mark Langan
Title: The S&M of HE: Surveys and Metrics
If you can hear the presenter speaking please change your status to agree
While we are waiting to start please take the opportunity to run through the Audio Setup Wizard (one last time )
Click “Meeting” (top left)
Select the Audio Setup Wizard option
Follow on-screen instructions
Let us know if you are having a problem by typing “Help” into the Chat box and we’ll see what we can do
Important Note
At this point you may be able to hear and see Rod and Calum in the video pod but not any of the other participants or yourself.
We have done this on purpose .
Please bear with us while we get organised and take the time to run the audio set up wizard again.
Tuesday 29th September 2014

2. We are about to start recording the webinar.
Please note that we make TLC recordings publicly available at:

3. The S&M of HE: Surveys and Metrics
Mark Langan
MMU
24th May 2016

4. Something of me… Teacher (compulsively)
Biologist (ecology and behaviour)
HE researcher:
L&T design and learner empowerment
Student surveys (NSS, UKES)
Learner engagement/disengagement (SoTL)
Learning gains (RAND)
Benchmarking HE (NHS)
Playful learning (July 13th-15th 2016)
About you – where are you based?

5. Pre-course Post-course In-course Expectations NSS Experience
KIS
A level tariff
Pre-entry experiences
Recruitment
Media
Reputation
Expectations
NSS
Engagement
Retention
Progression
Attainment
Good honours
Experience
‘3P’ model (Biggs, 1993), approaches education as a complex system with ‘Presage’, ‘Process’ and ‘Product’ variables interacting with each other. The ‘3P’ model is essentially the same as that used by large-scale studies in the US (e.g. Astin, 1977, 1993): the ‘Input-Environment-Output’ model.
Employment
Alumni
Collaboration
Capacity
Reputation
Consolidation

6. When you hear the words “National Student Survey” (or “NSS”) – what is your reaction?

7. Aims of the NSS (original)
to inform the choices of future students, alongside other sources of information about teaching quality
2. to contribute to public accountability by supporting external audits of institutions by the QAA.

8. What am I trying to achieve in HE?
Students
Learning goals (achievement, progression, motivation)
Satisfaction with experience (wants and needs)
Future success (employment, further education)
Staff and Institution
Quality assurance/enhancement
Staff satisfaction and well-being (motivation)
Productivity and success
Reputation/league tables
Financial security (resources and reputation)
Am I missing anything?

9. So what does the NSS ‘measure’?

10. NA

11. Source: Hewson (2011) www.mathstore.ac.uk/headocs/Hewson.pdf
… attempt to measure quality across disciplines ... you find that some disciplines emerge consistently better than others, across different studies and different
institutions. Either one has to accept that certain subjects are always taught less well than others, which seems highly unlikely, or that different measures of quality are better aligned with the consequences of some (disciplinary) pedagogic practices than with others… Comparing quality between disciplines is fraught with
difficulties. Gibbs, 2010, p.46 (this text is derived from Mantz Yorke

12. 2007
Langan, A.M., P.J. Dunleavy and A.F. Fielding (2013). Applying Models to National Surveys of Undergraduate Science Students: What Affects Ratings of Satisfaction? Education Sciences, 3, ; doi: /educsci
Source: Langan et al (2013)

13. NSS ratings: Pakistani students compared to White British students Aftab Dean (Leeds Met Uni, 2010)
Business subjects
Teaching Assessment & Academic Org &Man Learning Personal
Feedback Support Resources Development
Note Pakistani female students - lower NSS scores when living at home

14. Jacqueline H. S. Cheng & Herbert W
Jacqueline H.S. Cheng & Herbert W. Marsh (2010) National Student Survey:
are differences between universities and courses reliable and meaningful?, Oxford Review of
Education, 36:6, , DOI: /

15. Which of the NSS questions (Q1-Q21) best predict the final question (Q22) “Overall I am satisfied with my course”?
Originally asked to improve feedback to improve NSS Q22 (Alan Fielding)

16. NA

17. Effectiveness of Q1-21 to predict overall satisfaction (Q22)
Predicting questionnaire item
Inc MSE (%)
Q15 - The course is well organised and is running smoothly
119.89
Q1 - Staff are good at explaining things
71.45
Q4 - The course is intellectually stimulating
66.71
Q14 - Any changes in the course or teaching have been communicated effectively
60.79
Q10 - I have received sufficient advice and support with my studies
55.34
Q11 - I have been able to contact staff when I needed to
43.40
Q3 - Staff are enthusiastic about what they are teaching
40.08
Q2 - Staff have made the subject interesting
38.26
Q12 - Good advice was available when I needed to make study choices
35.27
Subject
32.35
Q6 - Assessment arrangements and marking have been fair
20.10
Q17 - I have been able to access general IT resources when I needed to
18.73
Q19 - The course has helped me present myself with confidence
17.35
Q18 - I have been able to access specialised equipment, facilities or room when I
15.41
Q16 - The library resources and services are good enough for my needs
15.34
Q20 - My communication skills have improved
13.29
Q13 - The timetable works efficiently as far as my activities are concerned
13.16
Q7 - Feedback on my work has been prompt
10.49
Q9 - Feedback on my work has helped me clarify things I did not understand
6.65
Q5 - The criteria used in marking have been clear in advance
6.60
Q21 - As a result of the course, I feel confident in tackling unfamiliar problems
3.32
Q8 - I have received detailed comments on my work
3.04
Source: Langan et al (2013)
Science subjects. MSE% when higher is a better predictor of Q22. All years are combined for analysis. Includes subject grouping. Point out lack of influence of feedback Qs on explaining residual variation (despite this being rated lower than many Qs).
Subject is in there to be accounted for. A lot of variability within subjects (rem Biology slide of Alan’s) still a better predictor than most!

18. Source: Fielding et al (2010)
Some questions are more related to each other (e.g. Teaching/Support) than others (e.g. Resources). Assessment and Feedback split for analyses
Fielding, A.F., P.J. Dunleavy and A.M. Langan (2010) Effective use of the UK’s National Student (Satisfaction) Survey (NSS) data in science and engineering subjects. Journal of Further and Higher Education, 34,
Source: Fielding et al (2010)

19. NSS Feedback Qs Feedback Question Subject prompt detailed helpful
Biological Sciences
Physical Sciences ✔
Physical Geography
Mathematical Sciences
Computer Sciences
Mechanically-based Engineering
Electrical and Electronic Engineering
Technology
Human Geography
Correlations (r) and p values (p) for the correlation between the level of agreement with AS AN EXAMPLE the three feedback questions (Q7 - Q9) and overall satisfaction Q22.

20. Predicting Q22 from Q1-Q21: winners and losers?

21. At MMU we run an internal survey (ISS) based on the ‘best predictors’ from the NSS survey from each dimension.

22. 1 Staff on my course are good at explaining things 2 Feedback on my work helped me to clarify things I did not understand 3 I have received sufficient advice and support with my studies 4 The course is well organised and is running smoothly 5 University resources are appropriate to my learning needs 6 The course has helped me develop confidence and skills to succeed

23. MMU’s Internal Student Survey
CONFIDENCE
18%
ISS score for building confidence & skills
ORGANISATION
16%
ISS score for course organisation
EXPLANATION
14%
ISS score for explanation (teaching)
ADVICE
13%
ISS score for advice & support
RESOURCES
11%
ISS score for learning resources (library, IT, etc)
FEEDBACK
ISS score for assessment feedback
CLEARING 3%
Whether student entered via clearing
OCC
Year and mode of study
JACS
High level subject code
FACULTY 2%
Faculty

24. MMU’s Internal Student Survey
Predictive Accuracy
Factor ranked #1
Factor ranked #2
Factor ranked #3
MMU (73%)
CONFIDENCE (18%)
ORGANISATION (16%)
EXPLANATION (14%)
School of Art (65%)
ORGANISATION (17%)
ADVICE (15%)
HPSC (72%)
CONFIDENCE (19%)
ORGANISATION (18%)
Hollings (70%)
EXPLANATION (15%)
HLSS (72%)
CONFIDENCE (22%)
EXPLANATION (18%)
Business & Law (74%)
Sci & Eng (75%)
CONFIDENCE (17%)
EXPLANATION (17%)
Cheshire (73%)
CONFIDENCE (20%)
ORGANISATION (19%)

25. … why?
If you run surveys like this, do you respond more to the metrics or what was written in the comments?

26. Text comments - Ratios of: TEACH (staff) v ORG & MAN
Higher achiever in NSS
Frequency of comments
Langan, A.M, N. Scott, S.N. Partington, and A. Oczujda (2015) Coherence between text comments and the quantitative ratings in the UK's National Student Survey. Journal of Further and Higher Education, DOI: / X
Lower achiever in NSS
Langan et al (2015)

27. Faculty-level + TEACH - ORG & MAN - TEACH + ORG & MAN
Langan, A.M, N. Scott, S.N. Partington, and A. Oczujda (2015) Coherence between text comments and the quantitative ratings in the UK's National Student Survey. Journal of Further and Higher Education, DOI: / X
- TEACH
+ ORG & MAN
Below National Mean Above National Mean
Langan et al (2015)

29. Casting a shadow? "a handful of lecturers are fantastic"...
"varies from tutor to tutor"...
"a few lecturers make it difficult…"

30. Does fear of ‘slipping in the metrics’ stifle L&T innovation?

31. Final thoughts
Investigator: QI suggests crediting William Bruce Cameron instead of Albert Einstein. Cameron’s 1963 text “Informal Sociology: A Casual Introduction to Sociological Thinking” contained the following passage, Boldface has been added to excerpts [WCIS]:
It would be nice if all of the data which sociologists require could be enumerated because then we could run them through IBM machines and draw charts as the economists do. However, not everything that can be counted counts, and not everything that counts can be counted.
“Not everything that can be counted counts, and not everything that counts can be counted”
Not Albert Einstein

32. If there’s time…

33. “The best predictors of educational gain are measures of educational process…”

34. Source: University of Indiana http://nsse. indiana

36. ‘Cuddle factor’: Individuals and masses
Institution
Processes and people