1. Utilising Module Evaluation data to explore outcomes from the Teaching Excellence and Student Outcomes Framework (TEF) Subject Level Pilot.
Natalie Holland
Teaching and Learning Academy
Liverpool John Moores University

2. The problem
TEF subject level pilot provides a lot of information at the subject level and across different groups of students.
It can be used to pinpoint where further exploration is needed to understand why particular groups of students studying a particular subject area are reporting a different university experience compared with their peers.
5 TEF metrics focus on NSS satisfaction.
NSS current datasets do not allow for the type of manipulation needed to explore this further.
My question – Can Module Evaluation at LJMU offer an alternative source of data?

3. What is the TEF subject level pilot?
35 subject groups (LJMU has programmes within 26 subject groups)
Each subject group given an initial rating based on its performance across a set of 9 metrics
Institutions write a 5 page narrative providing a context, highlighting successful features of the course as well as any mitigating factors.
Holistic assessment of the subject group by an independent panel

4. What is the TEF subject level pilot?
Aspect of Quality
Metric
Metric type
Flag Value
Source
Teaching quality
The teaching on my course
NSS-based
0.5
NSS Q1-4
Assessment and feedback
NSS Q8-11
Student voice
NSS Q23-25
Learning environment
Academic support
NSS Q12-14
Learning resources
NSS Q18-20
Continuation 2
HESA and ILR data
Student outcomes and learning gain
Highly skilled employment or higher study*
Employment 1
DLHE declared activity 6 months after graduation
Sustained employment or further study
LEO 3 years after qualification
Above median earnings threshold or higher study

5. The NSS based metrics and Module Evaluation
It is not currently possible to us NSS data to explore the differences in experience across groups at the subject group level.
Our own Module Evaluation survey provides an alternative source of information on how our students are experiencing their studies.
Q1_Staff are good at explaining things
Q2_The module has challenged me to achieve my best work
Q3_I have received helpful comments on my work
Q4_Feedback on my work has been timely
Q5_I have actively/fully engaged with this module
Q6_Overall, I am satisfied with the quality of this module
Q7_Please comment on the most interesting aspect of this module
Q8_Please comment on how this module could be improved
Ideal way to look into the differences in experience across groups at the subject level would be to explore the NSS survey data and comments.
The way that NSS data is provided to institutions currently, doesn’t allow for this level of interrogation – there does seem to be some changes being made to this for this academic year so hopefully in the future this will be possible.

6. Methodology
TEF looks at an aggregate of 3 years of NSS data – 2015/16, 2016/17 and 2017/18
Combined ME datasets from 2015/16, 2016/17 and 2017/18
Identified areas where split metrics indicated differentiated experiences across all subject groups.
Explored the ME data set for closed questions
Thematic analysis of open ME comments to look at the stories behind identified differences between groups.

7. Limitations Only have data on Age, Ethnic group, Origin, Gender
Ethnicity is only split by BAME, White, Not known, Refused
Questions in ME do not directly align / cover all NSS based aspects in the metrics – comments could provide more informative information.

8. Example – A humanities subject group
This subject area performs well in NSS based metrics overall with double positive flags
There is variation when split across different groups of students
Mature and BAME students no flag compared with double positive flags
Metric
White
BAME
Young
Mature
The teaching on my course
No Flag
Assessment and feedback ++
Student voice
Academic support +
Learning resources
 No Flag

9. Module Evaluation Closed Questions
BAME Students have lower satisfaction scores than any other group in this subject group across all 6 questions. Particularly Q2 and Q5.
Mature students actually have highest satisfaction levels across all questions.
All Subject group
BAME
White
Mature
Young
Q1_Staff are good at explaining things
84%
79%
85%
Q2_The module has challenged me to achieve my best work
73%
62%
75%
72%
Q3_I have received helpful comments on my work
80%
71%
82%
Q4_Feedback on my work has been timely
88%
86%
Q5_I have actively/fully engaged with this module
76%
69%
Q6_Overall, I am satisfied with the quality of this module
81%
BAME = 69
White = 1451
Mature = 267
Young = 1253

10. Comments analysis Thematic analysis of comments
I am looking at instances of comments – this means that every comment a student makes will be considered separately. Numbers of comments is not number of individual students.

11. Most interesting aspect of this module
BAME
32 comments
(46% of
responses)
White
824 comments
(57% of
Mature
65 comments
(62% of
Young
691 comments
(55% of
When looking at comments left to the question What was the most interesting aspect of the module, similar themes were identified for all groups of students:
Interest in topics
The level of challenge
HOWEVER – BAME STUDENTS MENTION CHALLENGE IN A NEGATIVE WAY – “The module was certainly challenging as I had no idea what I was doing for the XXX essay”.

12. Most interesting aspect of this module
White students saw the level of challenge as motivating:
“I like the challenge that this module has given in terms of pushing me to achieve”
“The challenge of finding your own sources for assignments, both interesting and challenging, very worthwhile!”
Some BAME students viewed challenge negatively :
“The module was certainly challenging as I had no idea what I was doing for the XXX essay”.
“The challenging aspect would be understanding the assignments and completing them on time”.

13. How could this module be improved?
Some consistent messages across all groups:
More teaching time
Some differences in opinion as to how this should look. BAME students would like more seminars rather than Lectures. Other groups were more mixed in opinion
Support and greater guidance with assessments and exam preparation.
Some consistent messages across all groups:
More teaching time
Some differences in opinion as to how this should look. BAME students would like more seminars rather than Lectures – other groups were more mixed in opinion
Support and greater guidance with assessments and exam prep

14. BAME – How could this module be improved?
More resources / support tools such as videos and online resources
Teaching approach/ format (seminars preferred to lectures):
“I struggled to learn through the methods [NAME REMOVED] wanted us to learn through. He is a good lecturer but his methods are not for all”.
More support:
“Reading was a bit overwhelming”
“Hard to follow at times”

15. Mature Students – How could this module be improved?
Main points were the same but there were some points more specific to mature students:
Some found lecturers could be patronising and didn’t take into account the life experience that mature students have.
Better co-ordination of deadlines:
“More co-ordination amongst the lecturers on deadlines across the subject would have been helpful, as for much of the year there have been no deadlines and all of the coursework assignments have been scheduled for the same days in a two-three week period which has caused stress and anxiety”.

16. Next steps . . .
Focus groups with BAME / mature students to understand how student experience can be improved.
Feedback to Programme teams – develop action plans to ensure students from all backgrounds are supported
Follow-up findings in NSS results in July