1. HEFCE Learning Gain PILOT PROJECT
University of Portsmouth
Dr Emily Mason-Apps
@UoPLearningGain

2. Some Context HEFCE Definition of Learning Gain:
“…the improvement in knowledge, skills, work-readiness and personal development made by students during their time spent in higher education”

3. Some context
“Work across the sector to develop new measures of learning gain is in progress14. Until new measures become available and are robust and applicable for all types of providers and students, we anticipate providers will refer to their own approaches to identifying and assessing students’ learning gain - this aspect is not prescriptive about what those measures might be.”
Section 4.9, Pg Department for Education (September, 2016). Teaching Excellence Framework: year two specification

5. The “Work across the Sector”
HEFCE funding 13 pilot projects involving over 70 universities and further education colleges
HEFCE now running their own pilot project: National Mixed Methodology Learning Gain Project (NMMLGP)
HEFCE also leading the Higher Education Learning Gain Analysis (HELGA) project

6. Our Project Team Other Portsmouth Team Members:
Professor Sherria Hoskins (Project Lead)
Dr Valerie Anderson
Professor Andy Thorpe
Dr Jessica Gagnon
University of Southampton (PL: Dr Michael Tomlinson)
Royal Holloway, University of London (PL: Professor Arnaud Chevalier)
University of the Arts, London (PL: Richard Sant)
Due to a bit of necessary “team shuffling”, UAL now a data collection institution

7. Longitudinal Psychometric Questionnaire
Project Overview
Mixed Methods
Qualitative
HE staff focus groups
Student focus groups
Employer interviews
Parent postcards
Quantitative
Longitudinal Psychometric Questionnaire

8. Qualitative Strand: AIms
Stakeholder views of Learning Gain
HE Staff
Parents
Students
Graduate Employers
Qualitative Strand: AIms
Aims:
To gain an understanding of stakeholders’ perceptions of learning gain and “graduateness”
To inform the psychometric tool development
Explore the similarities and differences between stakeholders’ perceptions
Seek stakeholder views on the measures produced

9. Qualitative Strand: Employer Interviews
Aiming for ~16 employer interviews (Complete so far: 6 Ports; 4 Soton)
A mixture of size and industry
Graduate Capital Scale
How would you describe a top graduate?
What do you think employers look for when they employ a graduate?
What are the key things you look for in a graduate’s application?
How important is the adaptability and resilience of a graduate?
Are you more inclined to recruit graduates from specific types of university?

10. Qualitative Strand: Student Focus Groups
Each institution running at least one focus group with Level 4, Level 5 and Level 6
Complete at Portsmouth and RH
Possibility of longitudinal follow up with Level 4s
What benefits do you hope to gain from your time at University?
How would you describe someone who has made the most of their time at University?
What benefits do you think your parents hope you will gain during your time at University?
What do you think employers look for when they employ a graduate?
Thinking about these employer expectations – which of them present most of a challenge for students like yourselves in a University setting?
… talk to us about who you were when you started your degree and who you are now.
… how you expect to see yourself when you finish your degree.

11. Qualitative Strand: Parent Postcards
“What benefits do you hope your child will gain from their time at university?”
67 Collected (currently all Portsmouth- current or prospective)
Aiming for 100
Hoping other institutions will also distribute…!

12. Parent PostCard Word Cloud…

13. Quantitative Strand
Developing psychometric tests to explore the development of undergraduates’ learning gain (non-cognitive skills)
Longitudinal
New LG Tools:
Graduate Capital (e.g. social, psychological, cultural, identity, human)
Learner Resilience Scale
Self-theories
False uniqueness/consensus

14. Quantitative Strand
Alongside Existing Measures
Implicit Theories (Dweck, 1999)
UKES
ASSIST (Tait, Entwistle & McCune, 1998)
Alongside Existing (or to-exist) Data
UCAS
KIS data
Transcript Data
First Destination Data
Exploring the development of these factors and relationships between them
We will also explore the development of these factors in relation to demographic factors

15. Quantitative Progress
Various challenges have led to a slight change of plan…
Now two strands: 1) Portsmouth Only Cohort and 2) Full Team Cohort
Portsmouth Only Cohort
April-Sep 2016
199 Year 1
94 Year 2
125 Year 3
April-Sep 2016
199 Year 1
April-June 2017
Retest Year 2
April-June 2018
Retest Year 3
July 2018
Gather Final Degree Outcome Data
Gather First Destination Data
January 2019?

16. Quantitative Progress
Full Team Cohort
(Data from Portsmouth, Southampton, UAL, RH)
October 2016-Jan 2017
1,116 Year 1
(341 Ports)
September-October 2017
Retest Early Year 2
April-June 2018
Retest End Year 2
July 2018
Gather Transcript Data for Year 1 and 2
Gather Final Degree Outcome and First Destination Data
Post Funding!

17. Quantitative Strand: Some Initial Findings
Full Team Cohort (Portsmouth students)
Group Variable
Level
Frequency
Percent
Gender
Male
161
41%
Female
232
59%
Age Group
Young
321
82%
Mature 72
18%
Ethnicity
White
272
69%
BME 76
19%
Not Known/Refused 45
12%
First Generation Status
None
221
56%
One Parent 88
22%
Both Parents 71

18. Quantitative Strand: Some Initial Findings
Initial Group Differences*: Ethnicity
No significant group differences for: ASSIST, Self-Efficacy, Mindset, Graduate Capital
Significant differences for:
Self-Esteem
Learner Resilience
People like me…
*Full Team Cohort (just Portsmouth students)

19. Quantitative Strand: Some Initial Findings
Initial Group Differences*: Age
No significant group differences for: Mindset
Significant differences for:
Self-Esteem
Self-Efficacy
Learner Resilience
Graduate Capital
All ASSIST Scales
*Full Team Cohort (just Portsmouth students)

20. Quantitative Strand: Some Initial Findings
Initial Group Differences*: First Generation Status
No significant group differences for: Mindset, ASSIST, Learner Resilience, Graduate Capital (approaching)
Significant differences for:
Self-Efficacy
Self-Esteem
People like me
*Full Team Cohort (just Portsmouth students)

21. Quantitative Strand: Some Initial Findings
Portsmouth Only Cohort: Some Cross-Sectional Analysis
A note of caution: Relatively small group sizes + Cross-sectional (so don’t get too excited just yet!)
Some interesting (and mostly positive) results though
Will be looking to confirm if these “changes” come out in the longitudinal data

22. Quantitative Strand: Some Initial Findings
Portsmouth Only Cohort: Some Cross-Sectional Analysis
No significant changes: Mindset; ASSIST – Deep; Self-Esteem
Significant changes:
Self-Efficacy
Learner Resilience
Graduate Capital
ASSIST – Strategic
ASSIST – Surface
Remember: Relatively small group sizes + Cross-sectional (so don’t get too excited just yet!)

23. Challenges: Our Project
Multi-institutional projects = multiple challenges!
Contracts and Staffing
Differing timelines and protocols
Data sharing
Student Engagement
Maintaining engagement
Weight of expectation…

24. Back to the Context…
“Work across the sector to develop new measures of learning gain is in progress14. Until new measures become available and are robust and applicable for all types of providers and students, we anticipate providers will refer to their own approaches to identifying and assessing students’ learning gain - this aspect is not prescriptive about what those measures might be.”
Section 4.9, Pg Department for Education (September, 2016). Teaching Excellence Framework: year two specification

25. Back to the Context…
“Work across the sector to develop new measures of learning gain is in progress14. Until new measures become available and are robust and applicable for all types of providers and students, we anticipate providers will refer to their own approaches to identifying and assessing students’ learning gain - this aspect is not prescriptive about what those measures might be.”
Section 4.9, Pg Department for Education (September, 2016). Teaching Excellence Framework: year two specification

26. Back to the Context…
“Work across the sector to develop new measures of learning gain is in progress14. Until new measures become available and are robust and applicable for all types of providers and students, we anticipate providers will refer to their own approaches to identifying and assessing students’ learning gain - this aspect is not prescriptive about what those measures might be.”
Section 4.9, Pg Department for Education (September, 2016). Teaching Excellence Framework: year two specification

27. The future…
Complete Qualitative Data Collection (pushing for September 2017)
Currently collecting Ports Only Time 2 Data (50% response so far)
Ports Only Time 3 (May 2018)
Collect Full Team Cohort Time 2 (September 2017) and Time 3 (May 2018)
Gather student records data etc.
Generate individual student reports
Stakeholder Conference January 2019

28. Any questions before we move on to the discussion?
Thank you!
Any questions before we move on to the discussion?
@UoPLearningGain

29. Three parts (if we have time!)
Discussion
Three parts (if we have time!)
Tell us what you think as stakeholders
Help us in the development of the project
Improving learning gain- non-changing mindsets