HEFCE Learning Gain PILOT PROJECT University of Portsmouth Dr Emily Mason-Apps emily.mason-apps@port.ac.uk @UoPLearningGain

Some Context HEFCE Definition of Learning Gain: “…the improvement in knowledge, skills, work-readiness and personal development made by students during their time spent in higher education”

Some context “Work across the sector to develop new measures of learning gain is in progress14. Until new measures become available and are robust and applicable for all types of providers and students, we anticipate providers will refer to their own approaches to identifying and assessing students’ learning gain - this aspect is not prescriptive about what those measures might be.” Section 4.9, Pg. 20 - Department for Education (September, 2016). Teaching Excellence Framework: year two specification

The “Work across the Sector” HEFCE funding 13 pilot projects involving over 70 universities and further education colleges HEFCE now running their own pilot project: National Mixed Methodology Learning Gain Project (NMMLGP) HEFCE also leading the Higher Education Learning Gain Analysis (HELGA) project http://www.hefce.ac.uk/lt/lg/research/

Our Project Team Other Portsmouth Team Members: Professor Sherria Hoskins (Project Lead) Dr Valerie Anderson Professor Andy Thorpe Dr Jessica Gagnon University of Southampton (PL: Dr Michael Tomlinson) Royal Holloway, University of London (PL: Professor Arnaud Chevalier) University of the Arts, London (PL: Richard Sant) Due to a bit of necessary “team shuffling”, UAL now a data collection institution

Longitudinal Psychometric Questionnaire Project Overview Mixed Methods Qualitative HE staff focus groups Student focus groups Employer interviews Parent postcards Quantitative Longitudinal Psychometric Questionnaire

Qualitative Strand: AIms Stakeholder views of Learning Gain HE Staff Parents Students Graduate Employers Qualitative Strand: AIms Aims: To gain an understanding of stakeholders’ perceptions of learning gain and “graduateness” To inform the psychometric tool development Explore the similarities and differences between stakeholders’ perceptions Seek stakeholder views on the measures produced

Qualitative Strand: Employer Interviews Aiming for ~16 employer interviews (Complete so far: 6 Ports; 4 Soton) A mixture of size and industry Graduate Capital Scale How would you describe a top graduate? What do you think employers look for when they employ a graduate? What are the key things you look for in a graduate’s application? How important is the adaptability and resilience of a graduate? Are you more inclined to recruit graduates from specific types of university?

Qualitative Strand: Student Focus Groups Each institution running at least one focus group with Level 4, Level 5 and Level 6 Complete at Portsmouth and RH Possibility of longitudinal follow up with Level 4s What benefits do you hope to gain from your time at University? How would you describe someone who has made the most of their time at University? What benefits do you think your parents hope you will gain during your time at University? What do you think employers look for when they employ a graduate? Thinking about these employer expectations – which of them present most of a challenge for students like yourselves in a University setting? … talk to us about who you were when you started your degree and who you are now. … how you expect to see yourself when you finish your degree.

Qualitative Strand: Parent Postcards “What benefits do you hope your child will gain from their time at university?” 67 Collected (currently all Portsmouth- current or prospective) Aiming for 100 Hoping other institutions will also distribute…!

Parent PostCard Word Cloud…

Quantitative Strand Developing psychometric tests to explore the development of undergraduates’ learning gain (non-cognitive skills) Longitudinal New LG Tools: Graduate Capital (e.g. social, psychological, cultural, identity, human) Learner Resilience Scale Self-theories False uniqueness/consensus

Quantitative Strand Alongside Existing Measures Implicit Theories (Dweck, 1999) UKES ASSIST (Tait, Entwistle & McCune, 1998) Alongside Existing (or to-exist) Data UCAS KIS data Transcript Data First Destination Data Exploring the development of these factors and relationships between them We will also explore the development of these factors in relation to demographic factors

Quantitative Progress Various challenges have led to a slight change of plan… Now two strands: 1) Portsmouth Only Cohort and 2) Full Team Cohort Portsmouth Only Cohort April-Sep 2016 199 Year 1 94 Year 2 125 Year 3 April-Sep 2016 199 Year 1 April-June 2017 Retest Year 2 April-June 2018 Retest Year 3 July 2018 Gather Final Degree Outcome Data Gather First Destination Data January 2019?

Quantitative Progress Full Team Cohort (Data from Portsmouth, Southampton, UAL, RH) October 2016-Jan 2017 1,116 Year 1 (341 Ports) September-October 2017 Retest Early Year 2 April-June 2018 Retest End Year 2 July 2018 Gather Transcript Data for Year 1 and 2 Gather Final Degree Outcome and First Destination Data Post Funding!

Quantitative Strand: Some Initial Findings Full Team Cohort (Portsmouth students) Group Variable Level Frequency Percent Gender Male 161 41% Female 232 59% Age Group Young 321 82% Mature 72 18% Ethnicity White 272 69% BME 76 19% Not Known/Refused 45 12% First Generation Status None 221 56% One Parent 88 22% Both Parents 71

Quantitative Strand: Some Initial Findings Initial Group Differences*: Ethnicity No significant group differences for: ASSIST, Self-Efficacy, Mindset, Graduate Capital Significant differences for: Self-Esteem Learner Resilience People like me… *Full Team Cohort (just Portsmouth students)

Quantitative Strand: Some Initial Findings Initial Group Differences*: Age No significant group differences for: Mindset Significant differences for: Self-Esteem Self-Efficacy Learner Resilience Graduate Capital All ASSIST Scales *Full Team Cohort (just Portsmouth students)

Quantitative Strand: Some Initial Findings Initial Group Differences*: First Generation Status No significant group differences for: Mindset, ASSIST, Learner Resilience, Graduate Capital (approaching) Significant differences for: Self-Efficacy Self-Esteem People like me *Full Team Cohort (just Portsmouth students)

Quantitative Strand: Some Initial Findings Portsmouth Only Cohort: Some Cross-Sectional Analysis A note of caution: Relatively small group sizes + Cross-sectional (so don’t get too excited just yet!) Some interesting (and mostly positive) results though Will be looking to confirm if these “changes” come out in the longitudinal data

Quantitative Strand: Some Initial Findings Portsmouth Only Cohort: Some Cross-Sectional Analysis No significant changes: Mindset; ASSIST – Deep; Self-Esteem Significant changes: Self-Efficacy Learner Resilience Graduate Capital ASSIST – Strategic ASSIST – Surface Remember: Relatively small group sizes + Cross-sectional (so don’t get too excited just yet!)

Challenges: Our Project Multi-institutional projects = multiple challenges! Contracts and Staffing Differing timelines and protocols Data sharing Student Engagement Maintaining engagement Weight of expectation…

Back to the Context… “Work across the sector to develop new measures of learning gain is in progress14. Until new measures become available and are robust and applicable for all types of providers and students, we anticipate providers will refer to their own approaches to identifying and assessing students’ learning gain - this aspect is not prescriptive about what those measures might be.” Section 4.9, Pg. 20 - Department for Education (September, 2016). Teaching Excellence Framework: year two specification

Back to the Context… “Work across the sector to develop new measures of learning gain is in progress14. Until new measures become available and are robust and applicable for all types of providers and students, we anticipate providers will refer to their own approaches to identifying and assessing students’ learning gain - this aspect is not prescriptive about what those measures might be.” Section 4.9, Pg. 20 - Department for Education (September, 2016). Teaching Excellence Framework: year two specification

Back to the Context… “Work across the sector to develop new measures of learning gain is in progress14. Until new measures become available and are robust and applicable for all types of providers and students, we anticipate providers will refer to their own approaches to identifying and assessing students’ learning gain - this aspect is not prescriptive about what those measures might be.” Section 4.9, Pg. 20 - Department for Education (September, 2016). Teaching Excellence Framework: year two specification

The future… Complete Qualitative Data Collection (pushing for September 2017) Currently collecting Ports Only Time 2 Data (50% response so far) Ports Only Time 3 (May 2018) Collect Full Team Cohort Time 2 (September 2017) and Time 3 (May 2018) Gather student records data etc. Generate individual student reports Stakeholder Conference January 2019

Any questions before we move on to the discussion? Thank you! Any questions before we move on to the discussion? emily.mason-apps@port.ac.uk @UoPLearningGain

Three parts (if we have time!) Discussion Three parts (if we have time!) Tell us what you think as stakeholders Help us in the development of the project Improving learning gain- non-changing mindsets