LIMITLESS POTENTIAL | LIMITLESS OPPORTUNITIES | LIMITLESS IMPACT Copyright University of Reading Dr Amanda Callaghan (Project Lead) Cesare Aloisi (Project Officer) 1 LEARNING GAIN: LINKING STUDENT ATTAINMENT WITH ENGAGEMENT, SATISFACTION AND WELLBEING

White Paper May 2016 Proposed that the TEF would consider teaching and learning excellence across three main aspects: Teaching Quality, Learning Environment, Student Outcomes and Learning Gain.

LIMITLESS POTENTIAL | LIMITLESS OPPORTUNITIES | LIMITLESS IMPACT DEFINITION Learning Gain can be described as the ‘distance travelled’ by students during their studies, demonstrable by an improvement in knowledge, skills, work-readiness and personal development between two points in time. 4 McGrath et al 2015 RAND

HEFCE Learning gain project The impetus for this project comes from BIS who asked HEFCE to ‘consider whether there are better indicators, such as measures of student engagement, to provide information on what a high quality student experience looks like’. Outcomes of this project will feed into ongoing debates about the quality and impact of higher education, and how we evidence the value of investment in it. 5

LIMITLESS POTENTIAL | LIMITLESS OPPORTUNITIES | LIMITLESS IMPACT UNIVERSITY OF READING LEARNING GAIN PROJECT – MIXED METHODS A combination of methods and indicators to track improvement in performance, for example through a combination of grades, student learning data and student surveys. A three-year longitudinal project to test and evaluate a range of available methodologies and to draw conclusions on what might be the right combination of instruments for the measurement of learning gain. 6

LIMITLESS POTENTIAL | LIMITLESS OPPORTUNITIES | LIMITLESS IMPACT INTRODUCTION The University of Reading project will be modelling the effectiveness of using data that measure: student attainment, student engagement, student satisfaction, student wellbeing and critical thinking skills. Taken together they present a global view of what students gain in knowledge, skills, work-readiness and personal development. and perceived learning gain to predict attainment.

LIMITLESS POTENTIAL | LIMITLESS OPPORTUNITIES | LIMITLESS IMPACT LEARNING PYRAMID Breadth and depth of knowledge Ability to learn through research and enquiry Academic and research skills Entrepreneurship Use/applicability of discipline in real-world contexts Professional and “employability” skills Desire for self-improvement, intellectual curiosity, commitment to lifelong learning Reasoning and problem-solving skills Personal effectiveness and self-awareness Human, social, civic and environmental responsibility Social skills Distance from academic discipline

LIMITLESS POTENTIAL | LIMITLESS OPPORTUNITIES | LIMITLESS IMPACT WHAT MEASURES WOULD YOU USE? Sheet in your pack. Of the tools you have access to, what would you use to measure: Discipline specific Academic and research skills Employability Self-awareness Social skills

LIMITLESS POTENTIAL | LIMITLESS OPPORTUNITIES | LIMITLESS IMPACT MEASURES WE CHOSE Skill typeMeasures Discipline-specific Grades and progression (from SITS data) Perceived Learning Gain Academic and research, including “critical thinking” Grades and progression CLA+ Need for cognition scale (NCS) UKES Professional and “employability” Internal survey from Careers Office Destinations of Leavers from HE (DLHE) Personal effectiveness and self-awareness UKES Student wellbeing survey (WBS) NCS Social UKES WBS

LIMITLESS POTENTIAL | LIMITLESS OPPORTUNITIES | LIMITLESS IMPACT DISCIPLINE SPECIFIC: GRADES AND PROGRESSION Differences between student grades at two points in time. 11

LIMITLESS POTENTIAL | LIMITLESS OPPORTUNITIES | LIMITLESS IMPACT DATASET 1754 students from 51 programmes of studies: -Undergraduate -Full-time -Graduated in 2014/15 (pass and fail) -On time -From a 3-year course = 75% of UG FT students who graduated on time in that year = 55% of UG students who graduated in that year

LIMITLESS POTENTIAL | LIMITLESS OPPORTUNITIES | LIMITLESS IMPACT SITS data

LIMITLESS POTENTIAL | LIMITLESS OPPORTUNITIES | LIMITLESS IMPACT SITS data Programme 1

LIMITLESS POTENTIAL | LIMITLESS OPPORTUNITIES | LIMITLESS IMPACT PERCEIVED LEARNING GAIN Students rate themselves against a set of criteria linked to the learning outcomes of modules and programmes. End of each academic year and on initial arrival. Based on methods used by a medical education researcher at a German University (Dr Raupach). outcomes-with-evasys.htmlhttp:// outcomes-with-evasys.html Schiekirka et al Estimating Learning Outcomes From Pre- and Posttest Student Self-Assessments: A Longitudinal Study. Academic Medicine: March Volume 88 - Issue 3 - p 369–375 15

LIMITLESS POTENTIAL | LIMITLESS OPPORTUNITIES | LIMITLESS IMPACT ACADEMIC AND RESEARCH CLA+ The Collegiate Learning Assessment (CLA+) is a US performance based test that measures critical thinking, problem solving, analytic reasoning, and writing skills The CLA+ uses a Performance Task and a series of selected-response questions measure these higher order skills No prior knowledge of any particular field is necessary in order to perform well The principal goal is to provide an objective assessment about the critical-thinking skills a student possesses as they enter and exit college

LIMITLESS POTENTIAL | LIMITLESS OPPORTUNITIES | LIMITLESS IMPACT min 30 min

LIMITLESS POTENTIAL | LIMITLESS OPPORTUNITIES | LIMITLESS IMPACT CLA+ APPROACH AND ISSUES Randomised sampling from a selection of modules Cross-sectional and longitudinal approach First administration: ceiling effect: 6 students out of 39 are already Advanced and cannot progress any further With +1 point in the essay and +1 correct answer in the MCQ 31 out of 39 students would sit in the highest two levels.

LIMITLESS POTENTIAL | LIMITLESS OPPORTUNITIES | LIMITLESS IMPACT ISSUE - ENGAGEMENT WITH SURVEYS ETC How do you incentivise students to engage when they see no benefit for themselves? 19

LIMITLESS POTENTIAL | LIMITLESS OPPORTUNITIES | LIMITLESS IMPACT CLA+ ISSUES

LIMITLESS POTENTIAL | LIMITLESS OPPORTUNITIES | LIMITLESS IMPACT Year 1Year 2Year 3 AutumnSpringAutumnSpringAutumnSpring 40 P1 UGs 40 P1 UGs 40+”160” P3 UGs 40+”160” P3 UGs X-sectional Longitudinal comparison 160 P1 UGs 160 P1 UGs matching 40 P2 UGs 40 P2 UGs 160 P2 UGs 160 P2 UGs CLA 1 CLA 2

LIMITLESS POTENTIAL | LIMITLESS OPPORTUNITIES | LIMITLESS IMPACT THE UK ENGAGEMENT SURVEY (HEA) It is the only undergraduate survey in the UK to focus on student engagement. Aimed at non-finalists (early warning). 22

LIMITLESS POTENTIAL | LIMITLESS OPPORTUNITIES | LIMITLESS IMPACT WELLBEING AND NEED FOR COGNITION Student Wellbeing Project ( Department of Economics UoR, 2011) is a longitudinal study to measure how student wellbeing, performance, productivity and satisfaction with university provision are related. Need for Cognition Scales (NCS) is a personality instrument of metacognition intended to measure the extent to which an individual desires to engage in a cognitive activity. It can be used in parallel with the CLA+ to track development in critical thinking skills. The NCS measures predisposition to think critically rather than apply critical thinking skills in prescribed situations. 23

LIMITLESS POTENTIAL | LIMITLESS OPPORTUNITIES | LIMITLESS IMPACT EXAMPLE VALIDATION MODEL Measures of psychological well-being, financial security, sense of belonging and engagement are entered as level 1 time-changing predictors. 24

LIMITLESS POTENTIAL | LIMITLESS OPPORTUNITIES | LIMITLESS IMPACT SAMPLING APPROACH All students SITS data Many students UKES + WBS Careers A few students CLA+Perceived LG

LIMITLESS POTENTIAL | LIMITLESS OPPORTUNITIES | LIMITLESS IMPACT MEASURING THE IMPACT OF ENGAGEMENT ON EMPLOYMENT Analysis of graduates from 11/12, 12/13 and 13/14 Completion of placement activities (Reading Internship Scheme, Mini, Micro, Maxi – 1 year placement, UROP), RED award, study abroad, gender, ethnicity, fee status, disability, home Faculty, degree studied, degree classification, age on entry. Placement data – criticism of correlation = academically better students more likely to engage in placements but also more likely to get a good job. 26

LIMITLESS POTENTIAL | LIMITLESS OPPORTUNITIES | LIMITLESS IMPACT ACKNOWLEDGEMENTS Thanks to Dr Karen Ayres (Statistics) for analysis of employment. HEFCE for funding under the Learning Gain Project 27