1. Learning Gain: An Agenda for Change
Thursday 27th September 2018
@LEGACYLGproject
#learninggain

2. A comprehensive conceptual framework and measurement instrument
Learning gain
A comprehensive conceptual framework and measurement instrument
Jan Vermunt, Sonia Ilie & Anna Vignoles, with Ashton Brown & Sara Curran

3. Cambridge Strand Aims
Develop a context-appropriate theoretical understanding of learning gain Develop and test an instrument to measure learning gain: reliability, validity and at-scale usability Test a longitudinal model of learning gain in relation to student background characteristics and contextual factors

4. 3,189 2,553 6,275 Qualitative Component
Literature and evidence review
Qualitative Component
Quantitative Component: Survey
Theoretical framework development
Interviews: 33 students in 3 universities: 4 subjects undergraduate and postgraduate levels
Round 1
6,275
Round 2
3,189
Round 3
2,553
Admin data
HESA matching
Oct 2016
June 2017
April 2018
Measurement instrument development and testing

5. Learning gain
“Students’ change in knowledge, skills, attitudes, and values that may occur during higher education across disciplines”
Vermunt, Ilie & Vignoles (2018)

6. Socio-communicative component
Conceptual framework
Cognitive
component
Critical thinking
Analytical thinking
Cognitive abilities
Synthesising
Analysing
Evaluating
Problem solving
Meta-cognitive
Self-regulation
Life-long learning attitude and motivation
Learning to learn
Need for cognition (information seeking)
Affective
Attitudes towards own discipline and learning/studying
Motivation
Engagement
Professional and academic interest
Socio-communicative component
Levels of belonging in social learning networks
Social embeddedness
Communication skills
View of knowledge dimension | Epistemological beliefs; View of intelligence; Open-mindedness
Research dimension | Curiosity; Interest in research; Interest in knowledge; Attitude to sharing ideas
Moral dimension | Moral reasoning

7. Concept to measurement
Cognitive
component
Critical processing
Relating and structuring
Reasoning ability
Meta-cognitive
Self-regulation
Lack of regulation
Grit
Self-management
Affective
Emotional engagement
Social engagement
Socio-communicative component
Academic writing practices
View of knowledge and learning dimension | Epistemological beliefs
Research dimension | Attitude to research

8. For more detail on the framework

9. What have we learned?
Student engagement with learning gain measures is not trivial, but is achievable, through explicit aims, considered timings, and good survey design The measures are on the whole robust, but some aspects continue to remain elusive to measurement (i.e. epistemic beliefs) The measures are user-friendly, with short survey times, even when including the test element, and with returners much more likely to engage fully and completely with the instrument

10. Can we capture learning gain?
Yes, across disciplines, we observe change in the skills, abilities, and competencies included in our measurement instrument. This change takes different trajectories for different: Skills, abilities, and competencies Academic disciplines Years of study and stages (undergraduate/postgraduate)

11. Example: critical processing of information

12. Example: self-management

13. What have we achieved?
Developed a context-appropriate conceptual framework Developed, tested, and validated a measurement instrument that Draws on the conceptual framework Introduces new measurement tools Is able to capture differential learning gain

14. What have we achieved?
Explored the patterns of learning gain for different
institutions,
disciplines,
stages,
backgrounds,
And concluded that the instrument could be effectively used:
for local course/programme/module improvement
for evidencing this improvement
but not as an external accountability tool at the institutional level

15. Employability Learning: Measuring the Gain
Dr Nalayini Thambar, The University of Nottingham Eluned Jones, The University of Birmingham Anne Wilson, The University of Warwick Dr Toni Wright, Newman University

16. Employability & Learning Gain
“The lifelong capability to make well informed realistic plans for the future and to be ready, willing and able to execute these in a changing world”
(Wright, Jones, Thambar, & Wilson 2018, adapted from Gilworth, 2017)
The Learning Gain Opportunity:
to investigate the impact of a tool which can be used to help students to understand their strengths and apply them to career decision-making and selection processes
to investigate ‘Internationalisation’ as a factor in student employability.
to develop a tool to help non-career-focussed students to develop career management capability
to consider processes which facilitate and limit the role of HEIs in enhancing student employability

17. Career Adapt-abilities Strand
The University of Nottingham with the Universities of Birmingham, Bristol, Durham and York, Imperial College and Kings College London

18. Career Adaptability
“The readiness to cope with the predictable tasks of preparing for and participating in the work role and with the unpredictable adjustments prompted by changes in work and working conditions” (Savickas, 1997 p. 254) Four Career Adapt-abilities
Concern - Developing a positive optimistic attitude to the future
Control - Exerting a degree of intrapersonal influence on situations
Curiosity - Broadening horizons by exploring social opportunities and possibilities
Confidence - Believing in oneself and in the ability to achieve a goal
CAAS-International validated internationally, large body of literature: CAAS-UK
Built the first openly accessible online Career Adaptability survey using CAAS – UK
Mixed methodology, longitudinal study

19. The online questionnaire
Introduced in partnership with academic colleagues with a further introductory video when students logged in to complete it
Translation of Concepts:
Concern = Interest in the future
Confidence… in developing your career
Curiosity = Thinking about your possibilities
Control = Shaping yourself and your environment
Question to understand student engagement with a range of careers activities
Feedback
Introduction with overall score and reference to expert careers services
Commentary that relates to their overall low, medium or high score
Feedback for each Adapt-ability relevant to a low, medium or high score

20. Career Adaptability - findings
CAAS-UK Scores
Significant difference between mean scores for concern between time 1 (Nov 16) and time 4 (June 18) not for other factors, nor total career adapt- ability
Significantly more students have engaged in employability development behaviour at time 4 than time 1 - supports the increase in engagement and positive attitude indicated by difference in concern scores
Correlation between career related activity and career adaptability factor scores – significant relationships, small effect sizes
Parallel cohorts identified as more engaged showed higher concern and career related activity and lower confidence than original cohorts

21. Career Adaptability - findings
Qualitative data Year 1
Specific identified gains from doing the online questionnaire included raising of awareness and prompting thought and action about career-related activity
Students were able to identify sources of support and guidance both within and outside their home departments as well as noting that face to face and subject related guidance were seen as preferable
Factors which impacted on engagement with both the research and career-related behaviour included varying degrees of interest, incentive, awareness and experience as well as timing of any opportunities.

22. Career Adaptability - findings
Qualitative data Year 2
Gains continued to be identified in terms of prompts to thought & action
Generally the feedback given by the online questionnaire was seen as helpful although some interviewees felt it could be repetitive and others called for more institutional specificity
Relatedness between academic teaching and careers was identified as an issue

23. Career Adaptability - outcomes
Learning is not linear
Learning can be qualitative and not quantitative
Prompt to thought and action
Online tool for use by HE sector in future

24. International Strand Led by Birmingham in partnership with Cambridge, University College London, Kings College London, Southampton, Exeter and Liverpool

25. Internationalisation
Impact of international experience on students’ employability (Wright and Jones, 2014)
Study abroad
Work abroad
On-campus experiences
Research suggests internationalisation enhances students’ employability skills (e.g. UUKI, 2017)
Qualitative methodology - Semi-structured interviews both pre and post experiences plus subjective quantitative ratings
Identify hopes, expectations and self-perceived employability learning gains
Identify good practice examples – on campus experiences

26. Internationalisation - findings
Take - Phase 2 report
Study abroad
all pre experience (saturation reached) Three main areas identified in terms of motivation and hoped for gains: academic related, employability related, personal development
Work abroad
all pre experience (saturation)
Employability gains -Extrinsic factors (CVs and experience) were mentioned, but personal devpt gains intrinsic motivations were more frequent and more emphasised in the accounts. Hoped for gains tended to match global competencies in the literature (Diamond et al, 2011) but were not necessarily related to employability by participants
On-campus experience
Pre - saturation
Reasons for not undertaking international experience tend to suggest risk averse behaviour. Importance of international competency emphasised
For those who have post interviews (not complete)
Despite no reported change in quantitative perception of global competencies (op cit), all reported a perceived positive impact (even if only slight) on at least one competency post experience. When the individual transcripts are more closely examined two patterns seem to emerge. A minority of participants give lower self rating scores post experience, having learned that they did not know as much as they thought they did, and a larger group whose self-ratings remained largely static.

27. Internationalisation - findings
Take - Phase 2 report
Study abroad
all pre experience (saturation reached) Three main areas identified in terms of motivation and hoped for gains: academic related, employability related, personal development
Work abroad
all pre experience (saturation)
Employability gains -Extrinsic factors (CVs and experience) were mentioned, but personal devpt gains intrinsic motivations were more frequent and more emphasised in the accounts. Hoped for gains tended to match global competencies in the literature (Diamond et al, 2011) but were not necessarily related to employability by participants
On-campus experience
Pre - saturation
Reasons for not undertaking international experience tend to suggest risk averse behaviour. Importance of international competency emphasised
For those who have post interviews (not complete)
Despite no reported change in quantitative perception of global competencies (op cit), all reported a perceived positive impact (even if only slight) on at least one competency post experience. When the individual transcripts are more closely examined two patterns seem to emerge. A minority of participants give lower self rating scores post experience, having learned that they did not know as much as they thought they did, and a larger group whose self-ratings remained largely static.

28. Internationalisation - findings
Take - report
Study abroad
all pre experience (saturation reached) Three main areas identified in terms of motivation and hoped for gains: academic related, employability related, personal development
Work abroad
all pre experience (saturation)
Employability gains -Extrinsic factors (CVs and experience) were mentioned, but personal devpt gains intrinsic motivations were more frequent and more emphasised in the accounts. Hoped for gains tended to match global competencies in the literature (Diamond et al, 2011) but were not necessarily related to employability by participants
On-campus experience
Pre - saturation
Reasons for not undertaking international experience tend to suggest risk averse behaviour. Importance of international competency emphasised
For those who have post interviews (not complete)
Despite no reported change in quantitative perception of global competencies (op cit), all reported a perceived positive impact (even if only slight) on at least one competency post experience. When the individual transcripts are more closely examined two patterns seem to emerge. A minority of participants give lower self rating scores post experience, having learned that they did not know as much as they thought they did, and a larger group whose self-ratings remained largely static.

29. Mean rating differences:
Internationalisation - findings
Mean rating differences:
Competence
On campus
Study Abroad
Work Abroad
Mean
rating difference P
Ability to work in diverse teams
+.10 NS
+.59
<.001
+.50
.001
Communication skills
+.17
+.69
+.79
Drive and resilience
+.11
+.46
+.38
<.05
Ability to appreciate the point of view of someone who is different to you
+.14
+.53
+.76
Ability to learn new things and adapt to the situation
+.19
+.42
<.005
+.40
Self-Awareness
<0.01
+.45
+.73

30. Outcomes
Provide the sector with better evidence on what students believe they gain from international experience
Use evidence to better design and promote international experiences
Help create enhanced pre-experience preparatory resources and post-experience reflection
Help to identify activities and approaches that enhance employability learning gain from on-campus opportunities

31. Anne Wilson, Head of Careers University of Warwick
R2 Strengths
Anne Wilson, Head of Careers
University of Warwick
Supported by:
Dr Heike Behle, University of Warwick Quantitative Research
Arianna Tassinari, PhD student University of Warwick Qualitative Research

32. R2 Strengths
Does the intervention of an R2 Strengths profile measure development of Finalist Undergraduate:
Self-awareness
Confidence
Self-efficacy
Career readiness

33. R2 4M Strengths Model

34. What worked well?
Qualitative interviews highlighted the value of 1-1 feedback with a Strengths trained careers adviser.

35. What worked less well?

36. Learning Gain?
‘There are known knowns. There are things we know that we know. There are known unknowns. That is to say, there are things we know we don’t know. But there are also unknown unknowns. There are things we don’t know we don’t know.’ Donald Rumsfeld.

37. Scalability

38. Going forward? Building Strengths into a University award
Training more staff
Recruiter involvement
Online resources and a quiz
Card sort activities
Staff team training
Workshops using cards on the Jobmi report
WP project funding with Strengths
Targeted profiles to female students on ‘Sprint’
Focused activity on smaller departments where DLHE/GO need improvement

39. Know yourself Explore Decide Action In summary:
Strengths clearly has value when used 1-1 with students
Learning Gain is not linear but iterative
R2 Strengths could be scalable to a limited extent depending though the personal nature of the learning gained means
There is a risk with Learning Gain if we’re not careful that we attempt to hit the target but miss the point.
There can be real richness in personal learning- but that this is not cheap- and the quantitative approach to measuring learning gain with this project does not capture the complexity of employability learning (see diagram)

40. Employability Framework

41. Employability Framework
Enabling Support Factors
Individual Circumstances
Individual Factors
Local & National Labour Market Factors
Employability
Degree
Life/ Work experience
Extra-curricular communities of practice
Awareness
Transferable Skills
Personal Qualities, Efficacy, Self-beliefs
Meta-Cognition
Skilful Practice
Subject Understanding
Individual factors – skills Basic skills, Soft skills, ‘employability skills’, subject-specific skills
Other Individual factors
Person-centred factors (which HEI? Class of degree?), subjective employability, Adaptability and mobility
Individual circumstances Work culture, Regional factors, Networks, Transitions
Labour Market Factors Labour Demand, Rules and Regulations, Employers’ practices
Enabling factors Labour Market Policy, Human Resource management, education and training - changing labour market and skills requirement - careers guidance
Wright & Behle, 2018 (adapted from Wright, 2017 & Green et al, 2013)

42. Employability Framework
HEIs can enhance student employability learning gain through the course studied, enriching life and work experience, and by encouraging engagement in and provision of extra-curricular communities of practice.
This process tends to be facilitated via what we have described as enabling support factors e.g. awareness and skills enhancement but is limited by other factors such as individual circumstances and local and national labour market factors

43. References
Gilworth, R (2017) ‘Let’s start from where you are’ Impact and implications of Careers Registration. LEGACY Seminar: Exploring Learning Gain in Higher Education. 8 February 2017, University of Nottingham
Green, A., de Hoyos, M., Barnes, S., Owen, D., Baldauf, B., & Behle, H. (2013) Literature Review on Employability, Inclusion and ICT, Report 1: The Concept of Employability with a Specific Focus on Young People, Older Workers and Migrants. JRC Technical Report, European Union: Luxembourg. Retrieved from: technical-research-reports/literature-review-employability-inclusion-and-ict-report-1-concept- employability-specific
Savickas, M. L. (1997). "Career adaptability: An integrative construct for life‐span, life‐space theory." The Career Development Quarterly, 45(3):
UUKI (2017) Gone International: Mobility Works. London: Universities UK International
Wright, T., and Jones, E. (2014) Student International Experience and Perceived Employability Gains, Retrieved from:
Wright, T. & Jeffries-Watts, S. (2017). Does it do what it says on the tin?– Evaluation of an established UK university employability award. Journal of Teaching and Learning for Graduate Employability, 8(1),
For further information