1. Professor Stuart Brand and Jamie Morris
The Collegiate Learning Assessment (CLA+): An institutional case study of using standardised testing to measure and develop students’ generic skills.
Professor Stuart Brand and Jamie Morris
European First Year Experience Conference
Utrecht 2018

2. Institutional Context – Birmingham City University
Post 1992 University
~24,000 Students
8 Campuses moving to 2/3
Vocational approach to learning
Multicultural student population
Commuter students (69%)

3. National Context - the Teaching Excellence Framework (TEF)
The TEF aims to recognise and reward excellence in teaching and learning, and help inform prospective students' choices for higher education.
The TEF was developed by the Department for Education in England.
It was voluntary to participate in.
Universities can qualify for awards by using the following metrics...

4. TEF cont…
The Teaching Excellence and Student Outcomes Framework (TEF) aims to recognise and reward excellence in teaching, learning and student outcomes.

5. TEF cont…
The TEF is being developed via a phased approach:

6. International Context - The RAND Report (2015)
/learninggain/
Commissioned by the Department for Business Innovation and Skills (BIS), the Higher Education Funding Council for England (HEFCE) and the Higher Education Academy (HEA)
Rand Europe

7. What is ‘Learning Gain’?
The Office for Students characterise ‘Learning Gain’ as the following:
An attempt to measure the improvement in knowledge, skills, work-readiness and personal development made by students during their time spent in higher education.
In our case - what level of skills do students enter and leave with when they undertake their course of study?
How do curriculum and extra-curricular activities contribute to the development of these skills?

8. Group Activity – Discussion
On your tables, discuss the following questions:
What does ‘Learning Gain’ mean to you and your own subject/discipline?
Do you do anything already in regards to tracking the ‘value added’ to your programmes/courses?
If so, what data do you use and how do you collect it?

9. HEFCE and ‘Learning Gain’
The Higher Education Funding Council for England (HEFCE) released a funding call in 2015 for new learning gain projects in selected Universities throughout England
Decisions released formally at end of July 2015
13 collaborative projects funded
Birmingham City bid with three partners: Coventry, Liverpool John Moores and Staffordshire

10. Project Overview - Revised Proposal
Original Proposal – longitudinal study, 1,000 students with 4 testing points.
Revised to:
2 longitudinal studies from and
Follows an additional ~1000 students
2 Cross sectional study in Semester 2 of and academic year
Test a mixture of L4 (1st year), L5 (2nd year) and L6 (3rd year)
4 testing points reduced to 3:
Autumn of year 1
Spring of year 2
Spring of year 3

11. Methodologies
Standardised tests - These measure the skills students acquire, which may be generic or specialised (for example, a skill specific to a discipline).
Self-reporting surveys - Such surveys ask students to report the extent to which they think they have gained knowledge and developed skills.
Other qualitative methods - These encourage students to reflect on their learning, acquired skills and skills gaps.
Mixed methods - This draws on a range of tools and indicators to track improvement in performance, (for example through a combination of grades, student learning data and student surveys).

12. Emergent approaches Cognitive gain – what students think and know
Soft skills development – affective measures of attitudes and how students feel
Employability and careers readiness – mainly behavioural measures of activities students have undertaken in preparation for the world of work

13. The Council for Aid to Education (CAE)
In 2002, CAE, a not for profit based in NYC, launched a national effort to assess the quality of undergraduate education.
CLA+ (Collegiate Learning Assessment) is a web administered, value added assessment programme.
The aim of CLA+ is to provide an objective assessment about the critical-thinking skills a student possesses as they enter and exit college.

14. The Collegiate Learning Assessment (CLA+)
The Collegiate Learning Assessment (CLA+) is a performance based test that measures critical thinking, problem solving, analytic reasoning, and writing skills.
The CLA+ uses a Performance Task (60”) and 25 Selected-response Questions (30”).
Can take up to 2 hours to complete - Tutorials and Demographic Survey.
PT responses are scored (1-6) by staff leads in each institution - requires scorer training.

15. Why CLA+? (CLA post 2011)
Generates an individual student report based on a Mastery Level (1-5)
Mastery Levels are formed via a combination of scores for the Performance Task and Selected-Response Questions.
Provides a rich source of data from the U.S and other participating countries e.g. Italy, Germany, Chile and Peru using the CLA+.
Key point - Emphasis on highlighting areas of development for the student.

16. How have we embedded the CLA+? – Optional vs Compulsory
1st and 2nd year of testing - financial incentives
Low attendance - first two years
Successful engagement with the Birmingham City Business School
Embedded into Professional Studies Module as formative assessment.
Importance of engaging programme and modular level staff

17. Findings from Optional Testing – Focus Groups
Structure – Issues with lack of spell check and viewing content.
Most students believed that the test was not subject independent.
The time the test was taken in the academic year was an issue
Students would take the CLA+ test if employers and other HEIs valued the scores from it.
Students who had taken the test twice noted enhancement of other skills e.g. time management

18. Focus Group Findings (optional testing) – cont…
Cross-sectional study Spring 2017 (n=78)
Students also reported specific disciplines may be advantaged in the test

19. Autumn (Fall) 2017 Testing – Compulsory
201 L4 students tested (81% response rate) in a week
Previous issues with recruitment no longer encountered
Focus on one participating School - Birmingham City Business School
Testing embedded into a core module
Tied directly to summative assessment – E-Portfolio

20. MKT4020 - Professional Development
Developed via the University’s Transforming the Curriculum (TtC) process
CLA+ embedded into Week 2 of the Module, focusing on Critical Thinking:
LO1 - Undertake an audit of skills and capabilities, reflecting on the outcomes and engage with a programme of professional development.
LO3 - Evidence skills in the context of professional development and evaluate impact.
Other sessions include:
Building competency with contemporary IT software e.g. Microsoft Office
Enhancing presentation skills
Forming a professional portfolio and profile via e-portfolio and Social Media

21. MKT4020 – Professional Development
Reports distributed in Week 7 employability week in bespoke activity
Tied into extra curricular awards framework – Graduate+
Majority of the reports collected!
CLA+ individual student report submitted as evidence for summative
assessment
Electronic copies also available to send to students who could not attend.

22. Results – Autumn (Fall) 2017

23. Cont..

25. Fall – Mastery Levels by Ethnicity

26. Native vs 2nd Language

27. Summary of Testing Numbers 2015/2018 - Birmingham City University
Institution total:
2015/16 academic year - 220
2016/17 academic year - 299
2017/18 academic year – 259
Overall Total (Across 4 partners):
longitudinal study – 845 students tested
longitudinal study – 693 students tested
2017 (Spring) cross-sectional study – 201 students tested
2017 (Autumn) cross-sectional study – 248 students tested
2018 (Spring) cross-sectional study – 167 students tested
Total number of tests administered – 2,154

28. CLA+ as a diagnostic tool – Future Developments
Interest from numerous staff at BCU on utilising the test as part of a foundation degree.
Could also provide comparative data if L4s are tested.
CLA+ to be continue being used in Birmingham City Business School programmes
CAE to provide anonymised, aggregated analysis of student performance across 4 partners

29. Group Activity – Discussion Point
On your tables, discuss the following questions:
Do you use diagnostic testing in your practice/institution? How so and what is its purpose?
Do you use any other diagnostic methods in your practice?
In your experience, what works and what doesn’t in relation to diagnostics?

30. Thank you for listening!
Any questions?