Professor Stuart Brand and Jamie Morris The Collegiate Learning Assessment (CLA+): An institutional case study of using standardised testing to measure and develop students’ generic skills. Professor Stuart Brand and Jamie Morris European First Year Experience Conference Utrecht 2018

Institutional Context – Birmingham City University Post 1992 University ~24,000 Students 8 Campuses moving to 2/3 Vocational approach to learning Multicultural student population Commuter students (69%)

National Context - the Teaching Excellence Framework (TEF) The TEF aims to recognise and reward excellence in teaching and learning, and help inform prospective students' choices for higher education. The TEF was developed by the Department for Education in England. It was voluntary to participate in. Universities can qualify for awards by using the following metrics...

TEF cont… The Teaching Excellence and Student Outcomes Framework (TEF) aims to recognise and reward excellence in teaching, learning and student outcomes.

TEF cont… The TEF is being developed via a phased approach:

International Context - The RAND Report (2015) http://www.hefce.ac.uk/pubs/rereports/year/2015 /learninggain/ Commissioned by the Department for Business Innovation and Skills (BIS), the Higher Education Funding Council for England (HEFCE) and the Higher Education Academy (HEA) Rand Europe

What is ‘Learning Gain’? The Office for Students characterise ‘Learning Gain’ as the following: An attempt to measure the improvement in knowledge, skills, work-readiness and personal development made by students during their time spent in higher education. In our case - what level of skills do students enter and leave with when they undertake their course of study? How do curriculum and extra-curricular activities contribute to the development of these skills?

Group Activity – Discussion On your tables, discuss the following questions: What does ‘Learning Gain’ mean to you and your own subject/discipline? Do you do anything already in regards to tracking the ‘value added’ to your programmes/courses? If so, what data do you use and how do you collect it?

HEFCE and ‘Learning Gain’ The Higher Education Funding Council for England (HEFCE) released a funding call in 2015 for new learning gain projects in selected Universities throughout England Decisions released formally at end of July 2015 13 collaborative projects funded Birmingham City bid with three partners: Coventry, Liverpool John Moores and Staffordshire

Project Overview - Revised Proposal Original Proposal – longitudinal study, 1,000 students with 4 testing points. Revised to: 2 longitudinal studies from 2015-18 and 2016-19 Follows an additional ~1000 students 2 Cross sectional study in Semester 2 of 2016-17 and 2017-18 academic year Test a mixture of L4 (1st year), L5 (2nd year) and L6 (3rd year) 4 testing points reduced to 3: Autumn of year 1 Spring of year 2 Spring of year 3

Methodologies Standardised tests - These measure the skills students acquire, which may be generic or specialised (for example, a skill specific to a discipline). Self-reporting surveys - Such surveys ask students to report the extent to which they think they have gained knowledge and developed skills. Other qualitative methods - These encourage students to reflect on their learning, acquired skills and skills gaps. Mixed methods - This draws on a range of tools and indicators to track improvement in performance, (for example through a combination of grades, student learning data and student surveys).

Emergent approaches Cognitive gain – what students think and know Soft skills development – affective measures of attitudes and how students feel Employability and careers readiness – mainly behavioural measures of activities students have undertaken in preparation for the world of work

The Council for Aid to Education (CAE) In 2002, CAE, a not for profit based in NYC, launched a national effort to assess the quality of undergraduate education. CLA+ (Collegiate Learning Assessment) is a web administered, value added assessment programme. The aim of CLA+ is to provide an objective assessment about the critical-thinking skills a student possesses as they enter and exit college.

The Collegiate Learning Assessment (CLA+) The Collegiate Learning Assessment (CLA+) is a performance based test that measures critical thinking, problem solving, analytic reasoning, and writing skills. The CLA+ uses a Performance Task (60”) and 25 Selected-response Questions (30”). Can take up to 2 hours to complete - Tutorials and Demographic Survey. PT responses are scored (1-6) by staff leads in each institution - requires scorer training.

Why CLA+? (CLA post 2011) Generates an individual student report based on a Mastery Level (1-5) Mastery Levels are formed via a combination of scores for the Performance Task and Selected-Response Questions. Provides a rich source of data from the U.S and other participating countries e.g. Italy, Germany, Chile and Peru using the CLA+. Key point - Emphasis on highlighting areas of development for the student.

How have we embedded the CLA+? – Optional vs Compulsory 1st and 2nd year of testing - financial incentives Low attendance - first two years Successful engagement with the Birmingham City Business School Embedded into Professional Studies Module as formative assessment. Importance of engaging programme and modular level staff

Findings from Optional Testing – Focus Groups Structure – Issues with lack of spell check and viewing content. Most students believed that the test was not subject independent. The time the test was taken in the academic year was an issue Students would take the CLA+ test if employers and other HEIs valued the scores from it. Students who had taken the test twice noted enhancement of other skills e.g. time management

Focus Group Findings (optional testing) – cont… Cross-sectional study Spring 2017 (n=78) Students also reported specific disciplines may be advantaged in the test

Autumn (Fall) 2017 Testing – Compulsory 201 L4 students tested (81% response rate) in a week Previous issues with recruitment no longer encountered Focus on one participating School - Birmingham City Business School Testing embedded into a core module Tied directly to summative assessment – E-Portfolio

MKT4020 - Professional Development Developed via the University’s Transforming the Curriculum (TtC) process CLA+ embedded into Week 2 of the Module, focusing on Critical Thinking: LO1 - Undertake an audit of skills and capabilities, reflecting on the outcomes and engage with a programme of professional development. LO3 - Evidence skills in the context of professional development and evaluate impact. Other sessions include: Building competency with contemporary IT software e.g. Microsoft Office Enhancing presentation skills Forming a professional portfolio and profile via e-portfolio and Social Media

MKT4020 – Professional Development Reports distributed in Week 7 employability week in bespoke activity Tied into extra curricular awards framework – Graduate+ Majority of the reports collected! CLA+ individual student report submitted as evidence for summative assessment Electronic copies also available to send to students who could not attend.

Results – Autumn (Fall) 2017

Cont..

Fall – Mastery Levels by Ethnicity

Native vs 2nd Language

Summary of Testing Numbers 2015/2018 - Birmingham City University Institution total: 2015/16 academic year - 220 2016/17 academic year - 299 2017/18 academic year – 259 Overall Total (Across 4 partners): 2015-2018 longitudinal study – 845 students tested 2016-2019 longitudinal study – 693 students tested 2017 (Spring) cross-sectional study – 201 students tested 2017 (Autumn) cross-sectional study – 248 students tested 2018 (Spring) cross-sectional study – 167 students tested Total number of tests administered – 2,154

CLA+ as a diagnostic tool – Future Developments Interest from numerous staff at BCU on utilising the test as part of a foundation degree. Could also provide comparative data if L4s are tested. CLA+ to be continue being used in Birmingham City Business School programmes CAE to provide anonymised, aggregated analysis of student performance across 4 partners

Group Activity – Discussion Point On your tables, discuss the following questions: Do you use diagnostic testing in your practice/institution? How so and what is its purpose? Do you use any other diagnostic methods in your practice? In your experience, what works and what doesn’t in relation to diagnostics?

Thank you for listening! Any questions?