International Assessment of Generic Skills Doris Zahner Kelly Rotholz October 12, 2018 |Toronto, Canada

Overview & Introduction

Overview The focus on Generic Skills Predictive Validity CLA+ International Example Use Cases Research & Development

Why Focus on Generic Skills? critical thinking problem solving qualitative and quantitative reasoning written communication are judged by employers, professors, and thought leaders to be very important in today’s knowledge economy.

Specifically National leaders want to know how well their universities are preparing their students Faculty and administrators want to know how to improve their teaching and learning Students want credible, third-party evidence (certificates) of their skill levels obtained when they graduate to show potential employers Employers want credible evidence of the skill levels job applicants have not only grades students’ received in college Context: Over 100 million college students world wide today. Up to 500 million projected by 2030.

Critical Policy Issues CLA+ International attempts to evaluate and solve the following kinds of problems: The inability of employers to identify high ability students who attend less selective universities The national level measurement of generic skills in tracking economic productivity growth The rising costs, access deficits, disruptive changes such as student loan debt, and the decline of permanent faculty affects the quality of student learning

Post-UNIVERSITY outcomes Study 1 Zahner & James, 2016

Post-University Outcomes, Study 1 University seniors – CLA+ in spring 2014 Surveyed 3, 6, and 12 months following graduation Approximately 13,000+ students 1,585 agreed to participate in the survey 993 persisted through all three phases

Post-University Outcomes, Study 1 Predictive Validity of CLA+ and Other Variables on Post-University Outcomes   Post-university outcomes (all) Salary Employ Full-time employ Grad school CLA+ and EAA * Barron’s Field of study Gender Parent education Race/Ethnicity

Post-University Outcomes, Study 1

Post-University Outcomes, Study 1 CLA+ is predictive of post-university outcomes such as employment and salary. Use of CLA+ scores can address some of the challenges employers and recent graduates face by highlighting these critical skills. Assessments like the CLA+ serve as both an effective instrument for identifying high-achieving students from less and non-competitive institutions and making their skills more visible to perspective employees.

Post-UNIVERSITY outcomes Study 2 Zahner & Lehrfeld, 2017

Post-University Outcomes, Study 2 University seniors – took CLA+ in spring 2014 Fall 2015 – Surveyed 75 employers of original cohort of students

Post-University Outcomes, Study 2 On a scale from 1 – 5, How important are the following skills to successful performance in the participant’s position: Analysis and Problem Solving Writing Effectiveness Writing Mechanics How would you rate the participant on the following skills: Overall, where does the participant’s performance rank compared to other recent university graduates in your workplace?

Post-University Outcomes, Study 2

Post-University Outcomes, Study 2 Ordinal logistic regression models for predicting participants’ post-university performance as measured by their managers   APS WE WM Relative Rank CLA+ * *   t-statistic  17.55** 26.00*  23.80** 16.56** 

Post-University Outcomes, Study 2 Employers find critical thinking and written communication skills to be important or very important for entry-level positions in the workforce. Students with higher CLA+ plus scores tended to have higher ratings from their employers Findings from this study offer support for the conclusion that critical-thinking and written-communication skills are important in predicting career placement and workplace success (Arum & Roksa, 2014).

CLA+ International

CLA+ International 90 minute assessment of generic skills based on the Collegiate Learning Assessment which requires the student to demonstrate their generic thinking abilities. It features: Performance Task (a real world problem) that is document based and requires an open ended response (60 minutes) 25 selected response questions (30 minutes)

The CLA+ International Protocol Measures six sub scores PT: Analysis and Problem Solving (APS) Writing Effectiveness (WE) Writing Mechanics (WM) SRQ Critical Reading and Evaluation (CRE) Critique an Argument (CA) Scientific and Quantitative Reasoning (SRQ)

Important Milestones and Activities Online test platform will be refitted and translated into one language per country/association Translation of all test materials and scoring rubrics Administrative instructions and guides provided to member teams Cognitive labs performed with CAE oversight Exemplary best practices, communications materials, training and logistics guidance provided Student recruitment with CAE support Test administration using secure online testing platform Program-wide in-person training of Lead Scorers by CAE Scoring by member scorers using CAE’s secure online scoring platform CAE review and analysis of data followed by preparation of member reports Individual student reports and secure badges/certicates prepared and distributed by CAE and/or member

Translation and Adaptation Overview Translatability overview Double translation and adaptation Member team review Focused verification Cognitive labs

Example use cases for CLA+ international

Institutions, consortia, and ministries use CLA+ International to:  Estimate institutional and individual student learning gains in critical-thinking and written communication Research the efficacy of general education programs and curriculum changes Demonstrate individual, class, and institutional proficiency 

Legal mandate to evaluate learning outcomes ANVUR TECO Testing Legal mandate to evaluate learning outcomes Translated and adapted CLA+ to Italian in 2013 and 2015 Assessed over 11,000 exiting students across 23 universities Zahner, D., & Ciolfi, A. (2018). International Comparison of a Performance-Based Assessment in Higher Education. In O. Troitschanskaia (Ed.), Assessment of Learning Outcomes in higher education – Cross-national comparisons and perspectives. (pp. 215-244). New York, NY: Springer.

England HEFCE funded pilot project to measure Learning Gains in universities across England Birmingham City University is leading a consortium of 4 partners (Coventry University, Liverpool John Moores University, Staffordshire University) University of Reading is participating independently, but collaborating with the BCU consortium Adapted testing platform and test items to British English Longitudinal and Cross-sectional testing model

Universidad de Guadalajara Translation and Adaptation of CLA+ to Spanish Assessment of learning outcomes for over 6,000 students across 55 bachelors degree programs Longitudinal and Cross-sectional Control Group (completed both EGEL and CLA+) University of Guadalajara has applied the Examen General de Egreso de la Licenciatura (EGEL for its acronym in Spanish) since 1995.  This examination is elaborated and administered by the National Center for the Evaluation of Higher Education (CENEVAL for its acronym in Spanish).  The objective of the EGEL is to measure the learning outcomes of undergraduate students.  However, our institution currently has 55 bachelor's degree programs that do not have an EGEL exam related to their discipline, mainly because they are innovative multidisciplinary programs.

Akita International University Cross-sectional testing model Cohort 1: Incoming students during orientation Cohort 2: Exiting students approaching degree completion Akita International University is a young university in Japan. They have been participating in CLA+ for 5 years. They use a traditional cross-sectional testing model, testing entering and exiting students. They are interested in their institutional value-added scores and mastery levels.

18 Universities in Finland  Universities, applied sciences, art, and defense Cross-sectional testing model Spring 2019

Future research

Future Research & Development CLA+ Deconstructed Predictive validity Efficacy of programs Construct equivalence across countries Educational policy impact (e.g., skills gap, leveling the playing field)

SkillMetric 60 minute assessment of generic skills based on the CLA+ It features: Performance Task (a real world problem) that is document based and requires students to demonstrate their analysis and problem solving skills (30 minutes) 25 selected response questions (30 minutes)

SkillMetric Measures Analytic Reasoning and Evaluation and Problem Solving with an optional writing section PT: Analysis and Problem Solving (APS) SRQ Critical Reading and Evaluation (CRE) Critique an Argument (CA) Scientific and Quantitative Reasoning (SRQ)

SkillMetric Completed: Assessment development Scoring key Pilot testing Scoring Item and data analysis Scaling and equating Standard setting study

SkillMetric To be completed: Equivalency (to CLA+) study Marketing and Branding Website

Questions & Comments