Assessing higher education learning outcomes globally Professor Hamish Coates

Study at a glance Are valid and reliable comparisons of learning outcomes scientifically and practically feasible? Conceived ~2005, announced 06, scoping 06-09, work An international feasibility study – not a ‘pilot’ Testing ‘bachelor degree’ students, with data from 5000 faculty, 25 national managers, 250 institutional coordinators, 17 policymakers, and hundreds of advisors and stakeholders Designed, developed, validated and evaluated frameworks, tests, infrastructure and processes, etc. HEI reports (not for systems), along with international reports and materials, and recommendations for a main study

Linguistically diverse… ArabicDutchEnglishFinish FlemishItalianJapaneseKorean Norwegian RussianSlovakSpanish Linguistically diverse… ArabicDutchEnglishFinish FlemishItalianJapaneseKorean Norwegian RussianSlovakSpanish

Is it feasible to develop frameworks and instruments to test discipline-specific learning outcomes? Test fields: Economics, Engineering Is it feasible to develop frameworks and instruments to test discipline-specific learning outcomes? Test fields: Economics, Engineering Contextual Dimension Generic Skills Economics Engineering Student Faculty Institution National Instrument architecture

Outcome specification Document analysis Consultation Synthesis, review Outcome specification Document analysis Consultation Synthesis, review Framework creation Gather existing materials Item workshops Technical review Framework mapping Adaptation, translation Verification Gather existing materials Item workshops Technical review Framework mapping Adaptation, translation Verification Item creation Qualitative testing Quantitative testing Operationalisation Qualitative testing Quantitative testing Operationalisation Instrument validation Framework and item development Tuning AHELO / QAA frameworks Curriculum documents Accreditation systems Discipline research Tuning AHELO / QAA frameworks Curriculum documents Accreditation systems Discipline research Authentic, hybrid item types ‘Above content’ reasoning Authentic, hybrid item types ‘Above content’ reasoning

Source version Translation 1 Translation 2 Final check National review Reconciliation Verification: verifier (linguist) and domain specialist Designed to maintain cross- national comparability of assessment materials A holistic, robust and flexible approach, linked with item production and validation Adaptations managed as a continuous process Native speakers of target language trained to detect specific pitfalls Economists or engineers who are speakers of target language Translation, adaptation and verification

Psychometric reviews

Student feedback for a task

Project implementation Leadership: International project management and supporting national teams Operationalisation: Preparing tests and context instruments for secure online delivery, and training makers Sampling: Engaging institutions, sampling faculty and students, and quality assurance Assessment: Supporting national training, managing testing, and managing marking quality Reporting: Compiling data products, psychometric and statistical analysis, and system, institution and stakeholder reports Evaluating: Scientific and practical feasibility, recommendations for full-scale study

Country communications

Defined engagement cycle established to support systems, institutions and students… but not faculty Designed sustainable business models for such work… but other models Assessment frameworks and test instruments developed to support multidimensional test/context instrumentation Established methods for test design, development, translation/adaptation and validation Defined operational workflow and quality control procedures required to support global testing Forming awareness of how such work is positioned globally Broad insights

Change horizons AHELO Institutional positioning Faculty improvement Policy research Learner feedback Stakeholder engagement System monitoring

Assessing higher education learning outcomes globally Professor Hamish Coates