1. Assessing higher education learning outcomes globally Professor Hamish Coates hamishc@unimelb.edu.au

2. Study at a glance Are valid and reliable comparisons of learning outcomes scientifically and practically feasible? Conceived ~2005, announced 06, scoping 06-09, work 10-13 An international feasibility study – not a ‘pilot’ Testing 23000 ‘bachelor degree’ students, with data from 5000 faculty, 25 national managers, 250 institutional coordinators, 17 policymakers, and hundreds of advisors and stakeholders Designed, developed, validated and evaluated frameworks, tests, infrastructure and processes, etc. HEI reports (not for systems), along with international reports and materials, and recommendations for a main study

3. Linguistically diverse… ArabicDutchEnglishFinish FlemishItalianJapaneseKorean Norwegian RussianSlovakSpanish Linguistically diverse… ArabicDutchEnglishFinish FlemishItalianJapaneseKorean Norwegian RussianSlovakSpanish

4. Is it feasible to develop frameworks and instruments to test discipline-specific learning outcomes? Test fields: Economics, Engineering Is it feasible to develop frameworks and instruments to test discipline-specific learning outcomes? Test fields: Economics, Engineering Contextual Dimension Generic Skills Economics Engineering Student Faculty Institution National Instrument architecture

5. Outcome specification Document analysis Consultation Synthesis, review Outcome specification Document analysis Consultation Synthesis, review Framework creation Gather existing materials Item workshops Technical review Framework mapping Adaptation, translation Verification Gather existing materials Item workshops Technical review Framework mapping Adaptation, translation Verification Item creation Qualitative testing Quantitative testing Operationalisation Qualitative testing Quantitative testing Operationalisation Instrument validation Framework and item development Tuning AHELO / QAA frameworks Curriculum documents Accreditation systems Discipline research Tuning AHELO / QAA frameworks Curriculum documents Accreditation systems Discipline research Authentic, hybrid item types ‘Above content’ reasoning Authentic, hybrid item types ‘Above content’ reasoning

6. Source version Translation 1 Translation 2 Final check National review Reconciliation Verification: verifier (linguist) and domain specialist Designed to maintain cross- national comparability of assessment materials A holistic, robust and flexible approach, linked with item production and validation Adaptations managed as a continuous process Native speakers of target language trained to detect specific pitfalls Economists or engineers who are speakers of target language Translation, adaptation and verification

7. Psychometric reviews

8. Student feedback for a task

9. Project implementation Leadership: International project management and supporting national teams Operationalisation: Preparing tests and context instruments for secure online delivery, and training makers Sampling: Engaging institutions, sampling faculty and students, and quality assurance Assessment: Supporting national training, managing testing, and managing marking quality Reporting: Compiling data products, psychometric and statistical analysis, and system, institution and stakeholder reports Evaluating: Scientific and practical feasibility, recommendations for full-scale study

11. Country communications

12. Defined engagement cycle established to support systems, institutions and students… but not faculty Designed sustainable business models for such work… but other models Assessment frameworks and test instruments developed to support multidimensional test/context instrumentation Established methods for test design, development, translation/adaptation and validation Defined operational workflow and quality control procedures required to support global testing Forming awareness of how such work is positioned globally Broad insights

13. Change horizons AHELO Institutional positioning Faculty improvement Policy research Learner feedback Stakeholder engagement System monitoring

14. Assessing higher education learning outcomes globally Professor Hamish Coates hamishc@unimelb.edu.au