1. Dr Sarah Richardson Senior Research Fellow, Higher Education Australian Council for Educational Research sarah.richardson@acer.edu.au

2. Eternal challenges How do we measure quality in teaching and learning? How effective are institutions at educating students? What knowledge and skills do students possess at the end of their degrees? How do students’ learning outcomes compare to international benchmarks?

3. Final-year Bachelor degree students To determine a robust approach to measuring learning outcomes in ways that are valid across cultures and languages, and across the diversity of institutional settings and missions To give institutions data on student learning outcomes which they can use to improve quality in teaching and learning

4. How do the learning outcomes of our students measure up against international benchmarks? Benchmarking against like institutions Students – demographics, time usage, performance, motivation, plans Teaching – curriculum structure, class sizes, staff contact, feedback, industry experience Institutional characteristics – size, location, funding, mission, student characteristics To give institutions data on student learning outcomes which they can use to improve quality in teaching and learning

5. AHELO: international participation – 17 countries (including Egypt, Kuwait and Abu Dhabi) and 12 languages (including Arabic)

6. Is it feasible to develop frameworks and instruments to test discipline-specific learning outcomes? Test fields: Economics, Engineering Is it feasible to develop frameworks and instruments to test discipline-specific learning outcomes? Test fields: Economics, Engineering 23000 students 5000 faculty Generic Skills Economics Civil Engineering 250 institutions

7. International agreement Expert Groups – Engineering and Economics Domain parameters / conceptual structure Measurement priorities – real world Review and map draft assessment materials Scoring guide – desired responses Structure reporting and review results Generic Skills – different approach used

8. Economics learning outcomes subject knowledge and understanding subject knowledge and its application to real world problems ability to make effective use of relevant data and quantitative methods ability to communicate to specialists and non- specialists ability to acquire independent learning skills

9. Assessment instruments Combine pre-existing / new materials International collaboration / resources Adapted / created for assessment framework Constructed response and MCQs Pilot & qualitative feedback Localisation – equivalency across languages Online delivery

10. Engineering

11. Engaging participants Countries / institutions / faculty / students What’s in it for me? Marketing and awareness raising Cultural variations Sample / population Authority / persuasion Incentives and inducements Intrinsic motivation

12. Analysis and reporting Psychometric analysis How did the test items function? What was the spread of difficulty? Did difficulty vary according to student and institution characteristics? Reporting – international, national and institutional levels

13. Using results to enhance teaching and learning Performance by student group Which students need greatest support? Performance by competency Which aspects of teaching need enhancing? Performance by country What is missing in the curriculum?

14. Insights from AHELO Measure learning outcomes internationally? International agreement on skills and knowledge? Assessment materials to suit diverse contexts? Maintain equivalence in translation? Engage institutions, faculty and students? Online delivery in different languages/alphabets? Useful benchmark reports without identification? AHELO Feasible?

15. Learning Employment Curricula Faculty Improve Quality Selection Diagnostic Progression Outcomes