The Assessment of Higher Education Learning Outcomes (AHELO) OECD’s Study of the Scientific and Practical Feasibility of Assessing Baccalaureate-Level Student Learning Outcomes Across Nations My assignment for this session: 1.Help set the context for discussion of state/system/SHEEO roles in learning assessment. 2.Use AHELO — a project that SHEEO and SHEEOs in three states have been involved in for the past three years—as example of evolving needs, tools, challenges and opportunities for assessment of learning in higher education. 3.Prompt reflection and perhaps additional steps by SHEEO and SHEEOs in shaping the roles and future of learning assessment in higher education, including international involvement.

The Context for Learning Assessment in Higher Education Is Complex Decades of evolution/development in learning assessment with distinct differences between K-12 and higher education in applications and acceptance, with neither sector yet satisfied Several prior episodes of state and SHEEO involvement in higher education learning assessment to stimulate/support primarily institution-based assessment Frequently and perhaps unavoidably mixed and overlapping assessment purposes – Formative and summative purposes – Level of analysis (student, institution, sector or systems) and whose needs are intended to be served (states, nations, governance structures) ? – Mismatches between assessment purposes and instruments Challenge of encouraging (or contradiction between) “faculty ownership” of authentic, curricular-embedded assessment and the need/pressure for external and face validity Lack of clearly defined and agreed-upon learning expectations or outcomes at the program, institutional and degree levels in higher education —Is assessment a useful tool for prompting such clarity of purpose? Lack of investment in state-of-the-art assessment instruments and approaches, and perception that assessments are simple, inexpensive, unnecessary add-ons.

What is AHELO—Assessment of Higher Education Learning Outcomes Following five years of multi-national planning and resourcing, AHELO is a closely examined study of the scientific and practical feasibility of defining and assessing degree-related learning outcomes across nations Initiated, organized and led by OECD, but not at all comparable to or linked with PISA and other studies Instrument development, assessment administration, and data analysis and reporting by international consortium led by Australian Council for Educational Research (ACER) Funded by foundations and participating country contributions, but housed in the institutionally-focused OECD unit IMHE (ministeries vs. institutions) OECD Feasibility Study of BA-level learning outcomes in Generic Skills, Economics and Civil Engineering

Why an AHELO? For what purposes? Official—2006 decision of Council of Education Ministers to examine available options, instruments, practicality, and national/institutional needs Practical—broad recognition of increasingly global context for higher education students, programs and institutions Substantive—growing focus on the competitive/economic roles of higher education for nations and students More consequentially and theoretically, AHELO less a test of “can it be done” than a question of “what purposes can and should it serve” relative to: – Changing student needs? – New international quality expectations and assurances? – Changing global workplace expectations? – Expanding institutional operations and ambitions?

Participants

AHELO: Feasibility Study Design Discipline strand in Economics Subject content and competence Application to real world problems Effective application of relevant data and quantitative methods Communication Discipline strand in Engineering (Civil) Engineering generic skills Basic and Engineering sciences Engineering analysis and design Engineering application and practice Generic Skills strand Above content (subject-matter) skills and competencies Baccalaureate-level preparation for global education and economic environment Performance task component-- assessment of actual application of critical thinking, mathematical reasoning, data interpretation and application, etc. Value-added /“learning gain” strand Research-based Indicative of growing interest and underlying evaluative potential Inadequately resourced

AHELO Instrument Design and Development Generic Skills Translation/adaptation of two CLA performance tasks Additional 25 selected response items in five clusters and multiple domains (from ACER in Australia) Limited instrument validation—one cognitive lab per country, 5+ students 120 minutes plus short student survey Entirely on-line delivery/data collection Institutions administered/proctored testing using international protocols Student incentives for participation defined and reported by institutions Scoring by participating nations using international rubrics /limited QA Separate on-line faculty survey and institutional contextual data collection Disciplinary Strands—Economics and Engineering Internationally developed instruments with international advisory group One performance task (application in practice) plus discipline content selected response items—45 in Economics and 30 in Engineering Validation through focus groups at participating institutions 90 minutes plus student survey Entirely on-line delivery/data collection Programs or institutions administered tests using international protocols Self-incentives of students and discipline self- interest Scoring by in-country disciplinary faculty using international rubrics Separate on-line faculty survey and institutional contextual data collection

1.The U.S. has a large, diverse and changing marketplace of colleges and universities 2.Essentially all institutions engage in periodic learning assessment, generally using a variety of approaches or instruments, and at program as well as institutional levels 3.Broad interest in the globalization of higher education and in the clarification of learning outcomes, but skepticism about measurement and the future of AHELO

What Did the Feasibility Study Accomplish? How successfully? Largely successful instrument development, translation and context adaptation in economics and engineering More challenging first international use of generic skills performance tasks (new to many countries) that were not easily translated and adapted to non-American contexts, not matched with selected-response (MCQ) questions, and not within clear, agreed upon assessment framework Web-based assessment administration on secure international platforms experienced few technical failures, but encountered linguistic differences and technical constraints (passage length, keyboarding patterns and practices, etc.) Successful completion of fieldwork in 17 nations, 25 different strands, nearly 250 institutions and 23,000 students Scoring of assessment results by participating nations using international guidelines and rubrics, but very limited quality controls Extensive but still incomplete data documentation and analysis—much more attention needs to be given to what institutions and sponsoring governments get back from participation (SHEEO and U.S. sponsoring an informal meeting on July to help meet this need)

Questions for SHEEO Discussion Raised by the Example of AHELO 1.Why get involved in this type of comparative assessment? 2.What does it take for such assessments to be worth the effort and costs? 3.Based on what was learned during and from the AHELO Feasibility Study, what purposes could such an assessment serve internationally? In and for the U.S.? For SHEEO/SHEEOs and states?