Conceptualising and Testing Generic Skills in an International Assessment Of Higher Education Learning Outcomes Doug McCurry Australian Council for Educational Research

OECD’s Assessment of Higher Education Learning Outcomes (AHELO) Feasibility Study 17 countries 248 higher education institutions faculty final-year engineering and economics students

The Feasibility Study aimed to test what is or could be feasible in international testing of learning in higher education determine what instruments would work well in such testing provide lessons on how learning outcomes might be effectively assessed in the future

The Feasibility Study looked at outcomes in the discipline-specific knowledge and skills of economics and engineering courses the GS common to all students (such as critical thinking, analytical reasoning, problem-solving, and written communication)

Why test generic skills? The attention to GS reflects a recognition that competence means more than technical knowledge and skills a demand for an increasingly skilled workforce dissatisfaction with work readiness of some exit students increasing internationalisation and greater international competition

The generic skills debate The rejecters Cognitive performance is knowledge dependent, everything is learned in a context, transfer of learning across contexts is uncertain, and competence and expertise are knowledge rich GS should be tested within domains The accepters GS are the fluid capacities that are used to learn new things, and that are developed by learning things GS assessment should be focussed on the ability to think about something new and unfamiliar rather than focussed on what has been learned in the past To the accepters the notion of ‘domain-specific generic skills’ is meaningless Testing GS in domains is testing the domain specific not the generic

What is meant by the term generic skills? These GS are distinguished from the specific knowledge and skills needed for a particular performance what the expert knows competencies or an area in which you have to be competent a class of tasks GS are the fluid capacities used to learn new things as distinct from what has been learned in the past The assessment of general abilities is focused on aptitude to learn rather than achievement or competence

What is a generic skills test? aims to minimise the testing of prior knowledge is stimulus based, and uses generally accessible, non- specialist public domain material as stimulus aims to maximise testing of the ability to understand unfamiliar material and undertake novel thinking tasks involves a broad and balanced set of tasks involves a range of tasks, but is consistent and coherent enough to be summarised in a single score

Some other international generic skills studies Prepared for life?: how to measure cross- curricular competencies 1997 OECD Program of International Student Assessment (PISA) The OECD Definition and Selection of Competencies project (DeSeCo) IEA Composition Study 1988

The Collegiate Learning Assessment Council for Aid to Education The CLA is a stimulus-based performance assessment of analytic reasoning and evaluation problem solving writing effectiveness writing mechanics

The GS MCQ The CC GS skills tested by ACER are seen as developed rather than innate abilities do not involve specialized knowledge or skills involve integrated and holistic thinking aim at higher-order skills rather than basic skills or abstract and de-contextualised information processing involve thinking about the material world and socio- cultural issues aim to be generally accessible and test a broad and balanced range of cognitive tasks

The ACER Tradition of CC Testing The art of item writing Psychometric methods but not psychometric constructs Based on rich stimulus – specific, real-world, socio-cultural Minimising the important of prior knowledge Meaningful questions about real problems Not abstract exercises or meaningless games Unfamiliar questions and higher-order thinking

The common central core of cognitive abilities is tested by minimising the testing of prior knowledge maximising the ability to understand unfamiliar material and undertake novel thinking tasks basing testing on generally accessible, non-specialist, public domain stimulus material using a set of tasks that is broad enough to involve a complex of abilities, but coherent enough to be summarised in a single score

Categories and characteristics for the AHELO MCQ classification

International reactions to proposed material Too logico-deductive and quantitative Too much emphasis on concrete and material issues and information processing Not conceptual enough Not girl friendly enough Some problems with language and cultural references Some socio-political problems

Development of the proposed GS MCQ Removed a unit of harsh and lenient prisons, but retained in similar set on a political issue Added passages by Tolstoy and Ibsen about socio-cultural issues Added part of a speech by Ghandi with historical and political significance The 55 items used were broad, rich and intelligible from all sorts of perspectives

Adapting material to work in other languages The risk of changing material that worked in English (in Australia) for easier transition to other languages 51 of 55 items had satisfactory to good difficulty and discrimination Only 4 items misfitted in general and were removed from the analysis A small number of items were removed for the analysis in individual countries In some respects the GS MCQ were more successful than the domain-specific material

Towards an optimal generic skills test for international use Testing GS across languages and cultures through unfamiliar but generally accessible material was successful The feasibility study showed that GS MCQ test did work well across languages and cultures CRI tests could work within languages and cultures

A optimal test of GS for international use would have a number of different tasks rather than one theme to give breadth and balance integrate CRI and MCQ be focussed on thinking rather than literacy have some specific and some broad and open tasks have item specific scoring for some SAQ and global scoring for some broad and open tasks

The AHELO feasibility study warrants an international GS test that is specific in some respects and broad and open in other respects Such a test can deal with cultural and socio-political issues A very encouraging and satisfying result