Australia’s Research Assessment Arthur Sale Emeritus Professor of Computer Science University of Tasmania

History : RQF  ERA Australia had embarked on planning a national assessment of the research of its 39 universities – called the Research Quality Framework (RQF) In November 2007, the Federal Government changed as the result of an election. The incoming Government had promised a lower cost, simpler evaluation scheme – the ERA, announced February 2008.

UK Background - RAE The UK has long operated a Research Assessment Exercise (RAE) which used extensive panel assessment, occurred at six- year intervals, and which determined the research funding of universities for six years. The very last RAE was completed in The UK government announced a change of policy prior to that round of assessment.

UK Background - REF The UK will replace the RAE by a mostly metric- based process, the Research Excellence Framework (REF). ‘Assessment will combine quantitative indicators - including bibliometric indicators wherever these are appropriate - and expert review. Which of these elements are employed, and the balance between them, will vary as appropriate to each subject. The economic and social impact of research, as well as impact upon public policy, will also contribute to the overall assessment of quality.’ bibliometric indicatorseconomic and social impact of research

Ironic Fun! RAE  REF RQF  ERA

Excellence in Research for Australia The RQF was based solidly on expert assessment of discipline groups like the RAE: expensive ERA uses the experience of the UK’s RAE but does not copy it, and is biased towards metrics, with concessions to panel assessment where essential. ERA is managed by the Australian Research Council (ARC) and reports to the Minister for Innovation, Industry, Science & Research.

Rolling assessment In another major break with the RAE/REF, the ERA assessment will be implemented in a rolling stream two discipline clusters at a time, rather than all at once in a snapshot year. Also the Minister has stated that assessment will not lead to direct financial outcomes for universities, though this remains to be tested. Instead it will create data that could lead to ‘league tables’.

Background Scholarly research is divided into eight discipline clusters (see next slide). The groups are defined by the first four digits of the six digit Australian and New Zealand Standard Research Classification (ANZSRC). All research outputs are classified by this scheme and have been for many years (under an older version). All Australian repositories use this scheme (not LC or Dewey).

Discipline clusters 1 Physical, Chemical and Earth Sciences (PCE) – Humanities and Creative Arts (HCA) – Engineering and Environmental Sciences (EE) – 2011? 4 Social, Behavioural and Economic Sciences (SBE) 5 Mathematics, Information and Communication Sciences (MIC) – 2012? 6 Biological Sciences and Biotechnology (BSB) 7 Biomedical and Clinical Research (BCR) – 2013? 8 Public and Allied Health and Health Services (PAHHS)

ANZSRC (sample) Preface Fields of Research DIVISION 01 MATHEMATICAL SCIENCES GROUP 0101 PURE MATHEMATICS GROUP 0102 APPLIED MATHEMATICS GROUP 0103 NUMERICAL AND COMPUTATIONAL MATHEMATICS GROUP 0104 STATISTICS GROUP 0105 MATHEMATICAL PHYSICS GROUP 0199 OTHER MATH SCIENCES DIVISION 02 PHYSICAL SCIENCES GROUP 0201 ASTRONOMICAL AND SPACE SCIENCES GROUP 0202 ATOMIC, MOLECULAR, NUCLEAR, PARTICLE & PLASMA PHYSICS GROUP 0203 CLASSICAL PHYSICS GROUP 0204 CONDENSED MATTER PHYSICS GROUP 0205 OPTICAL PHYSICS GROUP 0206 QUANTUM PHYSICS GROUP 0299 OTHER PHYSICAL SCIENCES DIVISION 03 CHEMICAL SCIENCES GROUP 0301 ANALYTICAL CHEMISTRY GROUP 0302 INORGANIC CHEMISTRY GROUP 0303 MACROMOLECULAR AND MATERIALS CHEMISTRY GROUP 0304 MEDICINAL AND BIOMOLECULAR CHEMISTRY GROUP 0305 ORGANIC CHEMISTRY GROUP 0306 PHYSICAL CHEMISTRY GROUP 0307 THEORETICAL AND COMPUTATIONAL CHEMISTRY GROUP 0399 OTHER CHEMICAL SCIENCEs DIVISION 04 EARTH SCIENCES GROUP 0401 ATMOSPHERIC SCIENCES GROUP 0402 GEOCHEMISTRY GROUP 0403 GEOLOGY GROUP 0404 GEOPHYSICS GROUP 0405 OCEANOGRAPHY GROUP 0406 PHYSICAL GEOGRAPHY AND ENVIRONMENTAL GEOSCIENCE GROUP 0501 ECOLOGICAL APPLICATIONS

Rolling implementation The first two clusters to be assessed are – PCE = Physical, Chemical and Earth Sciences – HCA = Humanities and the Creative Arts (for example literature, history, architecture, drama and music) July 2009 will see a trial evaluation, with the ‘real ERA’ to commence in It is expected that two clusters will be assessed per year giving a four year cycle of evaluation.

2009 Trial PCE and HCA are extremes. PCE will be evaluated mainly by bibliometrics, whereas HCA won’t use bibliometrics at all but will use panel evaluation. Both will use applied research metrics and staffing data. This will provide a basic for future evaluations which will probably be mixed and more complex.

PCE Submission Each university will provide an XML document describing its publication, grant and application record for the cluster. This will include a citation of each publication and its Scopus ID. The ARC will look up each publication and compile metrics for the university’s cluster (the SEER system). The panel will review the metrics and argue around the margins to arrive at an assessment.

Important Note Evaluations are not by a research group, but by all the university’s research output in that discipline cluster. Universities must also report their total FTE staff count in that cluster, and the researcher count (which includes adjunct and honorary staff). These are also taken into account in assessing quality. Research by adjunct staff is also counted.

Databases After a tender process, Scopus was selected as the journal citation data supplier for PCE. ARC has also created a public tier list of 21,000 journals, ranked by the various professional societies. The four tiers are – A*top 5% (top international standard) – Anext 15% (excellent, but not in the top tier) – Bnext 30% (good, perhaps local, often PhD outlets) – Cremaining 50% (the rest) These will also be used to create metrics.

Ranked Tier List (sample)

HCA Submission Like PCE, the university will provide an XML submission, but each publication or piece of research output will be linked (via URL) to a document in the university’s repository. The panel will evaluate the research outputs on SEER from face evidence and look at a nominal 20% of the documents. This will form part of the HCA assessment.

System to Evaluate the Excellence of Research The ARC’s SEER software provides the central glue to make this all work. It will automatically extract data from the XML submissions and databases and create summary metric data for review panels. Expert reviewers will access the university repositories (eg HCA) through SEER which (a)will anonymize them, and (b)has permission to look at restricted documents.

Sampled periods It is expected that for the 2010 ERA evaluations the census date and the dates for the reference periods will move forward one year compared to the trial. That is, the staff census date would be 31 March 2009, and the reference periods would be as follows: Research outputs - 1 January 2003–31 December 2008 (6 years) Citation analysis - 1 January 2003–1 March 2010 (6 years) Research income - 1 January 2006–31 December 2008 (3 years) Applied measures - 1 January 2006–31 December 2008 (3 years)

Summary ERA is planned to be low-cost and rolling. It is adapted to clusters, and within them to disciplines. Extensive consultation is taking place It will reshape Australian research I am sure that Australia is willing to share its experiences with any other country. Read the documentation, watch the space...

By now you probably deserve a picture of me. I am really sorry not to have been with you. Best wishes, Arthur

Questions?

© Copyright 2009 Arthur Sale All rights reserved Arthur Sale asserts the right to be recognized as author of this work Contact:

Links ERA (Aus) ERA ANZSRC REF (UK) REF