1. Excellence in Research for Australia 2015 Associate Law Deans National Meeting The University of Western Australia November 2015 Sarah Howard Director, Research Evaluation Australian Research Council

2. Introduction ERA 2015—where are we up to? ERA 2015 National Report ERA 2015 methodology Peer review in ERA ERA 2015 outcomes and data

3. ERA 2015—where are we up to? Currently finalising evaluation Preparing State of Australian University Research: ERA National Report 2015-16

4. ERA 2015 National Report WHAT? State of Australian University Research: Volume 1 ERA National Report 2015–16  Includes additional analysis of 10 year longitudinal dataset (2003–2013)  ERA 2010, ERA 2012, ERA 2015 comparisons PLUS—additional volumes to come… applied research, gender, open access WHEN? Early December….

5. ERA 2015 methodology

6. ERA Documentation Range of information about ERA development and ERA 2015 processes ERA 2015 documentation includes:  Submission Guidelines  Discipline Matrix  Notifications and FAQs  2015 REC members Background information includes:  National Reports  ERA development documentation www.arc.gov.au > ERA www.arc.gov.au

7. Useful Terminology to start … System to Evaluate Excellence of Research (SEER)—ERA IT system Research Evaluation Committee (REC)— committee of experts undertaking evaluations ANZSRC—Australia and New Zealand Standard Research Classification Field of Research (FoR)—ANZSRC research classification (e.g. 2101 Archaeology) Unit of Evaluation (UoE)—ERA unit of assessment for an institution by FoR (e.g. ANU 2101) Dashboard—suite of ERA indicators

8. ERA 2015 The ERA objectives are unchanged: 1.Establish an evaluation framework that gives government, industry, business and the wider community assurance of the excellence of research conducted in Australian higher education institutions. 2.Provide a national stock-take of discipline level areas of research strength and areas where there is opportunity for development in Australian higher education institutions. 3.Identify excellence across the full spectrum of research performance. 4.Identify emerging research areas and opportunities for further development and. 5.Allow for comparisons of research in Australia, nationally and internationally, for all discipline areas.

9. ERA 2015 Reference Periods Staff census date: 31 March 2014 Data typeReference periodYears Research outputs1 Jan 2008–31 Dec 20136 Research income1 Jan 2011–31 Dec 20133 Applied measures1 Jan 2011–31 Dec 20133 Esteem measures1 Jan 2011–31 Dec 20133

10. Discipline Matrix ERA 2015 Peer Review (30% of outputs to be nominated by output type) HERDC Research Income (Categories 1-4) Esteem Measures FoRCodeFoR Title Books Book Chapters Journal articles Conference Publications Non-traditional outputs by type Editor Prestigious Works of Reference Membership of learned academy Category 1 research fellowships Other NTRO Research Report for External Body 18Law and Legal StudiesYes 1801LawYes 1802Maori LawYes 1899 Other Law and Legal StudiesYes ERA 2015 Discipline Matrix

11. Evaluation Overview Note - There are no weightings Research Evaluation Committees Citation Analysis or Peer Review Volume and ActivityResearch Income Applied MeasuresEsteem UoE rating by FoR UoE rating by FoR Access to Repositories for Peer Review Disciplines Peer Reviewers

12. Stages of Evaluation Stage 1 REC Members evaluate assigned material and record preliminary evaluations in SEER Stage 2A REC Members evaluate assigned material and record preliminary evaluations in SEER Stage 2B REC Members are given access to the preliminary evaluation outcomes for 2 digit UoEs from other REC members and Peer Reviewers’ evaluations for moderation Stage 2C REC Members are given view only to access to moderated 2 & 4 digit evaluations to prepare for Stage 3 meeting Stage 3 REC Members meet to finalise all UoEs Stage 1 ERA Peer Reviewers evaluate assigned material and record evaluations in SEER REC Member Peer Reviewer

13. ERA 2015 REC Member and Peer Reviewer Recruitment Research Evaluation Committee (REC) Membership 8 REC Chairs announced in November—to ensure ERA evaluations are of the highest standard Full list of RECs members can be found on the ARC website Peer Reviewers The call for peer reviewer nominations closed on 26 November 2014 ~ 1300 peer reviewers—even better than 2012! ~ 1100 of these were assigned material for evaluation ~ 1000 peer reviewers submitted reports

14. Recruitment of REC members Universities invited to nominate REC members in mid–2014 Maximum of 50 nominations per university—10% of nominations from other countries Nominations against published criteria:  research excellence as well as a sound understanding of the importance of research  broad discipline expertise (interdisciplinary experience is desirable)  professional and academic standing  interpersonal and team skills Database of 700 researchers with broad range of expertise Nominees not selected were invited to participate as peer reviewers (in the ERA peer review disciplines)

15. Recruitment of Peer Reviewers Peer review is conducted by REC members in concert with external peer reviewers who are appointed to  extend the disciplinary expertise of the RECs and to  share the workload Peer reviewers for ERA 2015 were required for the following 2 digit FoRs: 08, 12–16, and 18–22 (as well as 0101, 1005 and 1006) All universities were invited to nominate peer reviewers (national and international) Peer reviewers are nominated by six-digit expertise to facilitate assignment

16. Peer Review in ERA 2015— Overview  Universities nominate a ‘representative’ sample 30% of total outputs for each assessable unit (e.g. ANU1801)  The sample is lodged in a digital repository and made available to ERA reviewers through SEER  REC members conduct peer review of the nominated sample  REC members also assign from a pool of peer reviewers to extend the expertise available and assist with workload  Peer reviewers use a structured report form  The final rating = all reviewer assessments + dashboard information

17. Peer Reviewer Assignment Every assessable unit is assigned three REC members who conduct peer review Every assessable unit is also assigned a minimum of two peer reviewers Assignment to peer reviewers is based on:  target maximum of 50 outputs for each reviewer  target maximum of 6 units for each reviewer  no reviewer should have only one assigned unit*  no unit has only one peer reviewer  notionally enough reviewers are assigned to mean all outputs in the unit will be read by at least one reviewer *This means that peer reviewers have a target of no more than 25 outputs in any one unit. In a unit with 200 outputs a minimum of 8 reviewers have to be assigned.

18. Peer Review Criteria Criterion—Approach What contextual information should the RECs know about the common practice in the discipline and the sample of outputs reviewed? Are the methodologies clearly reported? Are the methodologies appropriate for the research questions or not? If not, what are limitations/effects? Are the venues/outlets appropriate for reporting the research? Again, there might be reference to common practice for the discipline. What is the quality of the venues/outlets? Criterion—Contribution Contribution of the group of outputs reviewed to the field and/or practice – nationally and/or internationally Originality and degree of innovation in the research approach or research questions Theoretical and conceptual rigour Depth of discussion and analysis Contribution to the further development of knowledge and understanding

19. ERA 2015 Peer Reviewer Feedback

20. Survey of Peer Reviewers Peer reviewers invited to complete an online survey after they had finished their assessments 660 of approximately 1300 reviewers returned completed survey Peer reviewers were also invited to submit comment on any aspect of their engagement in ERA 2015 Most additional comment was already captured in the structured questionnaire and so there were no additional common themes

21. Were the instruction were clear?

22. Was the assigned workload manageable?

23. Main issues raised by Peer Reviewers Workload Allow supplementary metrics (e.g. Google Scholar, journal rankings etc.) Universities provide contextual/research statement for each 30% sample Standardise user-interface for university repositories Recognition of the contribution of peer reviewers (e.g. NHMRC example) Feedback—especially for ‘first time’ reviewers

24. Can we talk about ERA 2015 results? Not yet….

25. ERA 2015—some feedback Inclusion of text books—to be included these must meet the definition of research—we saw a number of these Watch out for predatory publishers which don’t peer review manuscripts—we saw a number of these Make sure the coding of outputs reflects the content of the output—know your ANZSRC codes Watch out for ‘recycled’ content—a revision or subsequent version must include substantial new research content

26. Know your ANZSRC codes 18 Law and Legal Studies 1801 Law 1802 Maori Law 1899 Other Law and Legal Studies Exclusions: a)Criminology, including policing and correctional theory, is included in Group 1602 Criminology. Group 1602 Criminology b)Legal ethics and human rights and justice issues are included in Group 2201 Applied Ethics. Group 2201 Applied Ethics c)History and philosophy of law and justice is included in Group 2202 History and Philosophy of Specific Fields.Group 2202 History and Philosophy of Specific Fields

27. Know your ANZSRC codes 1801 LAW This group covers law. It includes: legal institutions; legal theory and practice; and litigation, adjudication and dispute resolution. This group has twenty-seven fields: 180101 Aboriginal and Torres Strait Islander Law 180102 Access to Justice 180103 Administrative Law 180104 Civil Law and Procedure 180105 Commercial and Contract Law 180106 Comparative Law 180107 Conflict of Laws (Private International Law) 180108 Constitutional Law 180109 Corporations and Associations Law 180110 Criminal Law and Procedure 180111 Environmental and Natural Resources Law 180112 Equity and Trusts Law 180113 Family Law 180114 Human Rights Law 180115 Intellectual Property Law 180116 International Law (excl. International Trade Law) 180117 International Trade Law 180118 Labour Law 180119 Law and Society 180120 Legal Institutions (incl. Courts and Justice Systems) 180121 Legal Practice, Lawyering and the Legal Profession 180122 Legal Theory, Jurisprudence and Legal Interpretation 180123 Litigation, Adjudication and Dispute Resolution 180124 Property Law (excl. Intellectual Property Law) 180125 Taxation Law 180126 Tort Law 180199 Law not elsewhere classified Exclusions: a)Criminology, including policing and correctional theory, is included in Group 1602 Criminology. Group 1602 Criminology b)Legal ethics and human rights and justice issues are included in Group 2201 Applied Ethics. Group 2201 Applied Ethics c)History and philosophy of law and justice is included in Group 2202 History and Philosophy of Specific Fields.Group 2202 History and Philosophy of Specific Fields

28. Know your ANZSRC codes 1802 MAORI LAW This group covers Maori Law. This group has five fields: 180201 Nga Tikanga Maori (Maori Customary Law) 180202 Te Maori Whakahaere Rauemi (Maori Resource Law)) 180203 Te Tiriti o Waitangi (The Treaty of Waitangi) 180204 Te Ture Whenua (Maori Land Law) 180299 Maori Law not elsewhere classified

29. Know your ANZSRC codes 1899 OTHER LAW AND LEGAL STUDIES *DOES YOUR OUTPUT REALLY BELONG HERE? This group covers law and legal studies not elsewhere classified. This group has one field: 189999 Law and Legal Studies not elsewhere classified Exclusions: a)Criminology, including policing and correctional theory, is included in Group 1602 Criminology. Group 1602 Criminology b)Legal ethics and human rights and justice issues are included in Group 2201 Applied Ethics. Group 2201 Applied Ethics c)History and philosophy of law and justice is included in Group 2202 History and Philosophy of Specific Fields.Group 2202 History and Philosophy of Specific Fields

30. ERA-HERDC Alignment Update

31. ERA-HERDC Alignment Background PhillipsKPA Review of Reporting Requirements for Universities Department of Education and the ARC working together A single research data collection Part of Government’s Reducing Red Tape agenda Aim A more efficient streamlined process for the collection of data to inform RBG and evaluate excellence Potential Efficiencies University admin staff and IT systems only need to cater for one data set Reduces opportunities for error and misunderstanding—enhancing integrity, transparency and utility of data Auditing requirements in relation to the data will be streamlined

32. ERA-HERDC Alignment cont. Consultation paper on options for alignment (8 December–13 February) Sector working group (3 July–21 September) Draft specifications completed Next steps – sector consultation on draft specifications, when…?

33. Thank you