Peer review and assessment at the ARC Monash University 7 June 2016 Presented by Professor Marian Simms ARC Executive Director

Web: arc.gov.au I Outline Current Funding Landscape NCGP update ARC Assessment Processes—role of the College of Experts

2015–16 Federal Budget—$429 billion

2015–16 Federal Budget—R&D $9.7 billion

Web: arc.gov.au I Commonwealth Investment in R&D 2015–16 (%)

Web: arc.gov.au I Selected research agencies 2003–2016 funding ($m)

Web: arc.gov.au I Focusing Government Investment Research Priorities The Australian Government has developed a set of Science and Research Priorities, they are: - soil and water - transport - cybersecurity - energy - resources - advanced manufacturing - environmental change - Health More information at

ARC NCGP % funding by institution 2008–2015

Web: arc.gov.au I Linkage fraction vs Discovery fraction

Web: arc.gov.au I NCGP funding—VIC Universities

Web: arc.gov.au I NCGP funding—Monash University

ARC NCGP funding by Fields of Research 2007–2015

Selected Fields of Research—Success Funded proposals vs unfunded for (all rounds) by 4 digit FoR 1401 Economic Theory 1402 Applied Economics 1403 Econometrics 1499 Other Economics 1501 Accounting, Auditing and Accountability 1502 Banking, Finance and Investment 1503 Business and Management 1504 Commercial Services 1505 Marketing 1506 Tourism 1507 Transportation and Freight Services 1599 Other Commerce, Management, Tourism and Services

ITRP Linkages—All rounds Higher Education Government Industry/Other

Linkage Projects—Linkages 2011–2015 Higher Education Government Industry/business Non-profit/other

LP Linkages—VIC 2011–2015 Higher Education Government Industry/business Non-profit/other

LP Linkages—Monash 2011–2015 Higher Education Government Industry/business Non-profit/other

Histogram of Rank DP12–16

Histogram of Rank DE12–16

Web: arc.gov.au I NCGP Assessment processes

Web: arc.gov.au I Overall All grants that are successful should provide exciting new outcomes and be excellent investments Decisions will align with Scheme Objectives Not all excellent proposals can achieve funding; most applicants will be disappointed

Web: arc.gov.au I Insights into grants process Consider where to apply for funding; choose a scheme. Pay attention to eligibility and ARC ‘cross scheme limits’. The scheme objectives and the selection criteria - address every one of them Choosing Fields of Research Codes—assisting the ARC choose the right assessors Record and research and research-related outputs—career interruptions/industry and/or end-user experience—the ROPE provision Apply by yourself (DP, IN, DECRA, FT and FL) or as a team member… Seek mentors on writing good grant applications, and rejoinders! Your first grant application – Write for your peers—write so that someone broadly in your field will understand your project – Write for the public—write a plain English statement Don’t over-inflate authorship claims but don’t undersell yourself either

Web: arc.gov.au I Insights into assessment process 1 1.The external (detailed) assessor—1 or 2, or up to 5 or 6 proposals 2.The ARC panel member, depending on scheme—10–120 3.The ARC selection meeting—100–900 Detailed assessors are assessing a small volume of applications. Their comments are provided to applicants but not their grades The ARC provides detailed advice to assessors regarding the attributes of good quality (i.e. helpful) and reliable assessments The ARC aims for both a good volume of assessments, around 3 per application, and for good quality assessments

Web: arc.gov.au I Insights into assessment process 2: RNTA DP17

Web: arc.gov.au I Insights into assessment process 3—the rejoinder The rejoinder is provided to the panel members assigned to your proposal, NOT to the assessors. It provides valuable additional insights into your proposal for consideration by the panel—you may clarify issues but not provide additional information. At the selection meeting it is also available to all panel members in the room at the time (i.e. those not conflicted with the proposal).

Web: arc.gov.au I Insights into assessment process 4—the panel Who: Panels are normally selected from members of the College of Experts whose expertise, including in cross- disciplinary fields, is most relevant to the scheme How: For many schemes each proposal is assigned to two panel members, a first and second carriage What: The first carriage assigns the detailed assessors, reads the proposal carefully and prepares a budget. After initial consideration they confer with the second carriage. They also have access to the rejoinder before providing scores to the ARC When: Panels meet at specified times that are advertised on the ARC website

Web: arc.gov.au I ARC Assessment Process—overview

Web: arc.gov.au I Selecting Panels The ARC recognises the need to have a flexible approach to suit volume and disciplinary spread in each scheme The number of discipline panels varies by scheme. (Funds are apportioned according to demand) For example, Discovery Projects typically has five panels: BSB (Biological Sciences and Biotechnology) EMI (Engineering, Mathematics and Information Sciences) HCA (Humanities and Creative Arts) PCE (Physics, Chemistry and Earth Sciences) SBE (Social, Behavioural and Economic Sciences) However, proposals can be assigned across two panels to ensure appropriate expertise, and assigned to a breadth of detailed reviewers Some other schemes use a single multi-disciplinary panel (e.g. Australian Laureate Fellowships, ITRP).

Web: arc.gov.au I Disciplinary Panels

Web: arc.gov.au I The selection meeting Typically all panels are addressed by the ARC CEO at the outset and reminded of the selection criteria and other key issues such as ROPE All panels selected for the same scheme, regardless of discipline, utilise the same selection criteria from the relevant funding rules A growing number of proposals are cross panel ones, i.e. the carriages are from different panels The ARC has asked applicants to provide information regarding whether their proposal is cross-disciplinary

Web: arc.gov.au I Interdisciplinary research (IDR) Interdisciplinary research (IDR) is becoming increasingly important to the national and international research effort From December 2015 the ARC has asked applicants to identify whether their research is interdisciplinary and in what way In the recent ARC Discovery round 60% of applications identified as IDR and 54% in DECRA The ARC is seeking information on IDR activity to understand the scope of interdisciplinary research in Australia, to assess it more effectively and to inform future policy development.

Web: arc.gov.au I IDR—by type

Web: arc.gov.au I Discovery Projects 17— ‘Is your project interdisciplinary?’ by 2-digit FoR code

Web: arc.gov.au I Low ranked proposals, often: Use too much technical jargon Make grandiose and implausible claims about outcomes Do not support claims of excellence or progress with evidence Relate to research areas without momentum Are weakly linked into national and international research networks; unable to demonstrate quality

Web: arc.gov.au I Low ranked proposals also: Emphasise the collection of data rather than the analysis/solution of problems Set a negative or depressive tone about the state of the subject in Australia—all critique and no substance! Contain a high rate of spelling and grammatical errors Are badly structured and difficult to follow

Web: arc.gov.au I Considering Research Impact Research Impact means the demonstrable contribution that research makes to the economy, society, culture, national security, public policy or services, health, the environment, or quality of life, beyond contributions to academia. It is included in objectives and selection criteria The ARC is seeking better communication about research outcomes

Web: arc.gov.au I Tips for new players! (1) ROPE: success is important. The ARC is looking for the best people to conduct research and capacity building in the future Collaborations, linkages, networks: the whole must be greater than the sum of the parts

Web: arc.gov.au I Tips for new players! (2) Objectives of the scheme: think carefully about the objectives and how you address them Research programme: well thought out and developed – workshopping with relevant parties

Thank You