The Research Excellence Framework Data and Audit May 2012.

Slides:



Advertisements
Similar presentations
GSOE Impact Workshop Impact and the REF 19 th May 2010 Lesley Dinsdale.
Advertisements

Research Excellence Framework Jane Boggan Planning Division Research Staff Forum - January 2010.
Main Panel A: Subpanels and Chairs A1: Clinical Medicine - Christopher Day, Newcastle University A2: Public Health, Health services and Primary Care -
Research Excellence Framework and equalities Belfast 29 November 2011 Ellen Pugh Senior Policy Adviser, ECU.
REF2014 HODOMS Birmingham 8 th April Ann Dowling: Chairman of REF Main Panel B John Toland: Chairman of REF Sub-Panel B10: Mathematical Sciences.
Long live the REF!.  The RAE looks at three main areas: ◦ Outputs ◦ Environment ◦ Esteem  We are used evaluations of Environment and Esteem being “informed”
Research Supervisor Training Programme Obligations of the Supervisor.
Supporting & promoting Equality & Diversity through REF Dianne Berry, Chair REF E&D Advisory Panel Ellen Pugh, Senior Policy Officer ECU.
The Research Excellence Framework Assessment framework, guidance on submissions and panel criteria.
Guidance on submissions Chris Taylor, Deputy REF Manager Graeme Rosenberg, REF Manager.
SQA School Co-ordinator Conference June 2014 Results Services.
These slides have been produced by the REF team, and were last updated on 3 September 2011 They provide a summary of the assessment framework and guidance.
Communicating the outcomes of the 2008 Research Assessment Exercise A presentation to press officers in universities and colleges. Philip Walker, HEFCE.
The Research Assessment Exercise in the United Kingdom Paul Hubbard International colloquium “Ranking and Research Assessment in Higher Education” 13 December.
Research at York Presentation to Council Alastair Fitter Pro-Vice-Chancellor, Research.
CMP3265 – Professional Issues and Research Methods Research Proposals: n Aims and objectives, Method, Evaluation n Yesterday – aims and objectives: clear,
HRB Webinar Health Research Awards Content Objective of the call Scope and Panels Principal Investigator Response to peer-reviewers (rebuttal) Some.
Achieving and Demonstrating Research Impact John Scott.
REF2014 – results and the way forward SSHP Meeting 12 March 2015.
Orvill Adams, Orvill Adams & Associates B.V. Orvill Adams Orvill Adams & Associates B.V. Measuring the Products of Medical Education.
External Examiners’ Briefing Day Assessment Policy Tuesday 6 th January 2015.
Demonstrating research impact in the REF Graeme Rosenberg REF Manager
The Research Excellence Framework. Purpose of REF The REF replaces the RAE as the UK-wide framework for assessing research in all disciplines. Its purpose.
The Research Excellence Framework. Presentation outline The REF assessment framework and guidance on submissions: - Overview - Staff - Outputs - Impact.
Consultation on panel criteria and working methods.
REF Information Session August Research Excellence Framework (REF)
The UK Experience of Quality Assurance in Research and Doctoral Education Dr Robin Humphrey Director of Research Postgraduate Training Faculty of Humanities.
Workshop on Implementing Audit Quality Practices March 2006 Building Quality into the Financial Audit Process The NAO’s experience Gareth Caller.
Beyond the RAE: New methods to assess research quality July 2008.
A month in the life of a university bibliometrician Dr Ian Rowlands University of Leicester, UK.
The REF assessment framework and guidance on submissions Linda Tiller, HEFCW 16 September 2011.
S7: Audit Planning. Session Objectives To explain the need for planning To explain the need for planning To outline the essential elements of planning.
The Research Excellence Framework Briefing event for REF institutional contacts Graeme Rosenberg REF Manager.
The Research Excellence Framework Briefing events for HEI contacts 21 May: Glasgow 23 May: Manchester 24 May: London 28 May: Cardiff 31 May: London.
Research Assessment Exercise RAE Dr Gary Beauchamp Director of Research School of Education.
Page 1 RESEARCH EXCELLENCE FRAMEWORK : RESEARCH IMPACT ASESSMENT LESSONS FROM THE PILOT EXERCISE Professor John Marshall Director Academic Research Development.
Audit Planning. Session Objectives To explain the need for planning To outline the essential elements of planning process To finalise the audit approach.
The Research Excellence Framework Expert Advisory Groups round 1 meetings February 2009 Paul Hubbard Head of Research Policy.
Chair: Professor Dame Ann Dowling Sub-panel Chairs: Panel Advisers: SP07: Professor David Price Dr Karen Ness SP08: Professor Richard Catlow Ms Lesley.
Professor Andrew Wathey Vice-Chancellor and Chief Executive Northumbria University.
Research Quality Framework Presentation to APSR - ARROW - Repository Market Day 4 May 2007 Sandra Fox Department of Education Science and Training.
Main Panel D Criteria and Working Methods Main Panel D covers: Area Studies Modern Languages and Linguistics English Language and Literature History Classics.
The Research Excellence Framework Impact: the need for evidence Professor Caroline Strange 22 June 2011.
The REF assessment framework (updated 23 May 2011)
Delivering Strength Across the Piece David Sweeney Director, Research, Education and Knowledge Exchange HEPI, Royal Society 31 March 2015.
Main Panel A Criteria and Working Methods Cardiff School of Biosciences Ole H Petersen Chair.
REF 2014 Briefing Professor Edmund Burke Dr John Rogers May 2013.
Understanding Newspaper League Tables Nov 2015 Katarina Thomson Planning and Statistics.
Research Fellowships. Overview Introduction Why apply for a fellowship Finding the right fellowship The application process Assessment criteria for funding.
ACADEMIC PROMOTIONS Promotions Criteria Please note, these slides only contain a summary of the promotions information – full details can be found.
Research Excellence Framework 2014 Michelle Double Hyacinth Gale Sita Popat Edward Spiers Research and Innovation Support Conference.
Research Publications Service & Research Excellence Framework 2014 John Nicholas.
The Research Excellence Framework Assessment framework and guidance on submissions Graeme Rosenberg, REF Manager.
ELIGIBILITY CRITERIA (with notes)
Towards REF 2020 What we know and think we know about the next Research Excellence Framework Dr. Tim Brooks, Research Policy & REF Manager, RDCS Anglia.
Phil Quirke RAE 2008 & REF 2014 panels
Impact and the REF Tweet #rfringe17
REF 2021 Briefing 25 January 2018.
REF 2021 What we know and thought we knew, in preparation for the next Research Excellence Framework Dr. Tim Brooks, Research Policy & REF Manager, RDCS.
Law Sub-panel Generic Feedback - Impact
These slides have been produced by the REF team, and were last updated on 30 January 2012 They provide a summary of the assessment framework and guidance.
REF 2021 Briefing Consultation on the draft guidance
These slides have been produced by the REF team, and were last updated on 30 January 2012 They provide a summary of the assessment framework and guidance.
One year on: developments since Duxford 2016
REF 2021 & ECRs: policy & planning in an uncertain landscape
Research Update GERI May 2010.
Towards Excellence in Research: Achievements and Visions of
ELIGIBILITY CRITERIA FOR FULL-TIME UNDERGRADUATE AND PGCE STUDENTS (with notes) Student Finance is available so that students can make the most of their.
Subject Pilot Teaching Excellence and Student Outcomes Framework (TEF)
REF and research funding update
Presentation transcript:

The Research Excellence Framework Data and Audit May 2012

Presentation outline Citation data The submission system Contextual data REF 4 data Research doctoral degrees awarded Research income data Returning data Audit and data verification procedures

Citation data “Those panels using citation data will do so within the framework set out in ‘guidance on submissions’ (paragraphs ). In particular, they will consider the number of times an output has been cited as additional information about the academic significance of submitted outputs. Panels will continue to rely on expert review as the primary means of assessing outputs, in order to reach rounded judgements about the full range of assessment criteria (‘originality, significance and rigour’).” (Panel Criteria and Working Methods, Part 1 paragraph 51)

Citation data Sub-panel 1: Clinical Medicine Sub-panel 2: Public Health, Health Services and Primary Care Sub-panel 3: Allied Health Professions, Dentistry, Nursing and Pharmacy Sub-panel 4: Psychology, Psychiatry and Neuroscience Sub-panel 5: Biological Sciences Sub-panel 6: Agriculture, Veterinary, and Food Science Sub panel 7: Earth Systems and Environmental Sciences Sub-panel 8: Chemistry Sub-panel 9: Physics Sub-panel 11: Computer Science and Informatics Sub-panel 18: Economics and Econometrics Will only be used by the following sub-panels:

Obtaining Citation data Items matched to Scopus via the submission system Via the web interface When entering individual outputs For whole submissions On bulk import Support from Elsevier available

Obtaining Citation data

Support from Scopus Elsevier will provide technical support on the matching process during the submissions phase Initial means of support will be by We will publish further details of support arrangements before the submissions period

Contextual citation data Sub-panels using citation data will also receive contextual data about citation rates to help inform their interpretation of the citation counts.

Contextual citation data Scopus have assigned journals to ~350 subject categories Contextual data will be provided based on these subject categories, for each year The average number of times items were cited The number of citations required to be in the top 1%, 5%, 10% and 25% of papers in that category The data will also be provided at the level of UOA by combining subject categories, after the submission system closes. Data will be provided for the years 2008 to 2012

Contextual citation data (2) HEIs will be given the same data when the submissions system opens, except: Data will be provided for the years 2007 to 2011 UOA level data cannot be provided Unable to assign Scopus subject categories to UOAs until all REF data has been collected. Data will be available as an Excel workbook to download via the submission system. Data will be available shortly after the submission system opens HEIs will have access to the same data as the panels after the close of the submissions system

Questions?

Environment data All submissions to include data on: - Research doctoral degrees awarded (REF4a) - Research income (REF4b) - Research income in-kind (REF4c ) Definitions are aligned with HESA returns Data relates to whole unit of assessment - not just submitted staff Sub-panels 8, 9, 19, 25 and 26 request specific additional data, to be included within the environment template (REF5)

Environment data (2) HESA data will be presented to institutions at three stages: - Data for , and in May Excel workbook containing data - Guidance note explaining how the data were extracted - Data for in April Doctoral degree awarded data for will be available from HESA when the HESA student record is submitted

Research doctoral degrees awarded Taken from the HESA student record All instances who successfully complete with: D00: “Doctorate degree obtained primarily through advanced supervised research written up as a thesis/dissertation” D01: “New Route PhD” An award is only counted the first year it is given to an instance An award is not counted if the instance is not closed.

Subject mapping Awards data are mapped to: RAE 2008 UOAs Cost Centres Data linked between years back to To map awards to cost centres and RAE UOAs To detect instances awarded multiple times

Research income Taken from the HESA Finance Statistics Return (FSR) FSR Table 5b: External research grants and contracts Academic departments (Head 1) Academic services (Head 2) Less any income passed on to other institutions or organisations as part of a collaborative project or subcontracted work (Head 5)

Research income-in-kind Data on research income in kind will be provided to institutions from: The UK Research Councils Health research funding bodies: The National Institute of Health Research (NIHR) The Chief Scientist Office, Scottish Government Health and Social Care Directorates The Welsh Government’s National Institute for Social Care and Health Research (NISCHR) The Health and Social Care Research & Development (HSC R&D), Northern Ireland.

Income-in-kind data template InstitutionDepartment Researcher title Researcher surname Researcher initials Facility Income-in-kind (£ by academic year)

Returning data Data should be mapped to REF UOAs Research degrees awarded for all students in those UOAs should be returned, not just those with supervisors returned to the REF Income associated with all staff in those UOAs should be returned, not just those who are submitted REF4 data should align with corresponding HESA, RCUK, health research funding bodies data as closely as possible Validation checks will be run within the submission system software before this data can be submitted

Returning data (2) The total number of research degrees awarded submitted by the HEI as a whole should not exceed the total number of degrees awarded provided in the HESA data by: More than 20 and 5% over the period to More 10 and 10% in any AY to The total amount of research income submitted by the HEI as a whole should not exceed the total amount of income provided in the HESA data by More than £200,000 and 5% over the period to More than £100,000 and 10% in any AY to More than £100,000 and 10% in any category of income over the period to The total amount of research income in kind (REF4c) submitted by the HEI as a whole should not exceed the total amount of income in kind provided by the research councils and health agencies by: More than £100,000 and 5% over the period to 2011 ‑ 12 More than £50,000 and 10% in any AY to

Data adjustments Institutions should contact the REF team as soon as possible where they propose to return environment data that falls beyond the set tolerance limits. Requests should be directed to All requests must be submitted by 1 June 2013 at the very latest.

Questions?

Audit & data verification procedures Audit and data verification of REF submissions is necessary to: support principles of fairness and to adhere to good regulation provide sufficient confidence in the quality and reliability of data and assessment outcomes Any part of a REF submission can be audited and institutions should maintain accurate records in order to verify any information provided as part of their submission. Further guidance on audit and data verification will be published in June 2012 Audit procedures will involve the following processes: Routine checks Structured and targeted audits Systematic data comparisons Panel instigated audits

Audit – Staff eligibility A sample of randomly selected Category A and C staff will be audited in units of assessment for every institution Institutions will be contacted promptly following the submissions deadline to request evidence of staff eligibility. Evidence may include employment contracts or other HR personnel records

Audit – Individual staff circumstances Submitted details for every individual with a clearly defined reduction in outputs will be examined to check that the reductions has been calculated correctly. Further information may be requested. All complex circumstances will be reviewed by EDAP and further information may be requested to inform judgements. A sample of staff with individual circumstances will be audited from each institution to verify the circumstances declared. Evidence will be largely based on: Staff contracts HR confirmation of leave Confirmation that appropriate evidence has been consulted

Audit – Research outputs Audit of research outputs will include a systematic data comparison of: Outputs that may have been returned to RAE2008 Outputs that are listed more than twice within a submission Panels may also raise audit queries on research outputs, these will be followed up, where necessary, by the REF audit team

Audit - Impact Routine audit will be carried out on a sample of impacts from each HEI. Queries will be informed by advice from the panels and are expected to peak in March – June 2014 Queries are expected to relate to: Claims made that can be corroborated through independent sources Eligibility of underpinning research

Audit – Environment Targeted audit will be undertaken where: The environment data (REF4a/b/c) is close to the system limits OR Where a comparison at subject level or cost centre suggests discrepancies in the allocation of data to REF UOAs If audited, HEIs will need to demonstrate how any additional data is eligible; and/or justify how this has been allocated to REF UOAs.

Audit – next steps Guidance on audit and data verification is due to be finalised and published on the REF web-site in June

Questions?