Demonstrating research impact in the REF Graeme Rosenberg REF Manager

Outline: What is the REF? What is impact? How should it be demonstrated? How will it be assessed?

What is the REF? The REF is the new framework for assessing research in all disciplines in all UK higher education institutions It replaces the RAE Its purpose is: To inform research funding allocations by the four UK HE funding bodies (approximately £2 billion per year) Provide accountability for public funding of research and demonstrate its benefits To provide benchmarks and reputational yardsticks

The assessment framework Overall quality Outputs Maximum of 4 outputs per researcher Impact Impact template and case studies Environment Environment data and template 65% 20% 15%

Submissions There are 36 discipline-based units of assessment (UOAs) Each institution decides which UOAs to submit in Each submission in a UOA must contain a standard set of information: Staff details (REF1a/b/c) Research outputs (REF2) Impact template and case studies (REF3a/b) Environment data (REF4a/b/c) Environment template (REF5)

Assessment Submissions will be assessed by 36 expert sub-panels working under the guidance of 4 main panels Panels will carry out the assessment according to the published criteria and working methods Sub-panel responsibilities Contributing to the panel criteria and working methods Assessing submissions and recommending the outcomes Main panel responsibilities Developing the panel criteria and working methods Ensuring adherence to the criteria/procedures and consistent application of the overall assessment standards Signing off the outcomes

Example of a quality profile

Timetable Criteria phase 2011 Submissions phase 2012-13 Develop and publish Guidance on submissions (Jul) Develop, consult on and publish Panel criteria (Jan 2012) Submissions phase 2012-13 HEIs submit Codes of practice Launch the REF submissions system Submission deadline 29 Nov 2013 Assessment phase 2014 Panels assess submissions Publish outcomes Dec 2014

What is impact? Impact is defined broadly for the REF: an effect on, change or benefit to the economy, society, culture, public policy or services, health, the environment or quality of life, beyond academia The REF definition of impact is retrospective – benefits or changes that have already occurred, not future potential Impacts can be manifest in a wide variety of ways, may take many forms and occur in a wide range of spheres, in any geographic location

Examples of types of impact Enhanced professional standards, ethics, guidelines or training Improved health or welfare outcomes Improved risk management Public debate has been shaped or informed by research More effective management or workplace practices Improved business performance Improved quality, accessibility or efficiency of a public service A social enterprise initiative has been created Research has enabled stakeholders to challenge conventional wisdom Production costs have reduced Changes to the design or delivery of the school curriculum Improved forensic methods or expert systems Enhanced preservation, conservation or presentation of cultural heritage Improved access to justice, employment or education Policy debate or decisions have been influenced or shaped by research Improved management or conservation of natural resources Jobs have been created or protected Research has informed public understanding, values, attitudes or behaviours Organisations have adapted to changing cultural values The policies or activities of NGOs or charities have been informed by research Changes to legislation or regulations Enhanced corporate social responsibility policies Levels of waste have reduced Changes in professional practice Enhanced technical standards or protocols A new product has been commercialised New forms of artistic expression or changes to creative practice

Examples from the pilot exercise Improved treatment and quality of life for patients with type 1 diabetes Detector technology used to screen liquids at airports Conservation of bumblebees Influencing policy on child maintenance and better service for separated parents Enriched public appreciation of heritage at Hampton Court Palace Pilot examples are published at: www.ref.ac.uk/background/pilot an effect on, change or benefit to the economy, society, culture, public policy or services, health, the environment or quality of life, beyond academia Impacts can be manifest in a wide variety of ways, may take many forms and occur in a wide range of spheres, in any geographic location Panels provide examples of impact relevant to their disciplines, intended to be helpful - not as exhaustive or prescriptive lists

Demonstrating impact Specific examples of impacts already achieved Impact template Sets out the submi tted unit’s appro ach to impac t Case studies Specific examples of impacts already achieved 20% of the impact sub-profile 80% of the impact sub-profile

Case studies One case study must be submitted per 10 FTE staff (with a minimum of two cases in a submission) Impacts that took place during 2008 to 2013; underpinned by research since 1993 Submitted case studies need not be representative of activity across the unit: pick the strongest examples Case studies must be completed on a template - maximum 4 pages

The case study template Summary of the impact Underpinning research: What was it? Who undertook it and when? What were the key insights or findings? References to the research Details of the impact: How did the research lead/contribute to the impact? Who were the beneficiaries? How did they benefit? Sources of corroboration

Lessons from the pilot exercise All the material required to make a judgement should be included The narrative should be coherent and clearly explain the ‘links in the chain’ Clear definition of who benefitted, and what had changed Indicators should be meaningful and contextualised Key claims should be capable of verification Where the impact arises from public engagement: How was the engagement activity based on the research? Dissemination in itself is not impact – what was the benefit?

Preparing impact submissions HEIs seem to be at varying stages of: Briefing staff Identifying potential cases Drafting case studies Some issues seem to be: How to write compelling narrative, and who should be involved Evidence gathering – retrospective and from many sources Institutional memory – what if the staff have left? Second guessing the panels to decide on the strongest cases

How will impacts be assessed? Sub-panels will assess impacts against the criteria of reach and significance: A holistic judgement about each case study ‘Reach’ is not a geographic scale. Each case study will be examined by at least one academic and one ‘user’ on the panel Panels will make judgements on the merits of the evidence provided – not further investigations A sample of case studies will be audited to corroborate or verify information

Underpinning research Panels will assess the impact, not the underpinning research But each case study must be underpinned by research that satisfies the following criteria: produced by staff while working in the submitting HEI generated outputs published between 1 Jan 1993 and 31 Dec 2013 meets the quality threshold of at least equivalent to 2* made a material and distinct contribution to the impact

The criteria for assessing impacts are reach and significance* Assessment criteria The criteria for assessing impacts are reach and significance* Four star Outstanding impacts in terms of their reach and significance Three star Very considerable impacts in terms of their reach and significance Two star Considerable impacts in terms of their reach and significance One star Recognised but modest impacts in terms of their reach and significance Unclassified The impact is of little or no reach and significance; or the impact was not eligible; or the impact was not underpinned by excellent research produced by the submitted unit * Each main panel provides further description of the criteria

Further information www.ref.ac.uk (includes all relevant documents) Enquiries from staff at HEIs should be directed to their nominated institutional contact (see www.ref.ac.uk for a list) Other enquiries to info@ref.ac.uk