Research Support Conference, 21 June 2011 Preparing for the REF:   Experience from the HEFCE Impact Pilot Dr Andrew Walsh Director Research & Business Engagement Support Services

Nature of the pilot Five proposed REF UOAs were chosen in which to pilot the assessment of impact information: Clinical Medicine Physics Earth Systems and Environmental Sciences Social Work and Social Policy & Administration English Language & Literature Twenty-nine HEIs from across the UK were selected to take part in the pilot, to provide a spread of at least 10 institutions with differing characteristics submitting to each pilot UOA. Earth Systems and Environmental Sciences and English Language and Literature were piloted in Manchester

Submission contents For each UoA: overview information relating to the submitted unit as a whole a number of (template-structured) case studies illustrating specific impacts (one for every ten 2008 Cat A staff) Impacts must have occurred between Jan 2005 and Dec 2009 The underpinning research could date back to 1993 and should be >2* in quality Impact defined broadly as “any identifiable benefit to or positive influence on the economy, society, public policy or services, culture, the environment or quality of life.” Does not include impacts within the academic sphere or the advancement of scientific knowledge

Impact Indicators Delivering highly skilled people Creating new businesses, improving the performance of existing businesses, or commercialising new products or processes Attracting R&D investment from global business Better informed public policy-making or improved public services Improved patient care or health outcomes Progress towards sustainable development, including environmental sustainability Cultural enrichment, including improved public engagement with science and research Improved social welfare, social cohesion or national security Other quality of life benefits

Impact quality levels Criteria = ‘range’ and ‘significance’ of impacts Four star/Exceptional: Ground-breaking or transformative impacts of major value or significance with wide-ranging relevance have been demonstrated Three star/Excellent: Highly significant or innovative (but not quite ground-breaking) impacts relevant to several situations have been demonstrated Two star/Very good: Substantial impacts of more than incremental significance or incremental improvements that are wide-ranging have been demonstrated One star/Good: Impacts in the form of incremental improvements or process innovation of modest range have been demonstrated Unclassified: The impacts are of little or no significance or reach; or the underpinning research was not of high quality; or research-based activity within the submitted unit did not make a significant contribution to the impact.

Attribution Specific research-based activity within the institution must have made a significant contribution to achieving the impact. Either: Through research (fixed): Underpinning research that was undertaken by staff within the institution. This could be any type of research that meets the Frascati principles, including basic research, practice-based research, applied or translational research. or Through researchers (portable): Other research-based activity undertaken by research-active staff at the institution which drew or built substantially on their own research – e.g. the contribution of an individual as an expert advisor on a committee, where this was based to a significant degree on their personal research record.

Research 1 HEI X Impact Research 2 HEI Y Research 3 HEI Z influence

HEI Y + HEI Z collaboration Research 1 HEI X Impact influence Research 2 HEI Y + HEI Z collaboration influence

Research 1 HEI X Impact Exploited by HEI Z Research 2 HEI Y Research 3 HEI Z

Research 1 HEI X Impact Exploited by HEI Z Research 2 HEI Y

Impact Research 1(a) Prof A @ HEI X Research 1(b) Prof A @ HEI Y 1990-1995 1995-2000 Impact Research 1(a) Prof A @ HEI X Research 1(b) Prof A @ HEI Y

Impact Research 1(a) Prof A @ HEI X Research 1(b, c, d) Prof A @ HEI Y 1990-1995 1995-2000 2005-2009 Impact Research 1(a) Prof A @ HEI X Research 1(b, c, d) Prof A @ HEI Y

Impact 1 Impact 3 Research 1 HEI X Impact 2 1990-1995 1995-2000 2000-2005 2005-2009 Impact 1 Impact 3 Research 1 HEI X Impact 2

Research 1 Physics @ HEI X Impact Exploitation Engineering @ HEI X

‘Basic’ Research 1 Physics @ HEI X Impact ‘Applied’ Research 2 Engineering @ HEI X

Knowledge pool Public policy change Prof A @ HEI X Expert advisor influence

Knowledge pool Public policy change Prof A @ HEI X Expert advisor influence Research by Prof A @ HEI X

Knowledge pool Public policy change Prof A @ HEI X Expert advisor influence Research by Prof A @ HEI X, Y, Z

Developing submissions - challenges? Relatively low level of appreciation of what counts as impact for REF HEFCE advice ambiguous in some areas. Difficult always to distinguish fully between inputs (which “are not the focus of assessment”) and interim or full outcomes (which are) Relevant indicators of impact range and significance not readily available, sufficient or widely understood in the institution – ‘standing start’ Difficulty and burden of contacting past researchers and end users in order to identify and evidence impact Difficult to determine what some stakeholders do with data provided by the institution or what access we might have to impact indicators

Developing submissions – challenges(2)? Tendency, in case studies, for attribution of impacts to be stressed more than their range and significance Desire to avoid ‘mere’ knowledge transfer can lead to an overly inhibited account of the contribution of our research to major impacts Uncertainty about appropriate scope for case studies or the need for these to be representative of UoA activity as a whole Tendency to focus on recent research activity by current members of staff with strategic academic potential. Are we missing profitable case studies relating to research areas that may now be dormant?

Earth Systems & Environmental Science 4* 3* 2* 1* U UOA average 18 28 24 15 15 Institution A 0 50 0 0 50 Institution B 50 0 0 0 50 Institution C 25 0 25 50 0 Institution D 35 25 25 15 0 UoM 25 25 50 0 0 Institution F 25 25 25 25 0 Institution G 0 0 25 25 50 Institution H 0 50 25 0 25 Institution I 0 35 30 0 35 Institution J 0 100 0 0 0 * Impact statements not scored

English Language & Literature UOA average 19 30 30 19 2 Institution A 25 0 50 25 0 UoM 0 0 20 80 0 Institution C 20 40 40 0 0 Institution D 0 0 100 0 0 Institution E 10 20 50 20 0 Institution F 0 20 80 0 0 Institution G 40 60 0 0 0 Institution H 25 50 25 0 0 Institution I 35 50 15 0 0 Institution J 40 0 60 0 0 Institution K 0 60 0 40 0 Institution L 20 30 20 15 15 Institution M 0 0 0 100 0

What HEFCE and panels learnt from pilot Panels broadly content that impact can be assessed in REF by mixed academic/user panels Case study approach is the best mechanism for this assessment ‘Reach and ‘significance’ are appropriate broad criteria 15 year time-lag between research and impact reasonable in most cases Unconvinced of the need for separate impact statement (elements should be incorporated into environment (‘RA5’) statement) *rejected* Recommend that weighting for impact assessment should be lower than proposed 25% for first REF (but note CSR politics – likely to be 20%) Highest scoring cases studies provided “a coherent narrative with evidence of specific benefits” Many of the poorest case studies took the form: “Professor X is very well-known, here is a list of his/her media appearances and publications”

What have we learnt from the pilot? Variability of outcomes and likelihood of significant weighting indicate that this is a very significant challenge to REF performance Reasonable performance in Earth Systems and Environmental Sciences, including a 4* case study, provides useful exemplar for cognate disciplines Disappointing performance in English Language and Literature indicates that efforts to argue for dissemination per se as impact are unlikely to be successful. We must: “make a case for the benefits arising from public engagement activity… [T]his must go beyond showing how the research was disseminated” show what “distinctive contribution the department’s research made to the public engagement activity” and that the engagement went beyond “business as usual” activities (such as public lectures) HEFCE have published a selection of top-rated case studies from each panel

How might we move forward? Improve data on impact collected from Research Profiling Exercise Begin developing long-list of candidate case studies and collect evidence, e.g: Collection of impact statements and reports for all grant-funded research Information on commercialisation impacts (e.g. spinouts - involving UMIP) Information on business and other relationships where research has not been commercialised by the University but where impacts are significant (involving external and business relations, regional affairs, media relations, etc) Maintain an accurate picture of postgraduate alumni and research staff destinations, especially if entering areas of national strategic importance (involving careers and employability) Consider use of consultants to help with quantifying the commercial impact of our work or involvement of science journalists and other professionals to help case studies communicate impact or to review past 20 years’ of research for eligible impacts UoM process for reviewing draft case studies in Summer 2011 Further development in the light of panel criteria (released for consultation in July 2011)