1. Research Support Conference, 21 June 2011
Preparing for the REF:   Experience from the HEFCE Impact Pilot Dr Andrew Walsh Director Research & Business Engagement Support Services

2. Nature of the pilot
Five proposed REF UOAs were chosen in which to pilot the assessment of impact information:
Clinical Medicine
Physics
Earth Systems and Environmental Sciences
Social Work and Social Policy & Administration
English Language & Literature
Twenty-nine HEIs from across the UK were selected to take part in the pilot, to provide a spread of at least 10 institutions with differing characteristics submitting to each pilot UOA.
Earth Systems and Environmental Sciences and English Language and Literature were piloted in Manchester

3. Submission contents For each UoA:
overview information relating to the submitted unit as a whole
a number of (template-structured) case studies illustrating specific impacts (one for every ten 2008 Cat A staff)
Impacts must have occurred between Jan 2005 and Dec 2009
The underpinning research could date back to 1993 and should be >2* in quality
Impact defined broadly as “any identifiable benefit to or positive influence on the economy, society, public policy or services, culture, the environment or quality of life.”
Does not include impacts within the academic sphere or the advancement of scientific knowledge

4. Impact Indicators Delivering highly skilled people
Creating new businesses, improving the performance of existing businesses, or commercialising new products or processes
Attracting R&D investment from global business
Better informed public policy-making or improved public services
Improved patient care or health outcomes
Progress towards sustainable development, including environmental sustainability
Cultural enrichment, including improved public engagement with science and research
Improved social welfare, social cohesion or national security
Other quality of life benefits

5. Impact quality levels Criteria = ‘range’ and ‘significance’ of impacts
Four star/Exceptional: Ground-breaking or transformative impacts of major value or significance with wide-ranging relevance have been demonstrated
Three star/Excellent: Highly significant or innovative (but not quite ground-breaking) impacts relevant to several situations have been demonstrated
Two star/Very good: Substantial impacts of more than incremental significance or incremental improvements that are wide-ranging have been demonstrated
One star/Good: Impacts in the form of incremental improvements or process innovation of modest range have been demonstrated
Unclassified: The impacts are of little or no significance or reach; or the underpinning research was not of high quality; or research-based activity within the submitted unit did not make a significant contribution to the impact.

6. Attribution
Specific research-based activity within the institution must have made a significant contribution to achieving the impact.
Either:
Through research (fixed):
Underpinning research that was undertaken by staff within the institution. This could be any type of research that meets the Frascati principles, including basic research, practice-based research, applied or translational research. or
Through researchers (portable):
Other research-based activity undertaken by research-active staff at the institution which drew or built substantially on their own research – e.g. the contribution of an individual as an expert advisor on a committee, where this was based to a significant degree on their personal research record.

7. Research 1 HEI X Impact Research 2 HEI Y Research 3 HEI Z influence

8. HEI Y + HEI Z collaboration
Research 1
HEI X
Impact
influence
Research 2
HEI Y + HEI Z collaboration
influence

9. Research 1
HEI X
Impact
Exploited by HEI Z
Research 2
HEI Y
Research 3
HEI Z

10. Research 1
HEI X
Impact
Exploited by HEI Z
Research 2
HEI Y

11. Impact Research 1(a) Prof A @ HEI X Research 1(b) Prof A @ HEI Y
Impact
Research 1(a)
Prof HEI X
Research 1(b)
Prof HEI Y

12. Impact Research 1(a) Prof A @ HEI X Research 1(b, c, d) Prof A @ HEI Y
Impact
Research 1(a)
Prof HEI X
Research 1(b, c, d)
Prof HEI Y

13. Impact 1 Impact 3 Research 1 HEI X Impact 2 1990-1995 1995-2000
Impact 1
Impact 3
Research 1
HEI X
Impact 2

14. Research 1
HEI X
Impact
Exploitation
HEI X

15. ‘Basic’ Research 1
HEI X
Impact
‘Applied’ Research 2
HEI X

16. Knowledge pool Public policy change Prof A @ HEI X Expert advisor
influence

17. Knowledge pool Public policy change Prof A @ HEI X Expert advisor
influence
Research by Prof A
@ HEI X

18. Knowledge pool Public policy change Prof A @ HEI X Expert advisor
influence
Research by Prof A
@ HEI X, Y, Z

19. Developing submissions - challenges?
Relatively low level of appreciation of what counts as impact for REF
HEFCE advice ambiguous in some areas. Difficult always to distinguish fully between inputs (which “are not the focus of assessment”) and interim or full outcomes (which are)
Relevant indicators of impact range and significance not readily available, sufficient or widely understood in the institution – ‘standing start’
Difficulty and burden of contacting past researchers and end users in order to identify and evidence impact
Difficult to determine what some stakeholders do with data provided by the institution or what access we might have to impact indicators

20. Developing submissions – challenges(2)?
Tendency, in case studies, for attribution of impacts to be stressed more than their range and significance
Desire to avoid ‘mere’ knowledge transfer can lead to an overly inhibited account of the contribution of our research to major impacts
Uncertainty about appropriate scope for case studies or the need for these to be representative of UoA activity as a whole
Tendency to focus on recent research activity by current members of staff with strategic academic potential. Are we missing profitable case studies relating to research areas that may now be dormant?

21. Earth Systems & Environmental Science
4* 3* 2* 1* U
UOA average
Institution A
Institution B
Institution C
Institution D
UoM
Institution F
Institution G
Institution H
Institution I
Institution J
* Impact statements not scored

22. English Language & Literature
UOA average
Institution A
UoM
Institution C
Institution D
Institution E
Institution F
Institution G
Institution H
Institution I
Institution J
Institution K
Institution L
Institution M

23. What HEFCE and panels learnt from pilot
Panels broadly content that impact can be assessed in REF by mixed academic/user panels Case study approach is the best mechanism for this assessment ‘Reach and ‘significance’ are appropriate broad criteria 15 year time-lag between research and impact reasonable in most cases Unconvinced of the need for separate impact statement (elements should be incorporated into environment (‘RA5’) statement) *rejected* Recommend that weighting for impact assessment should be lower than proposed 25% for first REF (but note CSR politics – likely to be 20%) Highest scoring cases studies provided “a coherent narrative with evidence of specific benefits” Many of the poorest case studies took the form: “Professor X is very well-known, here is a list of his/her media appearances and publications”

24. What have we learnt from the pilot?
Variability of outcomes and likelihood of significant weighting indicate that this is a very significant challenge to REF performance
Reasonable performance in Earth Systems and Environmental Sciences, including a 4* case study, provides useful exemplar for cognate disciplines
Disappointing performance in English Language and Literature indicates that efforts to argue for dissemination per se as impact are unlikely to be successful. We must:
“make a case for the benefits arising from public engagement activity… [T]his must go beyond showing how the research was disseminated”
show what “distinctive contribution the department’s research made to the public engagement activity” and that the engagement went beyond “business as usual” activities (such as public lectures)
HEFCE have published a selection of top-rated case studies from each panel

25. How might we move forward?
Improve data on impact collected from Research Profiling Exercise
Begin developing long-list of candidate case studies and collect evidence, e.g:
Collection of impact statements and reports for all grant-funded research
Information on commercialisation impacts (e.g. spinouts - involving UMIP)
Information on business and other relationships where research has not been commercialised by the University but where impacts are significant (involving external and business relations, regional affairs, media relations, etc)
Maintain an accurate picture of postgraduate alumni and research staff destinations, especially if entering areas of national strategic importance (involving careers and employability)
Consider use of consultants to help with quantifying the commercial impact of our work or involvement of science journalists and other professionals to help case studies communicate impact or to review past 20 years’ of research for eligible impacts
UoM process for reviewing draft case studies in Summer 2011
Further development in the light of panel criteria (released for consultation in July 2011)