1. Director of the Leeds Social Sciences Institute
Impact Case Studies: ‘How to translate excellent research impact into a REF narrative’ Some reflections from the assessment of impact in the UK REF 2014
Adam Crawford
Director of the Leeds Social Sciences Institute

2. Aims
To consider and reflect upon the lessons learnt from preparing and assessing impact case studies in REF 2014.
To explore the implications and strategies for preparing impact cases studies for REF 2021.
To consider the institutional and individual support and allied actions necessary to ensure high quality impact case studies for submission to REF 2021.
I will draw on personal insights from my own involvement in REF 2014 – Law sub-panel - and as an impact case study author.
Richard Thorpe (Business & Management Studies) and Phil Rees (Archaeology, Geography & Environmental Studies) will add their own reflections, comments and possible corrections – from their disciplinary perspectives.

3. What is impact?
The translation, application and use of research findings and evidence in everyday practices.
A political imperative to justify public funding.
But also…
The desire to make a difference and ambition
to see research impact on real lives.
The moral/ethical imperative to harness knowledge to the ends of societal betterment.
‘Philosophers have hitherto only interpreted the world, in various ways; the point is to change it…’ (Marx’s Thesis on Feuerbach)

4. Impact: Definition for the REF 2014
An effect on, change or benefit to the economy, society, culture, public policy or services, health, the environment or quality of life, beyond academia
Impact includes an effect, change or benefit to:
The activity, attitude, awareness, behaviour, capacity, opportunity, performance, policy, practice, process or understanding
Of an audience, beneficiary, community, constituency, organisation or individuals
In any geographic location whether locally, regionally, nationally or internationally
It excludes impacts on research or the advancement of academic knowledge within HE; and impacts on teaching or other activities within the submitting HEI.

5. The criteria for assessing impacts were reach and significance
Impact: Criteria
The criteria for assessing impacts were reach and significance
Four star
Outstanding impacts in terms of their reach and significance
Three star
Very considerable impacts in terms of their reach and significance
Two star
Considerable impacts in terms of their reach and significance
One star
Recognised but modest impacts in terms of their reach and significance
Unclassified
The impact is of little or no reach and significance; or the impact was not eligible; or the impact was not underpinned by excellent research produced by the submitted unit

6. What we learnt from REF 2014
Some 6,975 impact case studies were assessed, across the 37 UoAs and 154 HE institutions
The REF impact case study database:
Impact scores significantly influenced HEI outcomes.
Some institutions performed poorly on impact.
Leeds generally performed well, notably in the social sciences (Main Panel C):
Across 7 UoAs, Leeds submitted 32 case studies;
Average GPA of 3.50;
57.75% at 4*.
10th in the UK (of HEIs submitting to more than one UoA)

7. Impact Case Studies In each case study, the impact described must:
Meet the REF definition of impact.
Have occurred between 1 Jan 2008 and 31 July 2013 (at any stage of maturity).
Be underpinned by excellent research (at least 2* quality) produced by the submitting unit between 1 January 1993 to 31 December 2013.
Submitted case studies needed not be representative of activity across the unit.
Case studies were complemented by an ‘impact template’ (REF 3a) to describe institutional strategies and approaches to supporting impact and their relationship to the case studies (20% of the total)

8. Impact: Case studies REF 2014
Each case study was limited to 4 pages and must:
Describe the underpinning research produced by the submitting unit
Reference one or more key outputs and provide evidence of the quality of the research
Explain how the research made a ‘material and distinct’ contribution to the impact (there are many ways in which this may have taken place)
Explain and provide appropriate evidence of the nature and extent of the impact: Who/what was affected? How were they affected? When?
Provide independent sources that could be used to verify claims about the impact (on a sample audit basis)

9. Impact Assessment
REF panels included non-academic assessors drawn from outside the research community – ‘Cuckoos in the self- regulatory nest’?
Law panel (for example) included:
17 academic members + 2 ‘user’ members;
8 academic assessors + 6 impact assessors
Reach – Not necessarily geographical, could be the number of beneficiaries.
Significance – The scale of change or benefit affected and what this means for the beneficiaries.

10. Reflections on REF 2014 Challenges of attribution
After the event, ‘dustpan and brush’ approach
Fitting impact into the criteria
Not all good research impact makes a good REF narrative – nor should the quality of all impact be judged in the narrow terms of REF!
Fitting impact into a 4 page template (with indicative word max)
Title
Summary (100)
Underpinning research (500)
References to the Research (Outputs) (6 References)
Details of the Impact (750)
Sources to corroborate the impact (10 References)
Challenge of the ‘one handed scientist’
How to be less modest, removing the caveats, without over-egging it (claiming too much)!

11. Reflections on the process of preparing impact case studies for REF 2014
Writing process as long and iterative; subject to review and multiple input.
Some possible case studies not selected for submission.
Writing a impact narrative was uncomfortable for some scholars.
Having other people (re-)draft or ghost-write the narrative.
Sourcing corroborative evidence – when and how?
Enhancing impact before the deadline.
Heavy burden - work and responsibility – on small number of shoulders.

12. REF 2014 – Law UoA

13. Impact – Law sub-panel 225 impact case studies
Subject-matter included (but was not limited to)
criminal justice/human rights
regulation of business, commerce, environment
medicine and bio-ethics
EU and international law
Audience primarily / beneficiaries
policy and law-makers
judiciary and the Bar
Small number of examples related to public engagement and legal education

14. Impact Case Studies in Law REF 2014
Impact usually demonstrated by how it had:
informed the development of policy and new legislation; and
influenced the work of national, EU and international policy makers, judges and legislators.
There were also convincing examples of how research had:
informed public debate; and
held proposed changes or practices up to scrutiny, sometimes resulting in preventing a change from taking place.
There was less consideration of the importance of research in improving NGO campaigning and input to policy or service provision, and less mention of the impact on end service users.

15. Some typologies of case studies
Eminent scholar(s) built on a reputation of research – advising government or organisations
Campaigning research – bringing an issue to public/policy attention:
Team built around funded empirical research project(s) – often including engagement with external partners:
Consultancy (type) research for particular company, organisation, government.
Public engagement – informing, educating (via media, tv)

16. Impact case studies: strengths and weaknesses
Gave clear indication of what was the underpinning research and explanation for its 2* (at least) quality
Understood the distinction between dissemination and impact
Gave clear explanation of how the research had been utilised to bring about change
Provided clear (and not overstated) account of the reach and significance of the impact
When corroborating sources were followed up, they justified the claims made
Failed to set out the thread of evidence linking the research to the impact and failed to establish 2* quality of research
Did not understand the distinction between dissemination and impact
Made unconvincing or overstated claims of the reach and significance of the impact
When corroborating sources were followed up, they did not back up the claims made

17. Impact templates: strengths and weaknesses
Provided evidence of thought in identifying pathways to impact and a strategic approach reflecting the unit’s own research context and priorities
Understood the distinction between dissemination and impact
Had (or planned) pro-active mechanisms for linking with end-users
Assisted and supported staff to develop impact skills whilst recognising that not all research leads to non-academic impact
Offered an unimaginative list of possible ways of engaging with end-users
Did not demonstrate a clear understanding of impact
Lacked a clear strategic approach to moving forward
Were unambitious in their plans for achieving impact in future

18. What makes a good impact case study?
The strongest case studies provided:
A coherent and convincing narrative.
A clear account of what the underpinning research identified as the issue and then how that research was utilised to achieve the specific change or effect being claimed.
Effectively pinpointed the nature and scale of the impact, describing not simply the change brought about, but the extent and depth of that change on those affected.
Distinguished clearly between the process of disseminating the research and the resulting impact.

19. What makes a poor impact case study?
Weaker case studies:
Lack of narrative chain - failed to set out clearly the thread of evidence linking the research to the claimed impact.
Limited description of the research / its findings which made it hard to understand how it had made a distinct and material contribution to the impact.
Failed to provide evidence of the quality of the underpinning research as required.
Difficulties of assessing the evidence presented for the impact of dissemination activity.
Did not supply convincing evidence to support the claims.
Made exaggerated claims for the impact of the research, which were often exposed by reference to the corroborating sources, particularly where the issue was the subject of wide public debate and academic commentary and endeavour.

20. Impact Assessment: REF Panel Insights
Many possible pathways to impact; planned, serendipity, co- production.
There is a difference between dissemination and impact.
Cases claiming impact from dissemination should ideally demonstrate a change as a result of the dissemination activity, but a range of other evidence could indicate the dissemination activity was valued (i.e. repeat visits, growing audience numbers over time, etc.).
In some instances a case study grouped together either underpinning research which appears unrelated or a range of impacts which seem unrelated (a ‘portfolio’ approach).
The assessment of these cases was based on the coherence of the overall narrative, rather than looking at individual elements separately.

21. Impact Assessment Questions:
What was the underpinning research? What was its key findings?
The ‘golden thread’ linking the chain of causation.
What impact is being claimed? What changed as a result of the research? What was the beneficial outcome of the change?
Be specific and clear about the claim being made: avoid a ‘shot-gun’ approach. Who were the beneficiaries and how did they benefit?
Did the research make a distinct / material contribution to the impact?
The research making a distinct and material contribution to the impact is a key factor; it needs to be clear that the research cited made such a contribution.
What were the mechanisms through which impact was realised? How did the research come to have the impact claimed?
How was the research communicated, disseminated and applied?
What evidence is there to corroborate the claims of change/impact?
Case studies must be assessed on the basis of the evidence provided in the (4 pages) template. Corroborating sources should be used to verify evidence cited in the case study and not to gather further information.

22. Links in the Impact Chain
A four page narrative to evidence:
Mechanisms of Dissemination
Underpinning Research
Publications
Key Research Findings
Details of Impact Claimed
Corroborating Evidence of Impact
Outcomes of Impact/Change

23. RAND Reports Impact was a ‘significant new burden’ on HEIs:
It cost UK HEIs around £55m to prepare impact submissions as part of REF 2014.
25.9% of total estimated cost of REF 2014 = £212m
Biggest challenges (and burdens) in preparing impact case studies were the requirement to ‘evidence’ impact and the need to develop an understanding of the concept of impact.
As a result of the impact agenda and changing culture, HEIs and individual researchers are changing their practices.
Concern that the impact agenda may begin to undermine ‘blue skies’ research.
The use of impact case studies ‘remains the most appropriate means of assessment’ (2015a: xiv)
The ‘challenge for assessing and evidencing research impact is in understanding what kinds of impact categories and (qualitative and quantitative) indicators will be most appropriate, and in what contexts’.
Manville, C. et al (2015a) Assessing impact submissions for REF 2014: An evaluation. Manville, C. et al (2015b) Preparing impact submissions for REF 2014: An evaluation.

24. Some (unintended) consequences
‘There is evidence of culture change within HEIs. Institutional strategies and processes have been or are being put in place to foster a culture of impact and maximise the impacts that occur from research being
undertaken.’ (Manville et al. 2015b: 23)
Consequences:
Low hanging fruit;
Focus on easy to measure;
Threshold effects;
Problems of interdisciplinarity:
The mean submission size for a unit of assessment in the REF was FTE staff—necessitating three case studies, each worth 5.33 per cent of that unit’s overall profile.
Compare this with publications.
With each researcher submitting an average of 3.67 outputs, the average submission would have just over 70 outputs.
These accounted for 65 per cent of the profile, or 0.92 per cent for each output.
In this example, a case study is worth nearly six times as much as a research output
Manville, C. et al (2015b) Preparing impact submissions for REF 2014: An evaluation Findings and observations, Rand
British Academy (2016) Crossing Paths: Interdisciplinary Institutions, Careers, Education and Applications, London: The British Academy, p. 10.
‘Many of the reasons for avoiding inter-disciplinary projects relate to the fact that it is harder to publish outputs; such work is perceived to have less value to hiring and promotion panels; and one is less likely to be selected for submission to REF.’ (British Academy 2016: 10)

25. Stern Review Working assumptions:
Past Research Assessment Exercises and the 2014 REF have contributed productively to driving competition and fostering research excellence.
Impact as a principle is important and, even though it can and should be improved, it made a useful contribution as part of REF2014.
‘the new impact element of the REF has contributed to an evolving culture of wider engagement, thereby enhancing delivery of the benefits arising from research, as captured through the impact case studies’ (p. 9)
Principles:
Making space for long-term research
Lower burden
Less game-playing
More rounded view of research activity
Less personalization, more institution focus
Interdisciplinary emphasis
Recognition for investment
Impact to be ‘deepened and broadened’

26. Impact in REF 2021
HEFCE’s Initial Decisions on REF 2021 (September 2017)
Impact to increase to 25% of the total assessment.
Number of impact case studies to remain broadly the same – ‘should not significantly exceed the number submitted in 2014’.
Impact template to move to explicit section in ‘environment’ element.
Institutional level impact to be piloted but not introduced in REF 2021.
Case studies continued from examples submitted in 2014 will be eligible for submission in REF 2021.
‘submitting units will need to identify continued case studies in the case study template’.
‘continued or developed case studies will need to provide evidence of additionality to the example submitted in 2014’.

27. Impact in REF 2021: Further guidance to come…
“We will work with the panels to provide additional guidance on:
- The criteria for impact of ‘reach and significance’.
Impact arising from public engagement.”
“The guidance on submitting impacts on teaching will be widened to include impacts within, as well as beyond, the submitting institution.
work with the panels to develop appropriate guidance on demonstrating evidence against the criteria for this type of impact.”
“Research activity and bodies of work. In line with Stern’s recommendation… we intend to broaden out the relationship between the underpinning research and impact from individual outputs, to include a wider body of work or research activity… [will be developed] to feed into the detailed guidance on submissions.”

28. Number of impact case studies
Number of staff submitted
No. of case studies
Up to 15
>15 – 30
>30 – 45
>45 – 60
>60 – 75
>75 – 90
>90 – 105
>105 – 155
>155 – 205 2 3 4 5 6 7 8 9 10
REF 2021: Decisions on Staff and Outputs (November 2017)

29. Impact: Case studies 2021
In each case study, the impact described must:
Meet the REF definition of impact
Have occurred between the designated period 1 August 2013 to 31 July 2020 (can be at any stage of maturity)
Be underpinned by excellent research (at least 2*) produced between 1 January 2000 to 31 December 2020.
Submitted case studies need not be representative of activity across the unit: pick the strongest examples.
Impact case study template will be revised to include:
a set of mandatory fields, to support the assessment and audit process (i.e. case study title, as well as information relating to the time period and employment of staff involved in the associated research).
include a section for additional contextual data to be provided in a standardised way (i.e. the research funder, where applicable).

30. Thinking about Impact
1. Direct – linear impact: X research led to Y impact
2. Indirect – Non-linear impact:

31. Impact as serendipity rather than design?
There is no simple relation between research findings and policy impacts (or other impacts).
Some impacts are fortuitous due to timing, wider context, prevailing political flavour or confirming what policy-makers want to hear – saying the ‘desired’ thing at the auspicious moment!
Impact doesn’t necessarily produce change/reform.
It may help to resist or hold back change.
It may foster a better quality debate/dialogue.
It may be who you know not what you know!
REF panels were aware of this and did take this into consideration;
‘User’ panel members were often more positive about impact – the dog that didn’t bark!

32. Tips for embedding impact
Think about the possible application of your research from the outset, before you start.
Engage with possible research users early – not simply once you’ve finished.
External partners might help shape the research or provide useful data and information
Research advisory board can help provide an investment.
Are there intermediaries who might help communicate your research? If so, engage them.
Communicate (translate) your findings in ways that are accessible to possible beneficiaries (think ‘application’):
Publish in different outlets
Summary Findings
Know your own limitations! Don’t try to do it all.

33. Further information:
Overview report by Main Panel C and Sub-panels 16 to 26
Lord Stern’s independent review of REF (2016) Building on Success and Learning from Experience
The Metric Tide: Report of the Independent Review of the Role of Metrics in Research Assessment and Management (2015)
RAND Reports on Impact:
Manville, C. et al (2015) Assessing impact submissions for REF 2014: An evaluation.
Manville, C. et al (2015) Preparing impact submissions for REF 2014: An evaluation.