1. What does ‘being returned’ to the REF mean?
Professor Carol Atkinson
Associate Dean for Research

2. Overview of the session
Research Excellence Framework (REF)
Being ‘REF-returned’ and its benefits
Measuring research quality
Us and REF2014
Us and REF2020
How to develop a REF-returnable profile

3. What is REF?
Research Excellence Framework: measure of research quality in UK universities (previously RAE)
Allocation of research funding (QR; HEFCE)
Important in league tables/rankings (double weighted)
Do not confuse research with REF: the tail should not wag the dog (but don’t tell HEFCE I said that!)

4. What does being REF-returned mean?
Simply that we include you as one of the researchers whose outputs/impact/esteem we present in our REF submission
NO individual feedback
We’ll see later what you need to do to get to this level

5. Benefits of being REF-returnable/ returned
Researcher Category A status; currently 40% research workload
So time to do research!
Career progress: internally or externally
And the sheer joy of research beyond REF!

6. How is research quality measured?
Outputs (65%: primarily journal articles, only funding for those at 3*/4*)
Environment (15%: systems and processes, research income, PGR recruitment and completions, esteem)
Impact (20%; new in 2014 – impact case studies; nothing to do with journal impact factors)
Maximising GPA

7. How did MMU do in REF2014? Outputs 25.90 5.0 48.8 31.2 13.8 1.2 2.43
FTE 4 3 2 1
unclassified
GPA
Business and Management Studies
Outputs
25.90
5.0
48.8
31.2
13.8
1.2
2.43
Impact
40.0
30.0
20.0
10.0
0.0
3.0
Environment
12.5
62.5
25.0
2.88
Overall 13 47 28 11
2.6

8. How does that compare to other UK Business Schools (UoA 19, 101 in total)
GPA: 61st
Research Power: 52nd (FTE submitted x GPA)
Research Intensity: 79 (FTE submitted/FTE eligible for submission x GPA)
Returned 26 academics (16%) – need to roughly double for REF2020. That means YOU!
Need GPA of 3+ to be submitted………

9. What will REF2020 look like? We don’t know!
We are working on the assumption that
a) there will be another REF
b) it will largely the same as REF2014 (impact may increase to 25%)
But the rules changed between RAE2008 and REF2014 (e.g. impact, ceasing to fund 2* research) and could change again. Watch this space……….

10. What do you need to do to be returned?
REF-returnable may not be REF-returned
Strategic (game playing?)
Not being returned may link to
Number of impact case studies
Shared authorships etc
Starting point is……….

11. OUTPUTS Each returned person needs 4 outputs of 3*/4* quality
You CANNOT be returned without these (unless you have particular circumstances to be taken into account e.g. sickness, maternity)
Do not be distracted from your outputs: impact, research income etc will be lost if you cannot be returned because you do not have 4 outputs

12. How is quality of outputs assessed?
Originality will be understood in terms of the innovative character of the research output. Research outputs that demonstrate originality may: engage with new and/or complex problems; develop innovative research methods, methodologies and analytical techniques; provide new empirical material; and/or advance theory or the analysis of doctrine, policy or practice.
Significance will be understood in terms of the development of the intellectual agenda of the field and may be theoretical, methodological and/or substantive. Due weight will be given to potential as well as actual significance, especially where the output is very recent.
Rigour will be understood in terms of the intellectual precision, robustness and appropriateness of the concepts, analyses, theories and methodologies deployed within a research output. Account will be taken of such qualities as the integrity, coherence and consistency of arguments and analysis, such as the due consideration of ethical issues.

13. What is 3*/4*?
In assessing work as being four star (quality that is world-leading in terms of originality, significance and rigour), is there evidence of, or potential for, some of the following types of characteristics:
outstandingly novel in developing concepts, techniques or outcomes
a primary or essential point of reference in its field or sub-field
major influence on the intellectual agenda of a research theme or field
application of exceptionally rigorous research design and techniques of investigation and analysis, and the highest standards of intellectual precision
instantiating an exceptionally significant, multi-user data set or research resource.
In assessing work as being three star (quality that is internationally excellent in terms of originality, significance and rigour but which falls short of the highest standards of excellence), is there evidence of, or potential for, some of the following types of characteristics:
an important point of reference in its field or sub-field
contributing important knowledge, ideas and techniques which are likely to have a lasting influence
application of robust and appropriate research design and techniques of investigation and analysis, with intellectual precision
generation of a substantial, coherent and widely admired data set or research resource.

14. How do we decide what is 3*/4*?
Proxies (e.g. ABS list)
Reading programme
REF panel members decide for themselves – not an exact science and they claim not to decide based on the journal!
WLM decisions based (primarily) around publication in journals rated 3*/4* on the ABS list

15. ABS list vs actual REF output scores
ABS 4
ABS 3
ABS 2
ABS 1
Not on ABS list
Book
Chapter
Total
REF 4 94 80 4 2 3 6 1
190
REF 3 95
296 29
433
REF 2 47
150 54 9 37
303
REF 1 28 10 21 74
239
554 97 18 67 16
1000
There were over twelve thousand outputs assessed by sub-panel 19 across 101 institutions, and these 1,000 output scores were probably contributed by all sub-panel members plus a few other evaluators, and is likely to be a fair representation of the outputs submitted.
About half the outputs got the same grade as their ABS journal rank, slightly more than a third scored below and about one in seven scored above.
Given that most outputs were in ABS 3 or 4* journals, there was more room on the scale to score them lower than their ABS ranking than there was to score them higher.
Implications for people working in universities might be:
An output in an ABS 4 journal was less likely to be scored a 4 than many thought beforehand (39% chance). There’s a 20% chance it will be a 2.
However, it’s likely that an ABS 3 will score at least a 3 (67% chance).
ABS 2s have a 30% chance of being a 3, and only a 10% chance of a 1. In future, it might be worth institutions being moderately bold about submitting 2s. [However, the ABS 2 articles submitted to REF may have been carefully selected, and therefore not typical of outputs in ABS 2* journals].
Books can get good scores. But presumably these too were carefully vetted by institutions.
Journals not on the ABS list did not score highly in REF. Is this us (or evaluators to whom outputs were cross-referred) not favouring inter-disciplinary research, or is it that a lot of these outputs really are weak? My personal output scores for non-ABS journal articles were quite a bit higher than in the table. Many of these were in mainstream journals in my discipline.
It’s worth institutions taking an intelligent look at their outputs before REF submission, and making judgments about their quality (originality, significance, rigour) that do not rely on ABS journal rankings. I'm pretty sure a bit of care in selection could eliminate a lot of the ABS 4s that were scored 2, and selective submission of ABS 2s can get a decent haul of 3s out of them.
Anecdotal: ABS list is predictive but inflates by about 20%

16. Environment and impact: your contribution
Research income
PGR students
Impact
Impact case study?
Expect to need about 8 for REF2020: must be underpinned by a minimum of 2* research
Anecdotal: the better the underpinning research the better the case study rating

17. Questions