1. What’s the good of REF? An insider-outsider perspective
Professor David James
University of Cardiff
Director, ESRC Wales Doctoral Training Partnership
Chair of REF2021 Sub-panel 23 (Education)

2. Five main issues The nature of REF and the rationale for REF
Sub-panel appointments – where we are and how it works
Key similarities & differences between 2021 and 2014
How the Education sub-panel will work
Education as a distinctive field and the constraints and affordances of REF

5. 1. The nature of REF and the rationale

6. What is REF - overview
Undertaken by the REF team on behalf of the four UK higher education funding bodies
Key purposes:
To provide accountability for public investment in research and produce evidence of the benefits of this investment.
To provide benchmarking information and establish reputational yardsticks, for use within the HE sector and for public information.
To inform the selective allocation of funding for research –ca. £2bn per year.
Assessed across three areas:
Outputs (e.g. publications, performance, prototypes) – 60%
Impact (the benefit of research for wider society) – 25%
Research environment (resources, strategies to support research) – 15%
Assessed by expert sub-panels (of senior academics and research users) in 34 disciplined-based ‘units of assessment’, overseen by 4 main panels

7. REF defines research in a specific way
From the Draft Guidance on Submissions "For the purposes of the REF, research is defined as a process of investigation leading to new insights, effectively shared. …It includes work of direct relevance to the needs of commerce, industry, culture, society, and to the public and voluntary sectors; scholarship; the invention and generation of ideas, images, performances, artefacts including design, where these lead to new or substantially improved insights…It excludes routine testing and routine analysis of materials, components and processes such as for the maintenance of national standards, as distinct from the development of new analytical techniques. It also excludes the development of teaching materials that do not embody original research…”

8. How is education research described?
Each Unit of Assessment has a ‘descriptor’
Education’s descriptor is inclusive, encompassing systems, issues, processes, provision and outcomes, substantive areas, a wide range of theoretical & methodological positions & disciplines.

9. 2021 framework Overall quality Outputs Impact Environment 60% 25% 15%
FTE x 2.5 = number of outputs required
Impact
Impact case studies
Environment
Environment data and template -
60%
25%
15%

10. UOAs Main panel Unit of assessment C 14
Geography and Environmental Studies 15
Archaeology 16
Economics and Econometrics 17
Business and Management Studies 18
Law 19
Politics and International Studies 20
Social Work and Social Policy 21
Sociology 22
Anthropology and Development Studies 23
Education 24
Sport and Exercise Sciences, Leisure and Tourism D 25
Area Studies 26
Modern Languages and Linguistics 27
English Language and Literature 28
History 29
Classics 30
Philosophy 31
Theology and Religious Studies 32
Art and Design: History, Practice and Theory 33
Music, Drama, Dance, Performing Arts, Film and Screen Studies 34
Communication, Cultural and Media Studies, Library and Information Management 
Main panel
Unit of assessment A 1
Clinical Medicine 2
Public Health, Health Services and Primary Care 3
Allied Health Professions, Dentistry, Nursing and Pharmacy 4
Psychology, Psychiatry and Neuroscience 5
Biological Sciences 6
Agriculture, Food and Veterinary Sciences B 7
Earth Systems and Environmental Sciences 8
Chemistry 9
Physics 10
Mathematical Sciences 11
Computer Science and Informatics 12
Engineering

11. 2. Sub-panel appointments – where we are and how it works

12. Expert panels
34 sub-panels working under the guidance of four main panels with advice from Equality and Diversity and Interdisciplinary Research advisory panels (EDAP and IDAP)
Two-stage appointment process (via nominations):
Criteria-setting phase – sufficient members appointed to ensure each sub-panel has appropriate expertise
Assessment phase – recruitment in 2020 of additional panel members and assessors to ensure appropriate breadth of expertise and number of panel members necessary for the assessment phase, informed by the survey of institutions’ submission intentions in 2019.

13. Expert panels Main panel responsibilities
Developing the panel criteria and working methods
Ensuring adherence to the criteria/procedures and consistent application of the overall assessment standards
Signing off the outcomes
Sub-panel responsibilities
Contributing to the main panel criteria and working methods
Assessing submissions and recommending the outcomes

14. Sub-panel appointments
So far, only about half the likely membership of the Education sub- panel has been appointed.
Remaining panel appointments following the 2019 ‘survey of submission intentions’.
HE institutions not allowed to nominate possible panel members. Learned societies and users of research can nominate.
So far, in respect of the Education Sub-panel, 158 nominations were made, pertaining to 127 distinct individuals. UCET nominated 14 of these.
Those already nominated are still ‘in the pool’. There will be an opportunity to make further nominations.

15. 3. Key similarities & differences between 2021 and 2014

16. Outputs Rigour Significance Originality
Assessed against three criteria:
Originality
the extent to which the output makes an important and innovative contribution to understanding and knowledge in the field
Significance
the extent to which the work has influenced, or has the capacity to influence, knowledge and scholarly thought, or the development and understanding of policy and/or practice
Rigour
the extent to which the work demonstrates intellectual coherence and integrity, and adopts robust and appropriate concepts, analyses, theories and methodologies
Scored one to four star (or unclassified)
Each main panel sets out its own understanding of the starred quality levels

17. Impact – submission
Impact remains eligible for submission by institution(s) where research was generated (i.e. non-portable)
Impact must be underpinned by research of minimum 2* quality
Timeframe:
1 January December 2020 for underpinning research
1 August July 2020 for impacts
Case studies continued from examples submitted in 2014 will be eligible for submission in REF 2021, provided they meet the same eligibility criteria

18. Impact – criteria Reach
Assessed against two criteria:
Reach
the extent and/or diversity of the beneficiaries of the impact, as relevant to the nature of the impact. (It will not be assessed in geographic terms, nor in terms of absolute numbers of beneficiaries.)
Significance
the degree to which the impact has enabled, enriched, influenced, informed or changed the performance, policies, practices, products, services, understanding, awareness or well-being of the beneficiaries.

19. Environment template More structured template Sections will include:
equality and diversity
approach to enabling impact
approach to supporting collaboration
structures to support interdisciplinary research
section on open research
‘Open research’ section will include submitting unit’s OA strategy, including where this goes above and beyond the REF OA policy requirements, and wider activity to encourage the effective sharing and management of research data. The panels will set out further guidance on this in the panel criteria.

20. Environment
Assessment criteria:
Vitality
the extent to which a unit supports a thriving and inclusive research culture for all staff and research students, that is based on a clearly articulated strategy for research and enabling its impact, is engaged with the national and international research and user communities and is able to attract excellent postgraduate and postdoctoral researchers.
Sustainability
the extent to which the research environment ensures the future health, diversity, well-being and wider contribution of the unit and the discipline(s), including investment in people and in infrastructure.

21. Key changes since REF 2014 Overall framework
Submission of all staff with significant responsibility for research
Transitional approach to non-portability of outputs
Decoupling of staff from outputs
Open access requirements
Additional measures to support interdisciplinary research
Broadening and deepening definitions of impact

22. Submissions – decoupling
Number of outputs per submission
Output pool to include
FTE of Cat A submitted
2.5
Number of outputs
Submitted outputs
Min of 1 per Cat A submitted
Max of 5 attributed to individuals
May include outputs of staff that have left
Multiplier was calculated using figures provided by HEIs in the staff submission with the aim of assessing roughly the same no. of outputs as in 2014.
This is an average across the unit, not the number required from each researcher (as has been claimed by some on twitter)
Fractional multiplier also underlines decoupling from individual researcher
Rounding (to the nearest whole number) will be applied to give a whole number of outputs for return.

23. Interdisciplinary advisers
oversee application of agreed principles and processes
Interdisciplinary Research Advisory Panel
facilitate cross-panel liaison
oversee calibration exercise for IDR outputs
Main panel interdisciplinary leads
Offer guidance to sub-panels on assessment of IDR outputs
Liaise with advisers on other panels
Sub-panel interdisciplinary advisers

24. Impact – types and indicators
Panels welcome case studies that describe any type(s) of impact
Panel will welcome, and assess equitably, case studies describing impacts achieved through public engagement, either as the main impact described or as one facet of a wider range of impacts.
Impact on teaching within (and beyond) own HEI is eligible
Case studies must provide a clear and coherent narrative supported by verifiable evidence and indicators
Should provide evidence of reach and significance of the impacts, as distinct from evidence of dissemination or uptake
Annex A includes an extensive – but not exhaustive – list of examples of impact and indicators, including evaluation frameworks from non-HE organisations

25. Professor Brian Cox, OBE
His extensive and important public engagement work was not easily submitted as a case study in REF 2014

26. 4. How the Education Sub-panel will work

27. Some key elements of sub-panel process
Calibration
Allocation
Assessment
Moderation
Collective responsibility
International comparison
Avoiding conflicts of interest

28. Educationalists note: our discipline has the conceptual tools to understand REF
REF is a process of FRESH peer review
The definition of research and the criteria are both written down, but there are limits to such codification (Cf. Eraut, or Sennett) and anyway, their meaning is a matter of negotiation, participation and reification in a community of practice (to use Lave and Wenger’s terms)
Parallels with well-known work on HE assessment (e.g. Sadler, or Boud, and many others) – judgement of quality is always more than the rigid application of well-defined criteria. Connoisseurship too.

29. 5. Education as a distinctive field and the constraints and affordances of REF

31. Mills, Oancea & Robson (2017) The Capacity and Impact of Education Research in the UK
Amongst other things, highlights the shape of the field:
Although ‘the total number of REF-eligible academic staff in Education in 2014 was 5255 FTE spread across 109 institutions…the total submitted to the REF exercise was 1442 in 76 submissions’. This = either 27% or 31% depending on the calculation used.
Submissions included between 3.8 and 219 FTE.
Diversity of professional and academic experience…about one quarter of a sample of REF 2014 research active staff had a background in teaching and/or held teaching qualifications
Not all education research is done in Education departments/faculties

32. British Academy/Royal Society (2018) Harnessing educational research
Adopts an ‘ecosystem’ perspective – flows of people, funding and information Notes a need for the teaching profession to be research-informed Offers eight recommendations to governments, UKRI and constituent bodies, universities, learned societies, regulators and professional bodies Finds that although two-thirds of education research was ‘world leading’ or ‘internationally excellent’ (REF 2014), there are ‘…disconnects between supply and demand’ Recommends an Office for Education Research (perhaps rather like the OSCHR)

33. Very good things to remember
There is both a generic REF definition of research AND an Education Sub-panel descriptor of Educational Research. The latter is broad. Both are important.
Institutional processes for estimating the quality of research outputs are often a poor proxy what what happens in REF.
It’s not meant to be about you. The serious intention of REF is to measure the quality of research in institutions – NOT that of individuals.
The panel will have been chosen carefully to reflect the substantive, methodological and epistemological breadth of the field. They will also all have relevant experience.
They will be of the field, AND will work in a collegial manner for the field.

34. To paraphrase Winston Churchill somewhat:
“Expert peer review is the worst possible system for assessing research quality, apart from all the others”.