Assessment of Law Faculties in the UK Nigel Duncan Professor of Legal Education

Who is assessing UK Law Faculties? Student Fees Formal Informal Professional Regulators eg Complete University Guide https://www.thecompleteuniversityguide.co.uk/league-tables/rankings?s=law Research Excellence Framework Law Faculty Employers Quality Assurance Agency Students and Parents Teaching Excellence Framework National Student Survey

Research Excellence Framework Discipline-specific. Last ran in 2014. Probably next in 2021. Recently reviewed by the Stern Report. Findings inform the allocation of central research funding. Institutions prepare a submission addressing: Outputs – 65% Impact – 20% Research environment – 15%

REF: Outputs Each research-active academic should produce at least four refereed publications at the requisite standard. 4* Quality that is world-leading in terms of originality, significance and rigour. 3* Quality that is internationally excellent in terms of originality, significance and rigour but which falls short of the highest standards of excellence. 2* Quality that is recognised internationally in terms of originality, significance and rigour. 1* Quality that is recognised nationally in terms of originality, significance and rigour.

REF: Impact 4* Outstanding impacts in terms of their reach and significance. 3* Very considerable impacts in terms of their reach and significance. 2* Considerable impacts in terms of their reach and significance. 1* Recognised but modest impacts in terms of their reach and significance. Examples from City Law School: Immigration detention law and policy Impunity and the Rule of Law: the Safety of Journalists National and international development and reform of the law of criminal evidence

REF: Environment For example: An environment that is conducive to producing research of: 4* world-leading quality, in terms of its vitality and sustainability. 3*  internationally excellent quality, in terms of its vitality and sustainability. 2*  internationally recognised quality, in terms of its vitality and sustainability 1*  nationally recognised quality, in terms of its vitality and sustainability For example: the number of research doctoral degrees awarded external research income

REF: Critique “the REF has become a monster, a Minotaur that must be appeased by bloody sacrifices.” Peter Scott,  Professor of higher education studies, Institute of Education, The Guardian, 4 Nov 2013. Jean-Etiennes Ramey £246 million

Teaching Excellence Framework (TEF) The Government says it has introduced TEF as a way of: Better informing students’ choices about what & where to study Raising esteem for teaching Recognising and rewarding excellent teaching Better meeting the needs of employers, industry, business & the professions Staged introduction: Year 1 (2017) All institutions graded: ‘Meets expectations’. Year 2 (2018) Institutions graded on basis of a series of factors – institution-level only. Thereafter: subject-level assessment to be introduced.

TEF: Assessment: Metrics plus a Narrative Submission The Metrics Teaching on my course (NSS Q 1-4 ) Assessment and Feedback (NSS Q5-9 ) Academic Support (NSS Q10-12) Non-Continuation (HESA & ILR Data ) Employment or Further Study (DLHE 6 months p.g.) Highly skilled employment or further study (DLHE) http://www.thestudentsurvey.com/content/NSS2017_Core_Questionnaire.pdf Data is benchmarked to allow meaningful comparisons between institutions by taking into account the different mix of students at each provider. Results that are significantly different from benchmark are highlighted. 2 percentage points above benchmark = one flag 3+ percentage points above benchmark = a double flag Similarly, if below benchmark (one or two negative flags)

The Narrative Submission Purpose: to enable the provider to: Add additional context e.g. mission Support or explain performance against the metrics Put forward evidence against the assessment criteria Further explore performance for specific student groups based on split metrics. Assessment Criteria: Teaching Quality Learning Environment Student Outcomes & Learning Gain TEF Descriptors (bronze, silver, gold)

Narrative Submission (2) Narrative must demonstrate the impact & effectiveness of teaching on student experience and the outcomes they achieve. Wherever possible, impact should be demonstrated empirically. No prior knowledge or external evidence can be taken into account. Teaching & learning excellence must be demonstrated across an institution’s entire provision (no pockets of excellence). Evidence should normally be from the last 3 years. (Not future promises)

TEF: Critique “Deeply-flawed and impractical” Prof Sir Keith Burnett, VC University of Sheffield “The TEF will not serve students, but an ill-defined set of employers” Dr. Joshua Forstenzer, VC Fellow, U. Sheffield “the TEF … risks increasing the bureaucracy of modern university life; and the management culture which is displacing an older and more noble idea of the university. We want academics to teach in the spirit of the community of which they are the citizens and makers, not to teach as they are directed to teach by bureaucrats.” Emran Mian, Director, Social Market Foundation “A Trojan Horse for raising fees” Gordon Marsden MP Domenico Tiepolo

National Student Survey Final year undergraduate only. 27 Questions – on a six-point Likert scale The teaching on my course Learning opportunities Assessment and feedback Academic support Organisation and management Learning resources Learning community Student voice Overall satisfaction Also: Higher Education Academy: Postgraduate Taught Experience Survey; Postgraduate Research Experience Survey

Assessment and Measurement Increasing tendency to demand metrics to assess the quality of a faculty’s work. Only valuing what is measurable Metrics in tension, ie High score in the REF diverts funds from teaching and learning Therefore funding research is in tension with financial sustainability Makes it difficult to adopt a holistic approach to managing a faculty Do we have the capacity to learn to measure what we value? https://svpow.files.wordpress.com/2017/02/c49rdmlweaaa4if.jpg

Odysseus and Polyphemus Measurement is inevitable. Can we use it to achieve our goals and find ways of avoiding its damaging consequences? Jacob Jordaens

Acknowledgement, links, references My thanks to Fiona Cownie, Pro Vice Chancellor (Education & Student Experience) Keele University, for content on TEF. Stern Report: https://www.gov.uk/government/publications/research-excellence-framework-review Derek Sayer: Rank Hypocrisies: the Insult of the REF. London, SAGE, 2015 https://www.timeshighereducation.com/news/teaching-excellence-framework-tef-everything-you-need-to-know http://www.hefce.ac.uk/lt/dlhe/ Stefan Collini, Who are the spongers now?, London Review of Books, 2016, Vol 38, No 2, p.33-37 Edwards, Marc A., and Siddhartha Roy. 2017. Academic Research in the 21st Century: Maintaining Scientific Integrity in a Climate of Perverse Incentives and Hypercompetition. Environmental Engineering Science 34(1):51-61.