1. Triple Accredited | World ranked EFFICIENCY IN ENGLISH HIGHER EDUCATION Jill Johnes Department of Economics, Lancaster University UK

2. Triple Accredited | World ranked Outline of talk 1.Introduction Main developments in English higher education Composition of the English higher education sector today 2. Efficiency and its measurement a) Stochastic frontier analysis (SFA) b) Data envelopment analysis (DEA) 3.An empirical study 4.Policy issues a) Comparison of various efficiency measures and rankings b) Mergers 5. Conclusions

3. Triple Accredited | World ranked 1. Introduction Timeline: English higher education (HE) from 1963 1963: Robbins Report led to expansion of HE in England 1986: First Research Assessment Exercise (RAE) 1992: Abolition of divide between universities and polytechnics 1997: Quality Assurance Agency (QAA) established to assess HE teaching 1998: Undergraduate (UG) tuition fees set at £1000 1999: Tony Blair aims for 50% of young people in HE by 2010 2003: Colleges of Higher Education allowed to apply for university status 2006: Introduction of variable UG tuition fees with ceiling of £3000 2008: Research Assessment Exercise 2012: Ceiling on variable UG tuition fees raised to £9000 2013: Research Excellence Framework

4. Triple Accredited | World ranked 1. Introduction Composition of the English higher education sector The current English higher education sector therefore comprises very diverse groups of HEIs: Pre-1992 universities Traditional HEIs including Oxford and Cambridge, and universities established in the 1960s; they offer traditional programmes and subjects and have a research mission Post-1992 universities Former polytechnics which offer a range of programmes including vocational degrees; some also have a research mission Former colleges of HE Often small, specialist HEIs; often do not have a research mission; allowed to obtain university status since 2004

5. Triple Accredited | World ranked 2. Efficiency and its measurement Why measure the efficiency of English higher education? Assessing the performance or efficiency of higher education institutions (HEIs) is of interest to: - potential students and their parents - HEI managers - government (since HEIs receive public funding)

6. Triple Accredited | World ranked 2. Efficiency and its measurement Official performance indicators The Higher Education Statistics Agency (HESA) has published performance indicators since 1999 ‘Performance indicators in higher education (HE) provide information on the nature and performance of the HE sector in the UK. They are intended as an objective and consistent set of measures of how a higher education institution is performing.’ The indicators currently cover the following data: widening participation indicators non-continuation rates module completion rates research output employment of graduates

7. Triple Accredited | World ranked 2. Efficiency and its measurement Official performance indicators Research Excellence Framework – REF (formerly Research Assessment Exercise – RAE) - evaluation of research quality in British Universities. The research of each subject is ranked by a peer review panel and the rankings are used in the allocation of government funding. Quality Assurance for Higher Education (QAA) assesses undergraduate teaching.

8. Triple Accredited | World ranked 2. Efficiency and its measurement Unofficial performance indicators Newspapers regularly provide rankings of higher education institutions (HEIs). For example - The Times Good University Guide - RIA Novosti/HSE ranking of Russian Universities - Forbes ranking of Russian Universities - Times Higher Education World University Rankings - The QS World University Rankings

9. Triple Accredited | World ranked 2. Efficiency and its measurement The Times Good University Guide: 8 measures The universities are compared on ‘8 well-established measures of importance to the undergraduate experience’. 1.Student satisfaction 2.Research quality 3.Entry standards 4.Student-staff ratio 5.Services and facilities spending 6.Completion 7.Good honours 8.Graduate prospects

10. Triple Accredited | World ranked 2. Efficiency and its measurement Problems with these rankings HEIs operate under different conditions and in different environments. HEIs produce an array of outputs from a range of inputs. A ratio of one output to one input is, at best, only a partial indicator. What is the appropriate way to combine information across a variety of indicators? We need a method for measuring efficiency which takes into account the multi-dimensional nature of production in HEIs.

11. Triple Accredited | World ranked 2. Efficiency and its measurement A multi-dimensional measure of efficiency Inputs ‘Raw materials’ Labour Capital Black Box Outputs Teaching Research Social

12. Triple Accredited | World ranked 2. Efficiency and its measurement A multi-dimensional measure of efficiency Teaching/staff Research/staff Production possibility frontier (PPF) O

13. Triple Accredited | World ranked 2. Efficiency and its measurement A multi-dimensional measure of efficiency Teaching/staff Research/staff F F’ O Efficiency = OF/OF’

14. Triple Accredited | World ranked 2. Efficiency and its measurement Ordinary least squares regression Teaching/staff Research/staff F F’ O Estimated efficiency = OF/OF’ Estimated inefficiency = FF’/OF’ i.e. based on the OLS residual

15. Triple Accredited | World ranked 2. Efficiency and its measurement Stochastic frontier analysis (SFA) Aigner, Lovell and Schmidt (1977) The error term is split into 2 components: - a random error component (as in OLS) - a half-normally distributed component to reflect efficiency

16. Triple Accredited | World ranked 2. Efficiency and its measurement Stochastic frontier analysis Teaching/staff Research/staff F F’ O

17. Triple Accredited | World ranked 2. Efficiency and its measurement Data envelopment analysis (DEA) Charnes, Cooper & Rhodes(1978) DEA estimates a piecewise linear frontier using linear programming methods No error term; no equation

18. Triple Accredited | World ranked 2. Efficiency and its measurement Data envelopment analysis Teaching/staff Research/staff F F’ O

19. Triple Accredited | World ranked 3. An empirical study Inputs and outputs Inputs Primary inputs: PGINPUT (x1): Numbers on postgraduate programmes UGINQUAL (x 2 ): Numbers on undergraduate programmes weighted by average A level score (UGINPUT*QUAL) Labour: STAFF (x 3 ): Number of FTE academic staff ADMIN (x 5 ): Expenditure on administration including staff Capital: ACSERV (x 4 ): Expenditure on library and computing facilities Black Box Outputs Teaching: PGOUTPUT (y1): Graduates from postgraduate programmes UGOUTQUAL (y 3 ): Graduates from undergraduate programmes weighted by degree result (UGOUTPUT*DEGREE) Research: RESEARCH (y 2 ): Income received in funding council grants plus income received in research grants and contracts

20. Triple Accredited | World ranked 3. An empirical study Estimation methods a) SFA Time Invariant (TI): Efficiency for each university does not vary over time b) SFA Time varying (TV): Efficiency for each university does vary over time a) DEA Constant Returns to Scale (CRS) - i) Estimation across the pooled data set - ii) Estimation within each year b) DEA Variable Returns to Scale (VRS) - i) Estimation across the pooled data set - ii) Estimation within each year

21. Triple Accredited | World ranked 3. An empirical study Data Higher Education Statistical Agency (HESA) data Unbalanced panel of data from 1996/97 to 2008/09 with n = 1444 (the number of HEIs varies from 108 to 113 in each year) Diverse groups of HEIs: Pre-1992 universitiesn=624 Post-1992 universitiesn=375 Former colleges of HEn=445 All money units in 2008 values

22. Triple Accredited | World ranked 3. An empirical study Outputs

23. Triple Accredited | World ranked 3. An empirical study Inputs

24. Triple Accredited | World ranked 3. An empirical study Inputs by HEI Type

25. Triple Accredited | World ranked 3. An empirical study Outputs by HEI Type

26. Triple Accredited | World ranked 3. An empirical study Efficiencies SFAMeanSDMinMax a) TI0.8030.0970.5150.987 b) TV0.8010.0970.4790.990 DEA a) CRSi) pooled0.7530.0900.3411.000 a) CRSii) within year0.8690.0830.4221.000 b) VRSi) pooled0.8270.0890.4291.000 b) VRSii) within year0.9320.0670.5471.000

27. Triple Accredited | World ranked 3. An empirical study Mean efficiencies over time

28. Triple Accredited | World ranked 3. An empirical study Mean efficiencies by HE type

29. Triple Accredited | World ranked 4. Policy issues Are these efficiency measures correlated? SFADEA Modela) TIb) TV a) CRS i) pooled a) CRS ii) within year b) VRS i) pooled SFA b) TV0.987 DEA a) CRSi) pooled0.4220.466 a) CRSii) within year0.2680.2960.712 b) VRSi) pooled0.4850.4940.7670.634 b) VRSii) within year0.3060.3320.5830.7450.657

30. Triple Accredited | World ranked 4. Policy issues Are these efficiency measures correlated with other measures? ModelVC Pay SFA a) TI0.046 b) TV-0.002 DEA a) CRSi) pooled-0.059 a) CRSii) within year-0.008 b) VRSi) pooled *0.192 b) VRSii) within year-0.026 Vice Chancellor Pay

31. Triple Accredited | World ranked 4. Policy issues Are our efficiency measures correlated with other measures? ModelVC PayTimes SFA a) TI0.046 -0.029 b) TV-0.002 -0.024 DEA a) CRSi) pooled-0.059*-0.248 a) CRSii) within year-0.008-0.163 b) VRSi) pooled *0.192 -0.048 b) VRSii) within year-0.026-0.100 Times Good University Guide Correlation between VC Pay and Times is 0.205

32. Triple Accredited | World ranked 4. Policy issues Merger activity Pre-mergerPost-mergerNon-merging N=142N=133N=1169 SFA a)TI0.7970.8900.793 b) TV0.8060.8820.791 DEA a) CRS i) pooled0.7640.7800.748 a) CRS ii) within year0.8730.8990.865 b) VRS i) pooled0.8330.8810.820 b) VRS ii ) within year0.9430.9540.929

33. Triple Accredited | World ranked 4. Policy issues Outputs of pre-, post- and non-merging HEIs

34. Triple Accredited | World ranked 4. Policy issues Inputs of pre-, post- and non-merging HEIs

35. Triple Accredited | World ranked 5. Conclusions Ratios which claim to measure efficiency or performance of universities (such as those found in media rankings) should be treated with caution as they do not adequately reflect the university context DEA and SFA can provide multi-dimensional measures of efficiency DEA and SFA provide different estimates of the levels of efficiency in English higher education Policy-makers should be wary of using the DEA within-year estimation approach Rankings of DEA and SFA efficiency scores are significantly positively related, but correlations are low Merging HEIs appear to have higher efficiency than non-merging HEIs Further analysis is necessary to discover whether this is caused by the merger or other characteristics