1. 1 ALTERNATIVE METHODS FOR MEASURING EFFICIENCY AND AN APPLICATION OF DEA IN EDUCATION Emmanuel Thanassoulis Professor in Management Sciences Aston Business School University of Aston Birmingham B4 7ET Tel: +44(0) 121 3593611 Ext 5033 Fax: +44 (0) 121 359 5271 Email: e.thanassoulis@aston.ac.uk

2. 2 PRESENTATION OUTLINE l How Data Envelopment Analysis (DEA) works in outline. l A conceptualisation of comparative efficiency measurement l How Corrected and Modified OLS regression work. l How Stochastic Frontier Analysis (SFA) works. l Decomposing pupil attainment between school and pupil effects using DEA. l Managerial information derived through decomposing pupil and school effects using DEA. PART 1: OVERVIEW OF COMPARATIVE EFFICIENCY ASSESSMENT METHODS PART 2: A REAL LIFE APPLICATION OF DEA ON BEHALF OF THE UK DEPARTMENT FOR EDUCATION AND SKILLS

3. 3 PART 1: OVERVIEW OF COMPARATIVE EFFICIENCY ASSESSMENT METHODS An Introduction to Efficiency and Productivity Analysis by Tim Coelli, D S Prasada Rao and George Battese (1998) Kluwer Academic Publishers (chapters 2, 3, 8, 9) ISBN: 0792380622 S.C. Kumbhakar and C.A. Knox Lovell (2000) Stochastic Frontier Analysis (Cambridge University Press) ISBN: 0521481848 Greene W H (1998) Frontier Production Functions in H. Peasaran and P. Schmidt (eds) Handbook of Applied Econometrics, volume 2, microeconomics (Blackwells) Useful references on parametric efficiency assessment methods:

4. 4 THE PROBLEM OF COMPARATIVE EFFICIENCY As an integral part of managing organisations in the public and private sector we need to have information such as: - How efficiently are the operating units using their resources? - Is the industry such that there are economies of scale? - Is the industry such that there are economies of scope? - What are the efficient marginal costs of outputs; - Has there over time been productivity change within the industry? - How has a given operating unit fared on productivity change?

5. 5 THE FUNDAMENTAL VIEW OF THE PROBLEM InputsOutputs Transformation The units to be assessed transform inputs into outputs The basic requirement is to compare the Decision Making Units (DMUs) on the levels of outputs they secure relative to their input levels.

6. 6 MEASURES OF COMPARATIVE EFFICIENCY InputsOutputs Transformation In a given operating context the measure of efficiency is normally one of: - The distance between observed and maximum possible output for given inputs (output efficiency); - The distance between observed and minimum possible input for given outputs (input efficiency);

7. 7 There are two broad types of method for arriving at measures of comparative efficiency: parametric and non-parametric methods. The parametric methods typically hypothesise a functional form and use the data to estimate the parameters of that function. The estimated function is then used to arrive at estimates of the efficiencies of units. The non-parametric methods, best known as Data Envelopment Analysis (DEA), create virtual units to act as benchmarks for measuring comparative efficiency.

8. 8 Hypothesise a production function. E.g. the expression PARAMETRIC METHODS FOR COMPARATIVE EFFICIENCY MEASUREMENT Where y is output, x ik are inputs, and e i is the residual for firm i. It is the residual e i the captures any inefficiency. Unfortunately, the residual also captures other random effects (e.g. omitted variables, measurement error, etc.) which makes it difficult to disentangle the component of inefficiency. [1] Two approaches exist: ignoring and not ignoring the random effects.

9. 9 OLS regression assumes implicitly the u i have zero mean. Let us treat the residual e i in [1] as capturing ONLY inefficiency ignoring other random effects. Then the model becomes Now suppose we use OLS regression to estimate the model in [2]. u i >= 0. Therefore we need to adjust the OLS model, to get the true residuals u i. Normally one of two types of adjustment is used: Corrected or Modified OLS. [2] where IGNORING RANDOM EFFECTS IN THE RESIDUAL

10. 10 The OLS model we estimated when we assumed residual mean of zero was in effect model [2] implicitly modified to They work as follows. Let a* = [a-E(u i )] be the intercept of [3] that OLS regression yields. In order to retrieve the true underlying model in [2] we need to add E(u i ) to a*. The problem is that E(u i ) is an estimate of the mean inefficiency of firms which we do NOT know. = To get an estimate of the mean inefficiency of firms we make an assumption about the theoretical probability density function of u i and then use the OLS regression residuals to estimate. [3]

11. 11 If u i is assumed to follow a half-normal distribution COLS. Adjust the intercept a* of the OLS model in [3] to Three estimates of normally are used as follows: where e i is the residual for the ith firm in model [3] (OLS model). MOLS - half normal adjust the intercept a* of the OLS model in [3] to where If u i is assumed to follow an exponential distribution MOLS - exponential adjust the intercept a* of the OLS model in [3] to where

12. 12 The exponential and half-normal assumption reflect the belief that larger values of inefficiency are less likely.

13. 13 MOLS half-normal log gdp/worker Illustrative application of COLS/MOLS to a set of data. Adapted from Weyman-Jones lecture notes, Aston Business School Inefficiency of country A: The colour of the arrow identifies the referent boundary A

14. 14 NOT IGNORING RANDOM EFFECTS IN THE RESIDUAL: Stochastic Frontier Analysis (SFA) Key departure from COLS and MOLS is that we now have a composed error term v is an identically distributed conventional two sided error term with zero mean. It stands for random noise, omitted variables etc. u is an identically distributed one sided error term with a non-zero mean. It stands for inefficiency. u is typically assumed to be exponential, half-normal or truncated normal.

15. 15 Illustration of the composed error term  i = v i - u i when u i is half normal. Source Tom Weyman Jones, lecture notes Aston Business School. Note the negative skew. We use a test for negative skew to decide whether there is evidence of inefficiency among the producers. If we reject the SFA model.

16. 16 The SFA model is usually fitted using Maximum Likelihood estimation. We need to estimate the inefficiency of the ith producer (u i ) by using its composed residual   = v i - u i. Depending on the assumption we make about the distribution of the inefficiency u i we arrive at a different formula for the conditional value We plug into this formula the values of   and other values we derive from the   to arrive at an estimate of the conditional inefficiency u i of the ith producer. The formulae differ depending on the distribution assumed for u i but are coded in available software such as Limdep.

17. 17 A GRAPHICAL OUTLINE OF THE BASIC DEA MODEL FOR ASSESSING COMPARATIVE EFFICIENCY An introduction to Data Envelopment Analysis can be found in: E. Thanassoulis (2001) Introduction to the Theory and Application of Data Envelopment Analysis: A foundation text with integrated software. Kluwer Academic Publishers, Boston, Hardbound, ISBN 0-7923-7429-0

18. 18 A Set of Observed Operating Units as Input -Output Correspondences

19. 19 By virtue of interpolation between C and D all input - output correspondences on CD are feasible

20. 20 Using interpolations between observed units the set of all feasible input -output correspondences is constructed and its boundary identified

21. 21 Using the set of all feasible input output correspondences the comparative efficiency and other information in respect of a unit (e.g. unit E) is derived as illustrated here:  Output Efficiency of E: FE/FG  Output benchmarks for E: Units C and D  Scope for output augmentation at E: EG  Returns to scale (increasing, decreasing, constant): Revealed by the intercepts of the segments of the efficient boundary.  Scale elasticity revealed by the slope of the segments on the efficient boundary.  Input Efficiency of E: HI/HE  Scope for resource conservation at E: IE

22. 22 Contrasting the Alternative Efficiency Assessment Methods l All depend on identifying a reference boundary relative to which efficiency is assessed. l DEA is non-parametric, can handle multiple inputs and outputs but assumes all distance from the boundary is inefficiency. l SFA allows for random noise in the distance from the boundary but needs assumptions on inefficiency distribution; l DEA reveals unit-specific peers, type of returns to scale, productivity change. l SFA and regression methods reveal industry level information. l COLS/MOLS can be very susceptible to outliers and do not allow for random noise. 246810 2 4 6 8 12 0 Input A B E F D C G H DEA OLS Regression Output SFA COLS

23. 23 PART 2: A REAL LIFE APPLICATION OF DEA ON BEHALF OF THE UK DEPARTMENT FOR EDUCATION AND SKILLS

24. 24 Most studies measuring school performance use school level data and are parametric (regression-based). One non - DEA approach which uses pupil - level data is Multi-level Modelling. It recognises the nested nature of school data (pupil within school, school within LEA etc.). In a similar manner our approach uses pupil level data in a DEA framework to decompose pupil attainment into any number of components (pupil, school, type of school, gender etc.) DECOMPOSING PUPIL ATTAINMENT USING DEA It decomposes variance in pupil attainment into pupil, school etc. effects.

25. 25 How the Decomposition Works Take pupils in a number of schools. We have a two-level model [pupil is level 1, school is level 2]. Any difference in attainment by pupils is as a result of a combination of: - Random Noise; - School effectiveness; - Differences in effort made by pupils. Our approach allows for random noise and attempts to separate school from pupil effects.

26. 26 How the Decomposition Works BCD - Pupil-within-all-schools efficient boundary BCH and GD Pupil- within-school efficient boundaries OZ/OZ’ = Pupil-within-school efficiency measure of pupil Z. OZ/OZ’’ = Pupil-within-all-schools efficiency measure of pupil Z. OZ/OZ’’ = OZ/OZ’ * OZ’/OZ’’ OZ’/OZ’’ is a measure of school-within- all-schools efficiency at pupil Z The graph shows data from school 1(dots) and school 2 (crosses)

27. 27 Pupil-Within-All-Schools DEA efficiency = Component attributable to pupil  Component attributable to school

28. 28 Application of the Methodology l Pupils of 122 schools who sat GCSE’s in 1992 and A or AS levels in 1994 were assessed. DEA Input - Output Variables:

29. 29 Estimation of DEA efficiencies Given a set of n pupils, the DEA efficiency of pupil j 0 relative to that set is 100/  j0 * % where  j0 * is the optimal value of  in: Max  S.t. GCSEpts j j <= GCSEpts j0 GCSEpts_att j j < = GCSEpts_att j0 Aptsj j > =  Apts j0 Apts_att j j > =  Apts_att j0 j = 1 j,  0,  free

30. 30 Computing Within-School and Within-all-Schools Efficiencies 3. Pool together the pupils of all schools, including the output - adjusted pupils of step 2  Computation of within -all-schools efficiencies Computation of within -all-schools efficiencies 2. Scale radially down the outputs of outlier pupils to place them on the final “incumbent” DEA boundary and combine them with the rest of the pupils of their school. 1. Taking the pupils of one school at a time, eliminate ‘outlier’ pupils whose attainment is deemed exceptional and so not to be used as a benchmark  Computation of within-school (“pupil” ) efficiencies

31. 31 Decomposing Pupil-within-All Schools Efficiencies: EFF wj0 - pupil-within-school efficiency EFF ij0 - pupil-within-all-schools efficiency  EFF wj0 EFF sjo - school-within-all-schools efficiency EFF ij0 = EFF sjo * EFF wj0

32. 32 Summary Statistics of Efficiency Scores Results obtained considering all the pupils sampled irrespective of school: -The median pupil attains only 65% of the A-level scores of the benchmark pupil(s) in his/her own school; -The median pupil attains only 46.6% of the A-level scores of the benchmark pupil(s) across schools -The median pupils benchmark within own school attain only 82.76% of the benchmark pupils across schools.

33. 33 Attribution of efficiencies Mean school-within-all-schools efficiency: School 121 93.8% School 60 60.56%.

34. 34 - School 121 is more effective with stronger pupils - School 2 is more effective with weaker pupils Identifying Differential School Effectiveness A school has differential effectiveness if it has a different effect on different groups of pupils. GCSE points per attempt can be taken as indicative of the innate ability of a pupil

35. 35 Pupil targets and peers Pupils targets can incorporate several components of interest both to the pupil and to the school : - Within-all-schools targets - A pupil can reach these by a combination of reaching within school targets while his/her school becomes more effective. (Within-all-schools peers can indicate role model schools.) - Within-school targets - achievement of these targets depends only on the pupil. (Peers are within-school.)

36. 36 Other Decompositions Possible We could in effect control for any number of categorical variables and estimate the impact of each one on pupil attainment: E.g: l Gender ( Estimate efficient boundaries within each gender and then pool the genders to compare the distance of the two boundaries.) l Type of school ( Estimate efficient boundaries within each type of school and then pool the types of school to compare the distance of the boundaries.) l Socio-economic factors ( Estimate efficient boundaries within each group of pupils (e.g. eligible v non-eligible for free school meals) and then pool the samples to compare the distance of the boundaries.) l Combinations of categorical variables (E.g. Gender and eligibility for free school meals) Estimate efficient boundaries within each combination of factors and then pool the samples to compare the distance of the boundaries.)

37. 37 Conclusion l The approach outlined recognises that pupil attainment is a combination of various effects including effort by the pupil, the school and other categorical factors. l It can provide more complete information to schools, teachers and other parties in managing school and pupil performance. Eg: - targets for individual pupils using other pupils within the school as benchmarks; - targets for schools to raise their boundary closer to the inter-school boundary; - identification of any differential school effectiveness with a view to its elimination or management.

38. 38 Thank you