Operational Conditions in Regulatory Benchmarking – A Monte-Carlo Simulation Stefan Seifert & Maria Nieswand Workshop: Benchmarking of Public Utilities November 13, 2015, Bremen

1 Motivation and Literature 2 Methodologies 3 The DGP 4 Simulation Design and Performance Measures 5 Initial Results 6 Conclusion and Outlook Agenda Stefan Seifert & Maria Nieswand 2 Benchmarking of Public Utilities

Motivation 1 Stefan Seifert & Maria Nieswand Benchmarking of Public Utilities 3 Regulatory Approaches for Electricity DSOs Source: Agrell & Bogetoft, 2013

Motivation Benchmarking widely used in regulation – sectors in which environmental factors play an important role Accuracy of estimates influences revenue caps, industry performance, firm survival, and ultimately customers via prices Methodological advances to account for environmental factors and heterogeneity Non-parametric approaches: z-variables in 1-stage DEA (Johnson and Kuosmanen, 2012), conditional DEA (Daraio & Simar, 2005 & 2007), … Parametric approaches: Latent Class (Greene, 2002; Orea & Kumbhakar, 2004), Zero-inefficiency SF (Kumbhakar et al., 2013), … Semi-parametric approaches: StoNEzD (Johnson & Kuosmanen, 2011), … BUT: Regulatory models typically based on standard DEA or SFA 1 Stefan Seifert & Maria Nieswand Benchmarking of Public Utilities 4

Motivation Aim of this study Systematical performance evaluation of Latent Class, StoNEzD and conditional DEA in the presence of environmental factors Generalization of results via Monte-Carlo-Simulation  Guidance for regulators to choose estimators given industry structure and industry characteristics Scope of this study Consideration of different model set-ups imitating real regulatory data  Cross section with variation in sample sizes, noise and inefficiency distributions and in terms of the true underlying technology  Consideration of different cases of impact of environmental variables 1 Stefan Seifert & Maria Nieswand Benchmarking of Public Utilities 5

Related Literature 1 Stefan Seifert & Maria Nieswand Benchmarking of Public Utilities 6

Stefan Seifert & Maria Nieswand 7 Benchmarking of Public Utilities Methodologies 2

Methodology – Notation 2 Stefan Seifert & Maria Nieswand Benchmarking of Public Utilities 8

Methodology – conditional DEA 2 Stefan Seifert & Maria Nieswand Benchmarking of Public Utilities 9

Methodology – Latent Class 2 Stefan Seifert & Maria Nieswand Benchmarking of Public Utilities 10

Methodology – Latent Class 2 Stefan Seifert & Maria Nieswand Benchmarking of Public Utilities 11

Methodology – StoNEzD 2 Stefan Seifert & Maria Nieswand Benchmarking of Public Utilities 12

Methodology – Comparison of cDEA, LC SFA and StoNEzD for production function 2 Stefan Seifert & Maria Nieswand Benchmarking of Public Utilities 13 cDEALC SFAStoNEzD TypeNon-parametricParametricSemi-parametric Error / InefficiencyDeterministicStochastic ShapeConstrainedParametrically constrained Constrained Scaling assumption NecessaryPossible Convexity of TYesNoYes Reference setObservation specificAll observations, weighted All observations Effect of z on frontier Observation specificGrouped, but observation specific via weighting General effect

Stefan Seifert & Maria Nieswand 14 Benchmarking of Public Utilities The DGP 3

Data Generating Process 3 Stefan Seifert & Maria Nieswand Benchmarking of Public Utilities 15

Data Generating Process 3 Stefan Seifert & Maria Nieswand Benchmarking of Public Utilities 16

Data Generating Process Environmental Factors4 different distributions considered, 1 symmetric, 3 skewed, 1 correlated with inputs 3 Stefan Seifert & Maria Nieswand Benchmarking of Public Utilities 17

Stefan Seifert & Maria Nieswand 18 Benchmarking of Public Utilities Simulation Design and Performance Measures 4

Simulation Design 3 Stefan Seifert & Maria Nieswand Benchmarking of Public Utilities 19

Simulation Design 3 Stefan Seifert & Maria Nieswand Benchmarking of Public Utilities 20

Performance measures 3 Stefan Seifert & Maria Nieswand Benchmarking of Public Utilities 21

Stefan Seifert & Maria Nieswand 22 Benchmarking of Public Utilities Initial Results 5

5 Stefan Seifert & Maria Nieswand Benchmarking of Public Utilities 23 Generally… LC most often outperforms cDEA and StoNEzD Distribution of z does not seem to matter concerning bias Correlation of z & x has only little effect (BL4 vs. the others) Also magnitude of environmental effect seems to play a minor role (HI vs BL)

Initial Results 5 Stefan Seifert & Maria Nieswand Benchmarking of Public Utilities 24 LC SFA Performs generally well, stable and efficient Frontier overestimation tendecies in higher noise cases cDEA High sensitivity against noise Underestimation of frontier in small samples, overestimation in larger samples StoNEzD General underestimation of the frontier  favorable for firms Performs well with low inefficiency and small samples But problems with high inefficiency … but does not seem to be generally efficient

Stefan Seifert & Maria Nieswand 25 Benchmarking of Public Utilities Outlook 6

Conclusion and Outlook 5 Stefan Seifert & Maria Nieswand Benchmarking of Public Utilities 26

Vielen Dank für Ihre Aufmerksamkeit. DIW Berlin — Deutsches Institut für Wirtschaftsforschung e.V. Mohrenstraße 58, Berlin Redaktion

Initial Results 5 Stefan Seifert & Maria Nieswand Benchmarking of Public Utilities 28

Initial Results 5 Stefan Seifert & Maria Nieswand Benchmarking of Public Utilities 29

Initial Results 5 Stefan Seifert & Maria Nieswand Benchmarking of Public Utilities 30

Initial Results 5 Stefan Seifert & Maria Nieswand Benchmarking of Public Utilities 31

Initial Results 5 Stefan Seifert & Maria Nieswand Benchmarking of Public Utilities 32

Initial Results 5 Stefan Seifert & Maria Nieswand Benchmarking of Public Utilities 33

Initial Results 5 Stefan Seifert & Maria Nieswand Benchmarking of Public Utilities 34

Initial Results 5 Stefan Seifert & Maria Nieswand Benchmarking of Public Utilities 35

References Agrell, P. and Bogetoft, P. (2013). Benchmarking and Regulation. CORE Discussion Papers Andor, M. and Hesse, F. (2014). The stoned age: the departure into a new era of efficiency analysis? a monte carlo comparison of stoned and the oldies (SFA and DEA). JPA, 41(1): Badunenko, O., Kumbhakar, S. (2015) When, Where and How to Estimate Persistent and Time-Varying Efficiency in Panel Data Models. WP. Cordero, J. M., Pedraja, F., and Santin, D. (2009). Alternative approaches to include exogenous variables in DEA measures: A comparison using Monte carlo. Comput. Oper. Res., 36(10): Daraio, C. and Simar, L. (2005). Introducing Environmental Variables in Nonparametric Frontier Models: a Probabilistic Approach. JPA, 24(1): Daraio, C. and Simar, L. (2007). Conditional nonparametric frontier models for convex and nonconvex technologies: a unifying approach. JPA, 28(1): Greene, W. H. (2005). Reconsidering heterogeneity in panel data estimators of the stochastic frontier model. Journal of Econometrics, 126(2): Haney, A. B. and Pollitt, M. G. (2009). Efficiency analysis of energy networks: An international survey of regulators. Energy Policy, 37(12): Johnson, A. and Kuosmanen, T. (2011). One-stage estimation of the effects of operational conditions and practices on productive performance: asymptotically normal and efficient, root-n consistent StoNEzD method. JPA, 36(2): Jondrow, J., Knox Lovell, C. A., Materov, I. S., and Schmidt, P. (1982). On the estimation of technical inefficiency in the stochastic frontier production function model. Journal of Econometrics, 19(2-3): Krüger, J. J. (2012). A monte carlo study of old and new frontier methods for efficiency measurement. EJOR, 222: Kuosmanen, T. (2012). Stochastic semi-nonparametric frontier estimation of electricity distribution networks: Application of the stoned method in the Finnish regulatory model. Energy Economics, 34(6): Lee, C.-Y., Johnson, A. L., Moreno-Centeno, E., and Kuosmanen, T. (2013). A more efficient algorithm for convex nonparametric least squares. EJOR, 227(2): Orea, L. and Kumbhakar, S. C. (2004). Efficiency measurement using a latent class stochastic frontier model. Empirical Economics, 29(1): Yu, C. (1998). The effects of exogenous variables in efficiency measurement - a monte carlo study. EJOR, 105(3): Stefan Seifert & Maria Nieswand Benchmarking of Public Utilities 36