A ‘Value for Money’ monitor that takes account of Customer Satisfaction, Quality and Investment (also know as ‘3 legged stool’) .... helping authorities make judgements about VFM, and find better quantified options for improvement and efficiency savings

Original Methodology Keep it simple, but keep developing it Use publically available data (whenever possible) Include all Highways Authorities in the analysis Preserve anonymity Align reporting to LTP and other national priorities Make like for like comparisons – on a common scale Make comparisons between similar Authorities Make comparisons over time Take account of demographic and other differences Improves on previous ‘Cost v Quality’ VFM tools by including ‘Customer’ data (from the NHT Survey) (e.g. Audit Commission - Value for Money Profiles and DfT – Benchmarking Toolkit)

Stages of Development Stage 1: Initial Development (2009-2011) (pilot funded by IDeA in 2009) Applied to all Highways & Transport Themes, results produced for all Authorities Used public data only, expressed results as scores out of 10 in stacked bar charts. Concerns about cost data and lack of quality data, better to use actual figures (e.g.in £’s) easier to understand. Stage 2: Focus on Maintenance (HMEP funding) (2012) Uses cost and quality data collected from participants for all Maintenance activities Difficulties expressing overall performance (analysis looked at measures separately) Problem determining ‘what is good’ (is low cost good?) Analysis didn’t account for geographical characteristics or historical maintenance strategies Stage 3: Frontier Benchmarking Pilot (HMEP funding) (April 2013) Econometric analysis of CQC data undertaken by ITS , University of Leeds. Simple pilot, uses data collected on Road Maintenance, Street Lighting & Winter Service Models too simplistic, models need extending and larger sample required Simple method required to explain the approach and communicate results Stage 4: Extend analysis & improve models (HMEP funding) (Oct 2013) Partial model to be developed for all Authorities (Cost & Quality only) Pilot models to be improved with further cost drivers and more attribute data

Frontier Benchmarking Content supplied by Phill Wheat of The University of Leeds

Econometric Analysis – Frontier Benchmarking Frontier Benchmarking estimates the relationship between cost and the factors which drive cost (these include measures of quality, public satisfaction and other attribute data). The analysis measures how close Authorities are to the minimum cost (cost frontier) of providing their current level of service Crucially Frontier Benchmarking recognises that all Authorities are different and thus their minimum possible costs are different.

Frontier Benchmarking – Efficiency Gap The Frontier Benchmarking model estimates ‘efficiency gaps’ the gap between an Authority’s actual cost and the minimum possible cost of producing a given output (e.g. maintaining their length of network, and keeping road condition and public satisfaction the same) An Authority’s efficiency gap quantifies their potential for efficiency savings Measuring changes in an Authority’s efficiency gaps over time quantifies their realisation of efficiency savings Efficiency gaps provide a measureable basis for identifying ‘who is good’ (which Authorities are operating most efficiently)

Efficiency Gap

Other Benefits – measures scale effects A feature of Frontier Benchmarking analysis is the way in which it estimates the effects of scale. The analysis can show whether costs change in proportion to output (i.e. if unit costs fall as output increases) This is important when comparing Authorities. It recognises that a small Authority can be more efficient than a large Authority even if it has higher unit costs than the large Authority.

Other Benefits - what if analysis Frontier Benchmarking Cost models can be used to predict the minimum costs for different levels of service. This allows Authorities to consider the cost implications of different policy options, for example: merging highway functions and increased network size for a given operation, changing quality (e.g. to improve average condition by 1%) changing levels of public satisfaction (e.g. Improving public satisfaction by 5%)

Frontier Benchmarking Outputs Identifies the best performing Authority for any maintenance activity (those at minimum cost) Identifies areas of activity in which an Authority has the greatest potential for improvement (the largest efficiency gaps) Measures efficiency savings achieved by an Authority and the sector as a whole (reductions in efficiency gaps over time) Measures the effect of scale Measures the impact on cost of changes to service level (including quality and public satisfaction)

Frontier Benchmarking Pilot

Details of Pilot The Institute for Transport Studies at the University of Leeds was commissioned to undertake the analysis. The Project Team chose areas of high expenditure for the development of three cost models: Highway Pavement Maintenance Street Lighting Winter Maintenance Data on cost and quality was requested from Authorities over a period of four years (2009 to 2012) for each model Other required data was accessed from public sources as appropriate. 45 Highways Authorities returned data A report of the Pilot has been published and each participant has received the details of their own results. The results have been kept anonymous at this stage

Composition of Cost Models Cost Category Cost Drivers Observations Highway Pavement Maintenance spend (Reactive + Structural Maintenance) Total Highway Network Length Average Highway Condition Public Satisfaction with Condition of Road Surfaces 90 Street Lighting spend Number of lighting columns % of units operational Public Satisfaction with Street Lighting Repairs 107 Winter maintenance spend Length of Precautionary Salting Routes Number of Salting Runs Public Satisfaction with Winter Maintenance 135 N.B. Data was only included in the analysis if an Authority had provided all the required data for a cost model in any one year – an observation is one year’s data for an authority.

Pilot Cost Model Results Average efficiency gap Average potential efficiency saving (per authority) Total potential Efficiency saving (all pilot authorities) Highway Maintenance 29% £ 4.7m £ 212m Street Lighting 28% £ 0.9m £ 41m Winter Maintenance 14% £ 0.3m £ 14m Notes: Total potential saving is calculated by multiplying average potential saving per authority by number of authorities to participate in the pilot.

Effects of Scale - Pilot Results Highway Pavement Maintenance - larger authorities have unit cost advantages over smaller authorities – there are economies of scale. The figures show that as highways length increases by 1%, costs increase by only 0.9% i.e. costs increase less than proportionally with highway length. Street Lighting – the model shows a u-shaped average cost curve with the minimum unit cost at approximately 50,000 lighting units. The analysis shows unit costs fall as the authority gets larger, however at approximately 50,000 lighting units, the unit costs start to rise as authorities get larger still. Winter Maintenance – the model indicates constant returns to scale. The results indicate that increasing salting routes by 1% increases costs by 1% (n.b. this is the weakest of the models from a statistical point of view and requires more work).

Limitations of analysis As this was a Pilot the Project Team chose to keep it simple The model uses limited cost driving variables; there will be other variables that explain cost differences, which are not included. These omissions are likely to inflate the gap between the estimated minimum cost frontier and an authority’s actual cost. Thus an authority’s apparent inefficiency may just be that they are facing different operating conditions rather than being inefficient. The analysis only identifies which authorities are efficient, it does not explain why they are efficient. This will require some further work to compare practices between the efficient authorities and others.

For more information on CQC please contact: Simon Pinkney or Jennie Simons of m2i Simon.pinkney@measure2improve.com Jennie.simons@measure2improve.com For more information on Frontier Benchmarking Phill Wheat of The Institute for Transport Studies, The University of Leeds P.E.Wheat@its.leeds.ac.uk