1. A ‘Value for Money’ monitor that takes account of Customer Satisfaction, Quality and Investment (also know as ‘3 legged stool’) helping authorities make judgements about VFM, and find better quantified options for improvement and efficiency savings

2. Original Methodology Keep it simple, but keep developing it
Use publically available data (whenever possible)
Include all Highways Authorities in the analysis
Preserve anonymity
Align reporting to LTP and other national priorities
Make like for like comparisons – on a common scale
Make comparisons between similar Authorities
Make comparisons over time
Take account of demographic and other differences
Improves on previous ‘Cost v Quality’ VFM tools by including ‘Customer’ data (from the NHT Survey)
(e.g. Audit Commission - Value for Money Profiles and DfT – Benchmarking Toolkit)

3. Stages of Development
Stage 1: Initial Development ( ) (pilot funded by IDeA in 2009)
Applied to all Highways & Transport Themes, results produced for all Authorities
Used public data only, expressed results as scores out of 10 in stacked bar charts.
Concerns about cost data and lack of quality data, better to use actual figures (e.g.in £’s) easier to understand.
Stage 2: Focus on Maintenance (HMEP funding) (2012)
Uses cost and quality data collected from participants for all Maintenance activities
Difficulties expressing overall performance (analysis looked at measures separately)
Problem determining ‘what is good’ (is low cost good?)
Analysis didn’t account for geographical characteristics or historical maintenance strategies
Stage 3: Frontier Benchmarking Pilot (HMEP funding) (April 2013)
Econometric analysis of CQC data undertaken by ITS , University of Leeds.
Simple pilot, uses data collected on Road Maintenance, Street Lighting & Winter Service
Models too simplistic, models need extending and larger sample required
Simple method required to explain the approach and communicate results
Stage 4: Extend analysis & improve models (HMEP funding) (Oct 2013)
Partial model to be developed for all Authorities (Cost & Quality only)
Pilot models to be improved with further cost drivers and more attribute data

4. Frontier Benchmarking
Content supplied by Phill Wheat of The University of Leeds

5. Econometric Analysis – Frontier Benchmarking
Frontier Benchmarking estimates the relationship between cost and the factors which drive cost (these include measures of quality, public satisfaction and other attribute data). The analysis measures how close Authorities are to the minimum cost (cost frontier) of providing their current level of service Crucially Frontier Benchmarking recognises that all Authorities are different and thus their minimum possible costs are different.

6. Frontier Benchmarking – Efficiency Gap
The Frontier Benchmarking model estimates ‘efficiency gaps’
the gap between an Authority’s actual cost and the minimum possible cost of producing a given output
(e.g. maintaining their length of network, and keeping road condition and public satisfaction the same)
An Authority’s efficiency gap quantifies their potential for efficiency savings
Measuring changes in an Authority’s efficiency gaps over time quantifies their realisation of efficiency savings
Efficiency gaps provide a measureable basis for identifying ‘who is good’ (which Authorities are operating most efficiently)

7. Efficiency Gap

8. Other Benefits – measures scale effects
A feature of Frontier Benchmarking analysis is the way in which it estimates the effects of scale.
The analysis can show whether costs change in proportion to output (i.e. if unit costs fall as output increases)
This is important when comparing Authorities. It recognises that a small Authority can be more efficient than a large Authority even if it has higher unit costs than the large Authority.

9. Other Benefits - what if analysis
Frontier Benchmarking Cost models can be used to predict the minimum costs for different levels of service.
This allows Authorities to consider the cost implications of different policy options, for example:
merging highway functions and increased network size for a given operation,
changing quality (e.g. to improve average condition by 1%)
changing levels of public satisfaction (e.g. Improving public satisfaction by 5%)

10. Frontier Benchmarking Outputs
Identifies the best performing Authority for any maintenance activity (those at minimum cost)
Identifies areas of activity in which an Authority has the greatest potential for improvement (the largest efficiency gaps)
Measures efficiency savings achieved by an Authority and the sector as a whole (reductions in efficiency gaps over time)
Measures the effect of scale
Measures the impact on cost of changes to service level (including quality and public satisfaction)

11. Frontier Benchmarking Pilot

12. Details of Pilot
The Institute for Transport Studies at the University of Leeds was commissioned to undertake the analysis.
The Project Team chose areas of high expenditure for the development of three cost models:
Highway Pavement Maintenance
Street Lighting
Winter Maintenance
Data on cost and quality was requested from Authorities over a period of four years (2009 to 2012) for each model
Other required data was accessed from public sources as appropriate.
45 Highways Authorities returned data
A report of the Pilot has been published and each participant has received the details of their own results.
The results have been kept anonymous at this stage

13. Composition of Cost Models
Cost Category
Cost Drivers
Observations
Highway Pavement Maintenance spend
(Reactive + Structural Maintenance)
Total Highway Network Length
Average Highway Condition
Public Satisfaction with Condition of Road Surfaces 90
Street Lighting spend
Number of lighting columns
% of units operational
Public Satisfaction with Street Lighting Repairs
107
Winter maintenance spend
Length of Precautionary Salting Routes
Number of Salting Runs
Public Satisfaction with Winter Maintenance
135
N.B. Data was only included in the analysis if an Authority had provided all the required data for a cost model in any one year – an observation is one year’s data for an authority.

14. Pilot Cost Model Results
Average
efficiency gap
Average potential efficiency saving
(per authority)
Total potential
Efficiency saving
(all pilot authorities)
Highway Maintenance
29%
£ 4.7m
£ 212m
Street Lighting
28%
£ 0.9m
£ 41m
Winter Maintenance
14%
£ 0.3m
£ 14m
Notes: Total potential saving is calculated by multiplying average potential saving per authority by number of authorities to participate in the pilot.

15. Effects of Scale - Pilot Results
Highway Pavement Maintenance - larger authorities have unit cost advantages over smaller authorities – there are economies of scale.
The figures show that as highways length increases by 1%, costs increase by only 0.9% i.e. costs increase less than proportionally with highway length.
Street Lighting – the model shows a u-shaped average cost curve with the minimum unit cost at approximately 50,000 lighting units.
The analysis shows unit costs fall as the authority gets larger, however at approximately 50,000 lighting units, the unit costs start to rise as authorities get larger still.
Winter Maintenance – the model indicates constant returns to scale.
The results indicate that increasing salting routes by 1% increases costs by 1%
(n.b. this is the weakest of the models from a statistical point of view and requires more work).

16. Limitations of analysis
As this was a Pilot the Project Team chose to keep it simple
The model uses limited cost driving variables; there will be other variables that explain cost differences, which are not included.
These omissions are likely to inflate the gap between the estimated minimum cost frontier and an authority’s actual cost.
Thus an authority’s apparent inefficiency may just be that they are facing different operating conditions rather than being inefficient.
The analysis only identifies which authorities are efficient, it does not explain why they are efficient. This will require some further work to compare practices between the efficient authorities and others.

17. For more information on CQC
please contact:
Simon Pinkney or Jennie Simons of m2i
For more information on Frontier Benchmarking
Phill Wheat of The Institute for Transport Studies, The University of Leeds