A comparison of established and newly developed benchmarking methods Jennifer Davies, Duncan Elliott (ONS), Homesh Sayal, John Aston (University of Cambridge)

Overview What is benchmarking? Methods Comparison of benchmarking methods Conclusions Further work

What is benchmarking and why is it needed? Two series from different sources measuring the same phenomenon at different frequencies Eg Total of monthly turnover not equal to annual Constrain high to low frequency series Low frequency – higher quality in terms of levels High frequency – higher quality in short term movements Used in extensively in official statistics

What is benchmarking?

Notation Path series – high frequency series Benchmark series – low frequency series Benchmarked series – path series constrained to benchmark series Constraints Sum Average Point in time

Methods - Denton Minimises difference between growth rates in observed path and benchmarked series Used for constraining seasonally adjusted annual totals to non-seasonally adjusted total in ONS

Methods – Cholette-Dagum Regression-based approach Assumes observed path series is ‘truth’ plus bias and variance Assumes observed benchmark is ‘truth’ plus variance (variance often assumed to be 0 – binding benchmarking) Denton is a sub-case Used in production of National Accounts

Methods - Wavelets New method for benchmarking using wavelets Developed by Homesh Sayal, John Aston (University of Cambridge), Duncan Elliott (ONS) and Hernando Ombao (University of California at Irvine)

Methods - Wavelets

Wavelet benchmarking Decompose high and low frequency series using same wavelet basis (with additional wavelets in the high frequency series) Swap low frequency wavelets in path series with those in benchmark series to get benchmarked series Further step - thresholding – removes wavelets that are considered noise Smoothing based on variance estimated from a dynamic linear model (differs to the model used by Sayal et al, 2014)

Comparison Index of Production (IoP) Two simulation studies 1.Using IoP as true path and simulating noise 2.Looking at the effect of outliers and level shifts on benchmarking

Comparison 1 – simulating noise 6241 simulated series True path known, noise simulated (irregular components from IoP) to get observed path Diagnostics: Did it work? Did it meet the benchmark constraints? MSE between benchmarked series and 1. observed path and 2. true path Percentage of growth rates with same sign as observed series

Comparison 1 – simulating noise All series ran using all methods All series met constraints using all methods MeasureDentonCholette- Dagum Elementary wavelets Threshold wavelets MSE true path MSE obs. path Growth true path 76.5% 52.1% Growth obs.path99.5% 99.7%59.7%

Comparison 1 – simulating noise

Comparison 2 – adding outliers and level shifts to path series Use IoP series as true path Outlier – reduce January 2006 observation by 5% - use as observed path Level shift – reduce series prior to May 2005 by 4% - use as observed path

Comparison – series with outlier

Comparison – series with level shift

Comparison - adding outliers and level shifts to path series Further work adding noise into the path series MeasureDentonCholette- Dagum Elementary wavelet Threshold wavelet MSE true path – outlier MSE true path – level shift

Conclusions Elementary wavelets performs slightly better than Denton and Cholette-Dagum in simulations Threshold wavelets smoothes so gets close to ‘truth’ but not observed path – doesn’t get close to growth rates because of smoothing Outliers and level shifts Elementary wavelets affected more by level shifts before the break and less affected by outliers All methods would be improved by adjusting for additive outliers and level shifts

Further work Further work into outliers and level shifts Revisions analysis – incorporate wavelet forecasting Practical problems for implementing benchmarking eg additional constraints

Thanks!