4th Joint EU-OECD Workshop on BCS, Brussels, October 12-13 Benchmarking SA softwares using Frankenstein’s series

Outline The problem: How to Benchmark SA softwares? Benchmarking Softwares and/or Methods The Seasonal Adjustment Problem The Basic Idea: Simulated and Frankenstein’s Series Constructing Frankenstein’s series Methodology Clustering Time Series The Many Faces of Seasonality Some examples A first comparison of software performances Turning point detection and phase-shifts

Benchmarking Softwares and/or Methods Benchmarking algorithms and methods is quite common Statistical softwares: the NIST benchmarks: Datasets for some statistical problems with certified results (descriptive, ANOVA, linear and non-linear regression). Diehard Tests for Random Number Generators (Marsaglia) Tests on Statistical Distributions (Knüsel) etc. Tests on Forecasting Methods The M-competitions (Makridakis et al) But “quite easy” because you know what the truth is!

The Seasonal Adjustment Model A quite complex decomposition model Trend, cycle, seasonality, outliers, trading-day effect, Easter effect, irregular Unobserved Component Models: only Xt is observed → no unique solution Each method-software has its own hypotheses to propose “THE solution”. Various schools and churches and the traditional “parametric versus non-parametric opposition”

Seasonal Adjustment Methods

Simulated and Frankenstein’s Series Simulated series: OK but “not fair” SEATS: The series follows a seasonal ARIMA model STAMP, BAYSEA, DECOMP: each component follows an ARIMA model X-12-ARIMA: Trend-Cycle and Seasonality locally behave as polynomial of low orders. Frankenstein’s series Reconstruct “real-simulated series” from “representative” estimated components

Frankenstein’s Series: Methodology (1) “Representative”? Looking for “frequent” trends, seasonality etc. We use “long” raw series from 2 large economic databases: Euro-Ind (Eurostat) & MEI (OECD). About 9000 quarterly and monthly series at least 15-years long. Pre-cleaning (outliers, ruptures, calendar effects) of the series using Tramo-Seats & X-12-ARIMA. We keep series with a clear seasonality (≈ 4000)

Frankenstein’s Series: Methodology (2) Seasonal adjustment of these series with 7 softwares: Baysea, Dainties, Decomp, Stamp, Tramo-Seats & X-12-Arima (releases 0.2 and 0.3). Cluster analysis of the 28000 seasonal components and of the 28000 trend-cycle components Objective: grouping trends and seasonal components into clusters containing “similar” objects But what does mean “similar” ?

Clustering Time Series (1) The “similarity/dissimilarity” problem

Clustering Time Series (2) Another example to be fair ….

Clustering Time Series (3) The “direct” Euclidian distance does not always fit our needs.

Clustering Time Series (2) Project the series in another distance-preserving space: Autocorrelation Functions (ACF, PACF, IACF) Discrete Fourier Transforms (DFT) Discrete Wavelet Transforms using Daubechies or Haar basis, Cosine Wavelets Chebyshev Polynomials ARIMA coefficients Linear Predictive Coding of the Cepstrum (LPC) Singular Value Decomposition through Principal Component Analysis Smooth Localized Complex Exponential model Piecewise Linear Approximation Piecewise Aggregate Approximation Adaptive Piecewise Constant Approximation.

The Many Faces of Seasonality We cluster the spectra How many clusters????

Back to our problem: simulations Reconstructing series from various parts …… 2 decomposition models additive and multiplicative 5 different irregulars (variance), 100 realizations of each n different seasonalities, 5 different variances p different trend-cycles 5000 np series n=p=20 appears to be a « reasonable choice » So …. Let us take 25!!!!

Looking to clusters, choosing medoids Choosing medoids in large and homogeneous clusters

Very similar shapes

Shorter and lagged shapes

Similar but inversed shapes

What applications? Choosing realistic models for the components Assessing and comparing the performances of 2 or more softwares …. But: Softwares do have many parameters one can tune to improve the output It is therefore easier to assess the relevance of default parameter values 2 examples (on Business Survey Data): A first comparison of software performances Turning point detection and phase-shifts

A first comparison of software performances We compute the distance between the SA series and the « real » SA series.

Turning point detection Of utmost importance for short-term economic analysis Dainties, used here as a reference point, is based on asymmetric filters which are known to introduce phase shifts. Results expressed as “balances”: % (software X leads Dainties) - % (software X lags Dainties)