1. 4th Joint EU-OECD Workshop on BCS, Brussels, October 12-13
Benchmarking SA softwares using Frankenstein’s series

2. Outline The problem: How to Benchmark SA softwares?
Benchmarking Softwares and/or Methods
The Seasonal Adjustment Problem
The Basic Idea: Simulated and Frankenstein’s Series
Constructing Frankenstein’s series
Methodology
Clustering Time Series
The Many Faces of Seasonality
Some examples
A first comparison of software performances
Turning point detection and phase-shifts

3. Benchmarking Softwares and/or Methods
Benchmarking algorithms and methods is quite common
Statistical softwares:
the NIST benchmarks: Datasets for some statistical problems with certified results (descriptive, ANOVA, linear and non-linear regression).
Diehard Tests for Random Number Generators (Marsaglia)
Tests on Statistical Distributions (Knüsel) etc.
Tests on Forecasting Methods
The M-competitions (Makridakis et al)
But “quite easy” because you know what the truth is!

4. The Seasonal Adjustment Model
A quite complex decomposition model
Trend, cycle, seasonality, outliers, trading-day effect, Easter effect, irregular
Unobserved Component Models: only Xt is observed → no unique solution
Each method-software has its own hypotheses to propose “THE solution”.
Various schools and churches and the traditional “parametric versus non-parametric opposition”

5. Seasonal Adjustment Methods

6. Simulated and Frankenstein’s Series
Simulated series: OK but “not fair”
SEATS: The series follows a seasonal ARIMA model
STAMP, BAYSEA, DECOMP: each component follows an ARIMA model
X-12-ARIMA: Trend-Cycle and Seasonality locally behave as polynomial of low orders.
Frankenstein’s series
Reconstruct “real-simulated series” from “representative” estimated components

7. Frankenstein’s Series: Methodology (1)
“Representative”? Looking for “frequent” trends, seasonality etc.
We use “long” raw series from 2 large economic databases: Euro-Ind (Eurostat) & MEI (OECD). About 9000 quarterly and monthly series at least 15-years long.
Pre-cleaning (outliers, ruptures, calendar effects) of the series using Tramo-Seats & X-12-ARIMA.
We keep series with a clear seasonality (≈ 4000)

8. Frankenstein’s Series: Methodology (2)
Seasonal adjustment of these series with 7 softwares: Baysea, Dainties, Decomp, Stamp, Tramo-Seats & X-12-Arima (releases 0.2 and 0.3).
Cluster analysis of the seasonal components and of the trend-cycle components
Objective: grouping trends and seasonal components into clusters containing “similar” objects
But what does mean “similar” ?

9. Clustering Time Series (1)
The “similarity/dissimilarity” problem

10. Clustering Time Series (2)
Another example to be fair ….

11. Clustering Time Series (3)
The “direct” Euclidian distance does not always fit our needs.

12. Clustering Time Series (2)
Project the series in another distance-preserving space:
Autocorrelation Functions (ACF, PACF, IACF)
Discrete Fourier Transforms (DFT)
Discrete Wavelet Transforms using Daubechies or Haar basis, Cosine Wavelets
Chebyshev Polynomials
ARIMA coefficients
Linear Predictive Coding of the Cepstrum (LPC)
Singular Value Decomposition through Principal Component Analysis
Smooth Localized Complex Exponential model
Piecewise Linear Approximation
Piecewise Aggregate Approximation
Adaptive Piecewise Constant Approximation.

13. The Many Faces of Seasonality
We cluster the spectra
How many clusters????

14. Back to our problem: simulations
Reconstructing series from various parts ……
2 decomposition models
additive and multiplicative
5 different irregulars (variance),
100 realizations of each
n different seasonalities,
5 different variances
p different trend-cycles
5000 np series
n=p=20 appears to be a « reasonable choice »
So …. Let us take 25!!!!

15. Looking to clusters, choosing medoids
Choosing medoids in large and homogeneous clusters

16. Very similar shapes

17. Shorter and lagged shapes

18. Similar but inversed shapes

19. What applications? Choosing realistic models for the components
Assessing and comparing the performances of 2 or more softwares …. But:
Softwares do have many parameters one can tune to improve the output
It is therefore easier to assess the relevance of default parameter values
2 examples (on Business Survey Data):
A first comparison of software performances
Turning point detection and phase-shifts

20. A first comparison of software performances
We compute the distance between the SA series and the « real » SA series.

21. Turning point detection
Of utmost importance for short-term economic analysis
Dainties, used here as a reference point, is based on asymmetric filters which are known to introduce phase shifts.
Results expressed as “balances”: % (software X leads Dainties) - % (software X lags Dainties)