TESTING AMI PERFORMANCE OF J-DEMETRA+ Turaç YAVUZ TURKSTAT 14.11.2018

Contents Why to test ? Log / Level Test Errors in detection of seasonality Average number of outliers Correct model identification Average number of stationary parameters Conclusion 14.11.2018

Why to test ? Importance of AMI An average user of a seasonal adjustment software might not be a modeling expert. It can get chaotic when one would have to deal with large number of series. 14.11.2018

Testing Environment; Data used; CPU: Intel Core i5 3.20 Ghz RAM: 4 GB Platform: Windows 7 64 bit Software: Jdemetra+ 1.1.0 & 1.2.0 Data used; 95 000 Simulated series by Agustin Maravall using Matlab 14.11.2018

Data Structure 120 obs Log Level 240 obs 14.11.2018

Data Structure For each category ≈500 series generated by 50 different type of ARIMA model; 16 Airline-type (0,1,q) (0,1,Q) 16 Non-seasonal 18 Other seasonal 14.11.2018

Errors in Log/Level Test (in % of series) TRAMO-SEATS Series in Logs Series in Levels 120 240 Airline-type 2.6 0.0 Non-seasonal 5.3 0.5 Other seasonal 3.3 0.2 0.1 TOTAL 3.7 TOTAL (TSW+) 1.0 0.4 0.2 0.0 Difference between version 1.1.0 and 1.2.0 is insignificant 14.11.2018

Errors in Log/Level Test (in % of series) X13 Series in Logs Series in Levels 120 240 Airline-type 4.1 0.0 Non-seasonal 10.0 1.0 Other seasonal 3.7 0.2 TOTAL 5.8 0.4 0.1 TOTAL (TSW+) 1.0 0.4 0.2 0.0 Difference between version 1.1.0 and 1.2.0 is insignificant 14.11.2018

Errors in Detection of Seasonality Test (%) TRAMO-SEATS Number of Obs. 120 240 Airline-type 0.0 Non-seasonal 3.3 2.6 Other seasonal 0.1 TOTAL 1.1 0.8 TOTAL (TSW+) 0.7 0.8 Difference between version 1.1.0 and 1.2.0 is insignificant 14.11.2018

Detection of Seasonality (frequencies for “Seasonality” variable in output) X13 120 obs. 240 obs. . 1 Airline-type 7 15593 3 15997 Non-seasonal 153 14846 80 14920 Other seasonal 16449 16500 For 1.1.0 version, all values of seasonality variable in output file are “1”. 14.11.2018

Detection of Seasonality Visual Spectral Test (%) TRAMO-SEATS Bad Good Severe Airline-type 120 4.3 95.1 0.6 240 4.8 94.4 0.8 Non-seasonal 21.8 68.1 10.1 20.9 68.3 10.8 Other seasonal 5.4 93.6 1.0 5.6 93.3 1.1 TOTAL 10.2 85.9 3.9 Difference between version 1.1.0 and 1.2.0 is insignificant 14.11.2018

Average Number of Outliers per Series TRAMO-SEATS Series in Logs Series in Levels 120 240 Airline-type 0.18 0.11 Non-seasonal 0.15 0.10 0.09 Other seasonal 0.17 0.16 TOTAL 0.12 TOTAL (TSW+) 0.17 0.10 Difference between version 1.1.0 and 1.2.0 is insignificant 14.11.2018

Average Number of Outliers per Series X13 Series in Logs Series in Levels 120 240 Airline-type 0.07 0.06 0.08 Non-seasonal Other seasonal 0.05 TOTAL TOTAL (TSW+) 0.17 0.10 Difference between version 1.1.0 and 1.2.0 is insignificant 14.11.2018

Correct Model Identification (%) TRAMO-SEATS Complete Model Orders Differencing 120 240 Airline-type 71.3 75.2 96.9 98.1 Non-seasonal 58.5 68.0 90.0 92.0 Other seasonal 45.0 70.6 90.6 96.1 TOTAL 58.1 71.4 92.5 95.5 TOTAL (TSW+) 64.5 78.9 94.0 97.5 Difference between version 1.1.0 and 1.2.0 is insignificant 14.11.2018

Correct Model Identification (%) X13 Complete Model Orders Differencing 120 240 Airline-type 68.1 73.8 94.4 96.6 Non-seasonal 33.6 29.1 72.2 74.8 Other seasonal 34.9 47.2 86.7 87.6 TOTAL 45.7 50.4 84.7 86.6 TOTAL (TSW+) 64.5 78.9 94.0 97.5 Difference between version 1.1.0 and 1.2.0 is insignificant 14.11.2018

Correct Model Identification By Model Order (Top 5) TRAMO-SEATS Complete Model Orders Differencing (3,0,0) (0,0,0) 93.2 99.5 (2,0,0) (0,0,0) 99.3 (1,1,0) (0,0,0) 92.4 99.2 (1,0,0) (0,0,0) 91.7 96.6 (0,0,2) (0,0,0) 90.3 97.0 Difference between version 1.1.0 and 1.2.0 is insignificant 14.11.2018

Correct Model Identification By Model Order (Bottom 5) TRAMO-SEATS Complete Model Orders Differencing (0,0,0) (0,0,0) 0.0 90.5 (0,1,0) (0,0,0) 98.2 (2,0,1) (0,0,0) 2.9 47.1 (1,0,1) (1,0,0) 14.0 74.8 (1,0,1) (0,0,0) 23.2 72.4 Difference between version 1.1.0 and 1.2.0 is insignificant 14.11.2018

Average Number of Stationary Parameters TRAMO-SEATS 120 240 In Simulation Model Airline-type 1.86 1.85 1.69 Non-seasonal 1.59 1.63 1.50 Other seasonal 2.36 2.58 2.64 TOTAL 1.95 2.03 1.96 TOTAL (TSW+) 1.93 1.97 1.94 Difference between version 1.1.0 and 1.2.0 is insignificant 14.11.2018

Average Number of Stationary Parameters X13 120 240 In Simulation Model Airline-type 1.86 1.84 1.69 Non-seasonal 2.03 2.44 1.50 Other seasonal 2.42 2.75 2.64 TOTAL 2.11 2.35 1.96 TOTAL (TSW+) 1.93 1.97 1.94 Difference between version 1.1.0 and 1.2.0 is insignificant 14.11.2018

Conclusions Run-time facts; Jdemetra+ is remarkably faster than other SA softwares (TSW, TSW+) Average process time of 8000 series with 120 observation is ≈ 29 seconds for TRAMO-SEATS method (45 seconds for X13) 10 of 95000 series could’nt be processed with TRAMO-SEATS method (and 1 of 95000 for X13) 14.11.2018

Conclusions I/O facts; Results of some diagnostic tests are not included in the csv output file (Friedman, Kruskall-Wallis, Evolutive Seasonality etc.). Even if the combined seasonality test result is “Identifiable seasonality not present”, the value of “Seasonality” variable in csv output is “1”. 14.11.2018

Conclusions ARIMA Model Identification; Reliability level of X13 is not adequate, For TRAMO-SEATS method, +Jdemetra is more reliable than TSW but not TSW+. 14.11.2018