04/05/2011 Seasonal Adjustment and DEMETRA+ ESTP course EUROSTAT 3 – 5 May 2011 Dario Buono and Enrico Infante Unit B2 – Research and Methodology © 2011 by EUROSTAT. All rights reserved. Open to the public

2 04/05/2011ESTP course Identification of type of Outliers Calendar Effects and its determinants Moving Averages X-12 ARIMA and TRAMO/SEATS Session 2 – May 4th Dario Buono, Enrico Infante

3 Outliers Outliers are data which do not fit in the tendency of the Time Series observed, which fall outside the range expected on the basis of the typical pattern of the Trend and Seasonal Components Additive Outlier (AO): the value of only one observation is affected. AO may either be caused by random effects or due to an identifiable cause as a strike, bad weather or war Temporary Change (TC): the value of one observation is extremely high or low, then the size of the deviation reduces gradually (exponentially) in the course of the subsequent observations until the Time Series returns to the initial level Level Shift (LS): starting from a given time period, the level of the Time Series undergoes a permanent change. Causes could include: change in concepts and definitions of the survey population, in the collection method, in the economic behavior, in the legislation or in the social traditions 04/05/2011ESTP courseDario Buono, Enrico Infante

4 Outliers 04/05/2011ESTP course Type of Outliers Dario Buono, Enrico Infante

5 04/05/2011ESTP course AO - Estate agency activity Outliers Dario Buono, Enrico Infante

6 04/05/2011ESTP course Outliers TC – Business machine Dario Buono, Enrico Infante

7 04/05/2011ESTP course Outliers LS – Tobacco Dario Buono, Enrico Infante

8 04/05/2011ESTP course Outliers Additive Outliers Unusual high or low singular values in the data series Transitory Changes Transitory changes in the trend, followed by slow comebacks to the initial tendency Level Shift Clear changes of the trend Assimilated to the Irregular Component Assimilated to the Trend Component Dario Buono, Enrico Infante

9 04/05/2011ESTP course TRAMO & X-12 ARIMA use regARIMA models where Outliers are regressors X t is the raw series, Z t is the “linearized” series (corrected from Outliers and other effects) Z t follows an ARIMA(p,d,q): where  is the rupture date,  is the rupture impact and  (B) models the kind of rupture (its “shape”) Outliers Dario Buono, Enrico Infante

10 04/05/2011ESTP course Outliers Automatic and iterative detection and estimation process are available in TRAMO and X-12 ARIMA Dario Buono, Enrico Infante

11 04/05/2011ESTP course Outliers The smoothness of series can be decided by statisticians and the policy must be defined in advance Consult the users This choice can influence dramatically the credibility Outliers in last quarter are very difficult to be identified Some suggestions: –Look at the growth rates –Conduct a continuous analysis of external sources to identify reasons of Outliers –Where possible always add an economic explanation –Be transparent (LS, AO,TC) Dario Buono, Enrico Infante

12 04/05/2011ESTP course Calendar Effects Time Series: usually a daily activity measured on a monthly or quarterly basis only –Flow: monthly or quarterly sum of the observed variable –Stock: the variable is observed at a precise date (example: first or last day of the month) Some movements in the series are due to the variation in the calendar from a period to another –Can especially be observed in flow series –Example: the production for a month often depends on the number of days Dario Buono, Enrico Infante

13 04/05/2011ESTP course Calendar Effects Trading Day Effect –Can be observed in production activities or retail sale –Trading Days (Working Days) = days usually worked according to the business uses –Often these days are non public holiday weekdays (Monday, Tuesday, Wednesday, Thursday, Friday) –Production usually increases with the number of working days in the month Dario Buono, Enrico Infante

14 04/05/2011ESTP course “Day of the week” effect –Example: Retail sale turnover is likely to be more important on Saturdays than on other weekdays Statutory (Public) Holidays and Moving Holidays –Most of statutory holidays are linked to a date, not to a day of the week (Christmas) –Some holidays can move across the year (Easter, Ramadan) and their effect is not completely seasonal  Months and quarters are not equivalent and not directly comparables Calendar Effects Dario Buono, Enrico Infante

15 04/05/2011ESTP course French Calendar Calendar Effects – Trading Days Dario Buono, Enrico Infante

16 04/05/2011ESTP course Average number of Working Days Calendar Effects – Trading Days Dario Buono, Enrico Infante

17 04/05/2011ESTP course Average number of Working Days Calendar Effects – Trading Days Dario Buono, Enrico Infante

18 04/05/2011ESTP course Calendar Effects can be partly considered seasonal: –The number of Trading Days is almost always smaller in February than in March –Some months may have more public holidays (ex: May in France), and therefore less trading days than others Thus, we can: –Either estimate the non seasonal Calendar Effects –Make the correction of both Seasonal and Calendar Effects Spectra of the “Weekday” series: number of Mondays, Tuesdays, …, Friday per month Calendar Effects – Trading Days Dario Buono, Enrico Infante

19 04/05/2011ESTP course Average length of a year: days Average length of a month: Average number of weeks per month: Calendar Effects – Trading Days Dario Buono, Enrico Infante

20 04/05/2011ESTP course Spectra of Weekdays series Calendar Effects – Trading Days Dario Buono, Enrico Infante

21 04/05/2011ESTP course Hypothesis: for the most common models, it is assumed that the effect of a day is constant over the span of the series Basic model: –N i,t = # of Mondays (i=1), …, Sundays (i=7) in month t –  i is the effect of day i (ex: average production for a Monday (i=1)) Calendar Effects – Trading Days Dario Buono, Enrico Infante

22 04/05/2011ESTP course In some cases (particular activities) you could have to make a distinction between working and non working Mondays, Tuesdays etc. You can of course adapt the model to other more simple or more complex cases Calendar Effects – Trading Days Dario Buono, Enrico Infante

23 04/05/2011ESTP course Certain groupings could be more sensible than others: –Public holidays –Non working weekdays –Trading days and weekends –Etc. Much better to use contrasts (W. Bell): where: –  is the average effect of a day (mean of the  and  ) –m t is the length of month t Calendar Effects – Trading Days Dario Buono, Enrico Infante

24 04/05/2011ESTP course The variables take into account a part of Seasonality: the length of the month is seasonal, the White Monday is always a Monday etc. If we want to estimate the non seasonal Calendar Effects, we can remove from the variables their long- term average In this case, the explanatory variables do not have any Trend or Seasonality. Thus, the dependant variable must be stationary, an Irregular Component for example (X-11), or a regARIMA model should be used X-12 ARIMA and TRAMO-SEATS Calendar Effects – Trading Days Dario Buono, Enrico Infante

25 04/05/2011ESTP course Coffee-Break!!! We will restart in 15 minutes Dario Buono, Enrico Infante

26 04/05/2011ESTP course Calendar Effects – Moving Holidays The date of some public holidays (usually religious events) moves across the year and can affect different months or quarters according to the year Examples: Ramadan, Chinese New Year, Easter (Catholic or Orthodox) etc. The effect of the event could be gradual and could affect the days before or after the event itself: Chocolate sales around Easter We focus in the following on the treatment of Easter but the methodology and the models can be easily adapted Dario Buono, Enrico Infante

27 04/05/2011ESTP course Calendar Effects – Moving Holidays Catholic Easter –Sunday Easter can fall between March 22 (last time: 1818, next time: 2285) and April 25 (last time: 1943, next time: 2038) i.e. during the first or second quarter Orthodox Easter –Between April 4 (last time: 1915, next time: 2010) and May 9 (last time before 1600, next time: 2173) i.e. always during the third quarter Easter affects the production of many economic sectors: –Statutory holidays (transportation, hotel frequentation), consumption habits (chocolate, flowers, lamb meat, etc.) Dario Buono, Enrico Infante

28 04/05/2011ESTP course Calendar Effects – Moving Holidays Catholic Easter dates ( ) Dario Buono, Enrico Infante

29 04/05/2011ESTP course Calendar Effects – Moving Holidays Catholic and Orthodox Easter dates ( ) Dario Buono, Enrico Infante

30 04/05/2011ESTP course Calendar Effects – Moving Holidays Usually models are based on the number of days “affected” by Easter in the various months (March or April for Catholic Easter) –Immediate Impact –Gradual Impact: effect on the n days preceding Easter Easter can affect the w preceding days; the impact can therefore affect 2 months The impact is supposed constant on the w days Dario Buono, Enrico Infante

31 04/05/2011ESTP course Calendar Effects – Moving Holidays The 2 Easter models are estimated using a regARIMA model: Where: –X t represents a regressor modeling Calendar Effects, Outliers (AO, LS, TS) and other user-specified effects including Easter Effects –D t represents a priori adjustments such as leap year, strikes, etc. Dario Buono, Enrico Infante

32 04/05/2011ESTP course Calendar Effects – Moving Holidays The w-day period, during which the activity is affected, does NOT include the Sunday Easter. The activity level remains constant on the w-day period Dario Buono, Enrico Infante

33 04/05/2011ESTP course Calendar Effects – Moving Holidays If i denotes the year and j the month, let us note and the number of the w days falling in month j of year i The regressor associated to this model is: The regressor values are 0 except for February, March and April depending on the value of w Dario Buono, Enrico Infante

34 04/05/2011ESTP course Calendar Effects – Moving Holidays A general constant effect model Dario Buono, Enrico Infante

35 04/05/2011ESTP course Calendar Effects – Moving Holidays A general linear effect model Dario Buono, Enrico Infante

36 04/05/2011ESTP course Orthodox Easter falls more in April than in May (83% versus 17%) Catholic Easter falls more in April than in March (78% versus 22%) There is a “Seasonal Effect” one can correct by removing the long-term monthly average: Calendar Effects – Moving Holidays Dario Buono, Enrico Infante

37 04/05/2011ESTP course Calendar Effects Definitions: –The Seasonally Adjusted series is logically defined as the raw series from which the Seasonality ( S t ) has been removed –The “Working Day” adjusted series is defined as the raw data from which the Calendar Effects (TD t, MH t ) have been removed –This definition can vary if you include or not the Calendar Effects Dario Buono, Enrico Infante

38 03/05/2011ESTP course Moving Averages A Moving Average of order p+f+1 and coefficients {  i } is defined by: The value at date t is therefore replaced by a weighted average of p past values, the current value and f future values Examples: simple moving averages of order 3 Asymmetric Symmetric Dario Buono, Enrico Infante

39 03/05/2011ESTP course Moving Averages Let us suppose the following decomposition model: One can want remove the Seasonality and the Irregular by using a Moving Average: So, we want to cancel some frequencies!! Dario Buono, Enrico Infante

40 03/05/2011ESTP course Moving Averages We would like to find a Moving Average which preserve the Trend and removes both Seasonality and Irregular –Example (preservation of constants): let us define a Series and a Moving Average M The coefficients must sum to 1 Property: if you want to preserve polynomials of degree d, the Moving Average coefficients must verify: Dario Buono, Enrico Infante

41 03/05/2011ESTP course X-12 ARIMA VS TRAMO/SEATS Seasonal Adjustment is usually done with an off-the-shelf program. Three popular tools are: –X-12 ARIMA (Census Bureau) –TRAMO/SEATS (Bank of Spain) –DEMETRA+ (Eurostat), interface X-12 ARIMA and Tramo/Seats X-12 ARIMA is Filter based: always estimate a Seasonal Component and remove it from the series even if no Seasonality is present, but not all the estimates of the Seasonally Adjusted series will be good TRAMO/SEATS is model based method variants of decomposition of Time Series into non-observed components Dario Buono, Enrico Infante

42 03/05/2011ESTP course X-12 ARIMA Auto-projective Models What is f ? –Under certain hypothesis (Stationary), it is possible to find an “ARMA function” which gives a good approximation of f –These models usually depend on a few number of parameters only –Box et Jenkins methodology allows to estimate the parameters and gives quality measures of the adjustment Dario Buono, Enrico Infante

43 03/05/2011ESTP course X-12 ARIMA AutoRegressive Model of order p AR(p): Moving Average Model of order q MA(q): Dario Buono, Enrico Infante

44 03/05/2011ESTP course X-12 ARIMA ARMA(p,q): –Corner method, triangle method, ODQ etc. ARIMA(p,d,q): –Non stationary Time Series, with a Trend SARIMA(p,d,q)(P,D,Q) S : –Non stationary Time Series, with Trend and Seasonality Dario Buono, Enrico Infante

45 X-12 ARIMA A regARIMA model is a regression model with ARIMA errors. When we use regression models to estimate some of the components in a Time Series, the errors from the regression model are correlated, and we use ARIMA models to model the correlation in the errors. ARIMA models are one way to describe the relationships between points in a Time Series Besides using regARIMA models to estimate regression effects (such as outliers, Trading Day, and Moving Holidays), we also use ARIMA models to forecast the series. Research has shown that using forecasted values gives smaller revisions at the end of the series Dario Buono, Enrico Infante

46 03/05/2011ESTP course X-12 ARIMA X-12 ARIMA runs through the following steps: 1.The series is modified by any user defined prior adjustments 2.The program fits a regARIMA model to the series in order to detect and adjust for outliers and other distorting effects to improve forecasts and Seasonal Adjustment 3.It detect and estimates additional component (e.g. Calendar Effects) and extrapolate forward (forecast) and backwards (backcast) 4.The program then uses a series of Moving Averages to decompose a Time Series into three components. It does this in three iterations, getting successively better estimates of the components. During these iterations extreme values are identified and replaced 5.In the last step a wider range of diagnostic statistics are produced, describing the final Seasonal Adjustment, and giving pointers to possible improvements which could be made Dario Buono, Enrico Infante

47 03/05/2011ESTP course A Filter is a weighted average where the weights sum to 1 Seasonal Filters are the filters used to estimate the Seasonal Component. Ideally, Seasonal Filters are computed using values from the same month or quarter, for example, an estimate for January would come from a weighted average of the surrounding Januaries The Seasonal Filters available in X-12 ARIMA consist of seasonal Moving Averages of consecutive values within a given month or quarter. An n x m Moving Average is an m- term simple average taken over n consecutive sequential spans X-12 ARIMA Dario Buono, Enrico Infante

48 03/05/2011ESTP course An example of a 3x3 filter (5 terms) for January 2003 (or Quarter 1, 2003) is: An example of a 3x5 filter for January 2003 (or Quarter 1, 2003) is: X-12 ARIMA Dario Buono, Enrico Infante

49 03/05/2011ESTP course TRAMO/SEATS The objective of the procedure is to automatically identify the model fitting the Time Series and estimate the model parameters. This includes: –The selection between additive and multiplicative model types (log-test) –Automatic detection and correction of outliers, eventual interpolation of missing values –Testing and quantification of the Trading Day effect –Regression with user-defined variables –Identification of the ARIMA model fitting the Time Series, that is selection of the order of differentiation (unit root test) and the number of autoregressive and Moving Average parameters, and also the estimation of these parameters Dario Buono, Enrico Infante

50 03/05/2011ESTP course TRAMO/SEATS The application belongs to the ARIMA model-based method variants of decomposition of Time Series into non-observed components The decomposition procedure of the SEATS method is built on spectrum decomposition Components estimated using Wiener-Kolmogorov Filter SEATS assumes that: –The Time Series to be Adjusted Seasonally is linear, with normal White Noise innovations –If this assumption is not satisfied, SEATS has the capability to interwork with TRAMO to eliminate special effects from the series, identify and eliminate outliers of various types, and interpolate missing observations –Then the ARIMA model is also borrowed from TRAMO Dario Buono, Enrico Infante

51 03/05/2011ESTP course TRAMO/SEATS The application decomposes the series into several various components. The decomposition may be either multiplicative or additive The components are characterized by the spectrum or the pseudo spectrum in a non-stationary case: –The Trend Component represents the long-term development of the Time Series, and appears as a spectral peak at zero frequency. One could say that the trend is a cycle with an infinitely long period –The effect of the Seasonal Component is represented by spectral peaks at the seasonal frequencies –The Irregular Component represents the irregular White Noise behaviour, thus its spectrum is flat (constant) –The Cyclic Component represents the various deviations from the trend of the Seasonally Adjusted series, different from the pure White Noise Dario Buono, Enrico Infante

52 03/05/2011ESTP course First SEATS decomposes the ARIMA model of the Time Series observed, that is, identifies the ARIMA models of the components. This operation takes place in the frequency domain. The spectrum is divided into the sum of the spectra related to the various components Actually SEATS decides on the basis of the argument of roots, which is mostly located near to the frequency of the spectral peak –The roots of high absolute value related to 0 frequency are assigned to the Trend Component –The roots related to the seasonal frequencies to the Seasonal Component –The roots of low absolute value related to 0 frequency and the cyclic (between 0 and the first seasonal frequency) and those related to frequencies between the seasonal ones are assigned to the Cyclic Component –The Irregular Component is always deemed as white noise TRAMO/SEATS Dario Buono, Enrico Infante

53 Questions? 04/05/2011ESTP courseDario Buono, Enrico Infante