Back-testing exercise Carlos Cuerpo Ángel Cuevas Enrique M. Quilis OG Working Group Workshop Vilnius; September 5th, 2018. National Audit Office of Lithuania Comunicación

Spanish case: Data set and processing Toolkit Index Introduction Spanish case: Data set and processing Toolkit An application to UK Data Conclusions 1

1. Introduction The main purpose of back-testing is to check the stability and robustness of the output gap estimates. The suggested approach is based on the “beauty contest” approach (i.e., a search on the list of available variables for a given model, being the model a multivariate structural time series model). 1

1. Introduction Selection criteria in “beauty contest” approach: Statistical significance of the coefficients, focusing on the loadings of the observables on the cycle; Average relative revision, defined as the average distance between one-sided and two-sided estimates, relative to the maximum amplitude of the output gap estimate; Average relative uncertainty surrounding the cycle estimates, as the average standard error relative to the maximum amplitude. Economic soundness, meaning that some key macroeconomic relationships could be captured by variables if included in the model Amplitude and profile alignment with consensus figures (range given by a panel of official institutions) and in agreement with commonly accepted business cycle chronology (e.g. ECRI dating). Stability of the one-sided cycle estimate in real time (revisions). 1

2. Spanish case: Data set and pre-processing

2. Spanish case: Data set and pre-processing Seasonal and calendar effects correction All series have been extended since and/or completed until the first quarter of 1980, taking into account their specificities (sources, concepts, different statistical bases, mixed frequencies, etc.) Backward linking retropolation and temporal disaggregation when needed Additional benchmarking techniques are implemented whenever the seasonal adjustment process breaks the temporal consistency with respect to the annual reference At the end of the process, we have a balanced panel

3. Toolkit Data Base Seasonal adjustment Benchmarking Series links Estimation Selector Procyclical Integration order Transformation Output One-sided estimation (trend, cycle and drift) Two-sided estimation (trend, cycle and drift) Backtest Number of backtest Loadings stabilty S.E. stability Trend and cycle stability

4. Application to UK data Main problems: Unbalanced panel Only 11 series starting in 1980Q1 and ending in 2017Q4 All seasonal and calendar corrected, except “New buildings direction permissions. Total building surface” Tramo-Seats has been used to adjust it from seasonality and calendar effects

4. Application to UK data Candidate series: Indicator Unit Real GDP (Q) Millions of pounds (constant) Current account (Q) Millions of pounds Gross national savings (Q) Housing completions (Q) Number of permanent dwellings completed Unemployment rate (Q) Level Nominal earnings (Q) Nominal earnings index, including bonuses Total employment (Q) Thousands of people Average hours (Q) Average actual hours per week Interest rate (Q) Exchange rate (Q) ERI exchange rate for the domestic currency Asset prices (Q) Equity price index Oil price (Q) $/barrel

4. Application to UK data Preliminary inputs: Selection of the variable [sel=0 not selected; sel=1 selected] Cyclical behavior of the selected variables, accompanying the GDP [pro=1 procyclical; pro=-1 countercyclical] Order of integration [drift=0 I(1); drift=1 I(2)] Unit specification [transf=0 no transformation; transf=1 logs; transf=2 z/100]

4. Application to UK data Selection process: Every variable is modelled in a bivariate framework together with real GDP, the candidates not passing the significance test are removed Average relative revision indicator: average gap between the filtered (one-sided) and smoothed (two-sided) estimates of the output gap, normalized by the maximum range of the filtered estimate Average relative uncertainty indicator: ratio between the average standard error and the maximum range of the filtered estimate

4. Application to UK data Bivariate models:

4. Application to UK data Bivariate models: SPEC 4 (UR) SPEC 3 (HC) SPEC 9 (ER)

4. Application to UK data Backtest: loadings SPEC 4 (UR) SPEC 3 (HC) SPEC 9 (ER)

4. Application to UK data Backtest: cycles SPEC 4 (UR) SPEC 3 (HC) SPEC 9 (ER)

4. Application to UK data 4 Variables model:

4. Application to UK data UR Backtest 4V: loadings HC ER

4. Application to UK data Backtest 4V: cycle

4. Application to UK data Backtest 4V: When turning from the bivariate to the full model set-up, which includes GDP altogether with the three selected variables, the transition is far from smooth. Collinearity amongst the cyclical components can potentially generate imprecise point estimates that, combined with a flat likelihood function, may cause “jumps” in the estimations, rendering output gap estimates unstable. In particular, instability is directly related with the estimates of the autoregressive dynamics of the cyclical component of GDP (ϕ parameters). A practical and operational fix consists in incorporating additional information in the estimation process. For this purpose, model averaging through the more stable bivariate estimates is performed.

4. Application to UK data 4 Variables model paramfix: Paramfix

4. Application to UK data Comparison:

5. Conclusions Database needs to be improved in order to avoid the problem of an unbalanced panel, and can consider a large number of candidate series. There is no clear algorithm for the selection of the variables to be included in the final specification: incrementalist vs brute force consideration. Despite all the above, the estimate is in accordance with official recession dating. It is well aligned with external estimations, although some of them are two-sided filters and thus include additional information.

ANNEX

4. Application to UK data HC_ER Trivariate models: HC_UR UR_ER