1. Back-testing exercise
Carlos Cuerpo
Ángel Cuevas
Enrique M. Quilis
OG Working Group Workshop
Vilnius; September 5th, National Audit Office of Lithuania
Comunicación

2. Spanish case: Data set and processing Toolkit
Index
Introduction
Spanish case: Data set and processing
Toolkit
An application to UK Data
Conclusions 1

3. 1. Introduction
The main purpose of back-testing is to check the stability and robustness of the output gap estimates.
The suggested approach is based on the “beauty contest” approach (i.e., a search on the list of available variables for a given model, being the model a multivariate structural time series model). 1

4. 1. Introduction Selection criteria in “beauty contest” approach:
Statistical significance of the coefficients, focusing on the loadings of the observables on the cycle;
Average relative revision, defined as the average distance between one-sided and two-sided estimates, relative to the maximum amplitude of the output gap estimate;
Average relative uncertainty surrounding the cycle estimates, as the average standard error relative to the maximum amplitude.
Economic soundness, meaning that some key macroeconomic relationships could be captured by variables if included in the model
Amplitude and profile alignment with consensus figures (range given by a panel of official institutions) and in agreement with commonly accepted business cycle chronology (e.g. ECRI dating).
Stability of the one-sided cycle estimate in real time (revisions). 1

5. 2. Spanish case: Data set and pre-processing

6. 2. Spanish case: Data set and pre-processing
Seasonal and calendar effects correction
All series have been extended since and/or completed until the first quarter of 1980, taking into account their specificities (sources, concepts, different statistical bases, mixed frequencies, etc.)
Backward linking retropolation and temporal disaggregation when needed
Additional benchmarking techniques are implemented whenever the seasonal adjustment process breaks the temporal consistency with respect to the annual reference
At the end of the process, we have a balanced panel

7. 3. Toolkit Data Base Seasonal adjustment Benchmarking Series links
Estimation
Selector
Procyclical
Integration order
Transformation
Output
One-sided estimation (trend, cycle and drift)
Two-sided estimation (trend, cycle and drift)
Backtest
Number of backtest
Loadings stabilty
S.E. stability
Trend and cycle stability

8. 4. Application to UK data Main problems: Unbalanced panel
Only 11 series starting in 1980Q1 and ending in 2017Q4
All seasonal and calendar corrected, except “New buildings direction permissions. Total building surface” Tramo-Seats has been used to adjust it from seasonality and calendar effects

9. 4. Application to UK data Candidate series: Indicator Unit
Real GDP (Q)
Millions of pounds (constant)
Current account (Q)
Millions of pounds
Gross national savings (Q)
Housing completions (Q)
Number of permanent dwellings completed
Unemployment rate (Q)
Level
Nominal earnings (Q)
Nominal earnings index, including bonuses
Total employment (Q)
Thousands of people
Average hours (Q)
Average actual hours per week
Interest rate (Q)
Exchange rate (Q)
ERI exchange rate for the domestic currency
Asset prices (Q)
Equity price index
Oil price (Q)
$/barrel

10. 4. Application to UK data Preliminary inputs:
Selection of the variable [sel=0 not selected; sel=1 selected]
Cyclical behavior of the selected variables, accompanying the GDP [pro=1 procyclical; pro=-1 countercyclical]
Order of integration [drift=0 I(1); drift=1 I(2)]
Unit specification [transf=0 no transformation; transf=1 logs; transf=2 z/100]

11. 4. Application to UK data Selection process:
Every variable is modelled in a bivariate framework together with real GDP, the candidates not passing the significance test are removed
Average relative revision indicator: average gap between the filtered (one-sided) and smoothed (two-sided) estimates of the output gap, normalized by the maximum range of the filtered estimate
Average relative uncertainty indicator: ratio between the average standard error and the maximum range of the filtered estimate

12. 4. Application to UK data
Bivariate models:

13. 4. Application to UK data Bivariate models: SPEC 4 (UR) SPEC 3 (HC)
SPEC 9 (ER)

14. 4. Application to UK data Backtest: loadings SPEC 4 (UR) SPEC 3 (HC)
SPEC 9 (ER)

15. 4. Application to UK data Backtest: cycles SPEC 4 (UR) SPEC 3 (HC)
SPEC 9 (ER)

16. 4. Application to UK data
4 Variables model:

17. 4. Application to UK data UR
Backtest 4V: loadings HC ER

18. 4. Application to UK data
Backtest 4V: cycle

19. 4. Application to UK data Backtest 4V:
When turning from the bivariate to the full model set-up, which includes GDP altogether with the three selected variables, the transition is far from smooth.
Collinearity amongst the cyclical components can potentially generate imprecise point estimates that, combined with a flat likelihood function, may cause “jumps” in the estimations, rendering output gap estimates unstable.
In particular, instability is directly related with the estimates of the autoregressive dynamics of the cyclical component of GDP (ϕ parameters). A practical and operational fix consists in incorporating additional information in the estimation process. For this purpose, model averaging through the more stable bivariate estimates is performed.

20. 4. Application to UK data
4 Variables model paramfix:
Paramfix

21. 4. Application to UK data
Comparison:

22. 5. Conclusions
Database needs to be improved in order to avoid the problem of an unbalanced panel, and can consider a large number of candidate series.
There is no clear algorithm for the selection of the variables to be included in the final specification: incrementalist vs brute force consideration.
Despite all the above, the estimate is in accordance with official recession dating.
It is well aligned with external estimations, although some of them are two-sided filters and thus include additional information.

23. ANNEX

24. 4. Application to UK data
HC_ER
Trivariate models:
HC_UR
UR_ER