1. The Big Data for Official Statistics Competition
– results and lessons learned
Bogomil Kovachev, Martin Karlberg, Boro Nikic, Bogdan Oancea, Paolo Righi and Patrick Weber

2. Competition outline Coherence with official statistics
Big Data use encouraged but not required
Open submissions encouraged but not required (often conflicts with the above requirement, as Big Data sources are often proprietary)
Up to 12 rounds of monthly submissions (reference month January onwards; final effective reference month depending on the track)
Scientific Committee composition at launch
Martin KARLBERG (Eurostat)
Bogomil KOVACHEV (Eurostat)
Boro NIKIC - Statistical Office of Slovenia (SURS)
Bogdan OANCEA - National Institute of Statistics - Romania (INS)
Paolo RIGHI - National Institute of Statistics - Italy (ISTAT)
Vincenzo SPIEZIA - Organization for Economic Co-operation and Development (OECD)
Patrick WEBER - Central Bank of the Federal Republic of Germany (Deutsche Bundesbank)
Chair:

3. Need monthly data (otherwise too few points)
Tracks
Seven tracks
Need monthly data (otherwise too few points)
Need to be of interest to researchers (to attract participation)
(section 5)

4. Participants
Prof. George Djolov: University of Stellenbosch and Statistics South Africa
Team ETLAnow: Research Institute of the Finnish Economy
JRC Team: Joint Research Centre - European Commission
Dr. Roland Weigand
University of Warwick Forecast Team

5. Evaluation measures Point estimate accuracy Directional accuracy
Density estimate accuracy
(section 7)

6. Results Full results are now publicly available online
Point estimate accuracy
Directional accuracy
Density estimate accuracy

7. Track1: Unemployment
Only one big data participant - ETLAnow AR(1) model using trends from Google search volumes
Robust Nowcasting Algorithm (RNA) - only IE
A series of univariate benchmark algorithms based on the R forecast package

8. Track1: Unemployment - variability in the data
LU and MT not part - too stable series due to rounding
R² = 0.44

9. Track 1: results
Univariate (benchmark) approaches performing quite well in the point estimate accuracy measure - models were retrained automatically (not changed) every month
RNA was fielded for only one Task (IE) and it performed best wrt directional accuracy
ETLAnow method only participated in the pointset accuracy - it was the best performing method for one of the tasks - BG

10. Track 2: HICP all items
Only one participant using big data in some approaches (JRC)
The first release (our benchmark) is relatively stable in general
All countries part of the competition
No seasonally adjusted data published
Re-referencing from 2005 to 2015 was announced after the competition was launched (old reference series kept -> small impact)

11. Track 2: HICP variability in the data

12. Track 2: Results
No model retraining for the JRC models during the competition
The big data approach P3Approach12 performed well for several countries - UK, NL, IE, FR (point estimate accuracy)
Many ties in the directional accuracy
A Bayesian model performed best for UK (point estimate accuracy and density estimate accuracy)
A combination using many exogenous variables (“almost” big data) performed best for the EA (point estimate and density estimate accuracy), FR (density estimate accuracy)
Models with exogenous data performed better than in Track 1
The approach containing the oil price exogenous variable performed particularly well only for IT

13. Track 3: HICP excluding energy
Only two participants (a third participant dropped out during the competition for technical reasons)
Re-referencing from 2005 to 2015 was announced after the competition was launched (old reference series not kept -> big impact). Month 1 submissions had to be discarded

14. Track 3: HICP excluding energy - results
Big economies (EA, EU, DE, FR, IT) seem to be easier to forecast in comparison with the case of the HICP headline aggregate (Track2)
Most models with exogenous data outperformed the benchmark models for DE, EA, FR and UK. Only for IT the benchmark was better (point estimate accuracy)
For directional accuracy the picture is often reverse suggesting the complementarity of the two measures

15. Track 4 and 5: Tourism nights spent
First estimate is not as stable (data is revised often for some countries)
Flight booking information was used as a big data variable (a closed submission)
For all countries there were big data approaches used - in 8 cases for Track 4 and 3 in Track 5 they were best
IE, EL, LU and the UK were not in this track due to data quality issues

16. Tracks 6 and 7
Most unstable of all tracks. Data are often revised; sometimes revisions are big
Only one participant with benchmark models took part
Gross data are not published for EU and EA
For many countries removing automotive fuel from the indicators seems to make the series significantly harder to predict. ( SI: 3.1% ->8.2%, RO: 1.3% -> 3.0%, PL: 0.7% -> 4.4%, NL:1.4% -> 4.7%, MT:1.3% -> 5.5% ...)

17. Conclusions
Attracting participants is hard (need to commit for one year)
Different evaluation measures seem to be complementary
Some evaluation measures are not sufficiently discriminatory (owing to their discrete nature)
Need for scientific committee to deliberate on the treatment of unforeseen situations (precision requriements, re-referencing…)
Using final (more stable) data would be better from a scientific viewpoint but would make results appear much later
Performance of big data models seems to vary between the indicators
The value of the data gathered extends beyond the official evaluation; they are available to all wishing to analyse them (at