1. Benchmarking for short-term economic statistics
Richard McKenzie & David Brackfield OECD

2. Definition
Benchmarking is the process of aligning estimates of an economic variable at high frequency (e.g. monthly / quarterly) with independent estimates of the same economic variable produced at low frequency (e.g. annual)
Assumes the low frequency source is more accurate than the high frequency source

3. Background Presentation on methods at 2002 STESEG
Presentation on specific conditions for benchmarking STES at 2003 STESEG
OECD / Eurostat workshop on benchmarking in November 2003
Collection of relevant technical documents brought together on OECD website – also part of the STES Timeliness Framework
Availability of the ECOTRIM software
As part of the work of the taskforce on timeliness and benchmarking.
This presentation will not discuss methodology, this has been done before and there are a wealth of reliable references provided on the OECD website

4. Use of benchmarking techniques
Common practice for quarterly and annual national accounts
Not common practice amongst NSOs for other short-term economic statistics
Advantages of using benchmarking techniques for STES emphasised in EU-US comparison study from 2001
3 The outcome of this report was a major reason why benchmarking was given a high priority at the 2002 STESEG meeting

5. Advantages of using benchmarking techniques for short-term economic statistics
Consistency in high and low frequency data for the same economic variable
Procedure to review discrepancies in high and low frequency estimates improve estimation methods
Improved accuracy lower sample sizes, lower costs or opportunity to improve timeliness
Produces an accurate long time series of high frequency data important for empirical analysis

6. Advantages of using benchmarking techniques for short-term economic statistics …. cont
Improved quality of input series to quarterly and annual national accounts helps minimise discrepancies for distinguishable components
BEWARE
Need to ensure less frequent (e.g. annual) estimates are consistent between years to be suitable for benchmarking
Focus of NSOs may often be on producing the highest quality data on annual levels, independent of previous estimates.
May often introduce changes to annual survey processes between years, e.g. incorporation of administrative data, sample redesigns, questionnaire redesigns, changes to data collection methodology etc.
Such changes can have a large impact on the comparability of annual level estimates for a particular variable between years, such that the estimate of percentage change from the previous years estimate may not be reliable. Under these circumstances, the annual series would not be appropriate to use as a benchmark source for the related short-term series.
Therefore, the impact of such changes needs to be estimated and annual series backcast before it could be used for benchmarking

7. Is benchmarking for STES a priority?
Judging from countries comments on the benchmarking paper ……………..…
40% high, 50% medium, 10% low
Some significant country comments:
Very important due to the push towards using more administrative data for STES
Needs to be considered in conjunction with the design of structural surveys …
More discussion needed on the series requiring benchmarking ……. (Recent Eurostat study)
Data confrontation at unit record level for annual and short-term surveys is more important …
At the recent Eurostat STS Working Party meeting, a presentation was given to look at the difference between indexes of annual estimates derived from Structural Business Statistics and those from Short-Term Statistics for Retail and Wholesale Trade turnover, Production in Manufacturing and Production in Construction.

8. Achieving international comparability ECOTRIM
Software package developed by Eurostat
Made freely available after the OECD / Eurostat workshop on benchmarking
14 organisations have requested the software to use on a a variety of topics
Contains internationally recognised techniques and there are plans for further development
Country comments on the benchmarking paper:
50% interested in trialling, 50% needed more information
About half of the requests have concerned GDP, with the other half being a combination of general economic modelling and other STES. OECD used it in a research project to look at producing quarterly unit labour costs through benchmarking annual national accounts based estimates to quarterly indicator series.
Presented as a resource to both provide NSIs with the facility and ensure comparability of methods used. Of course countries who already have integrated programs may not be interested.

9. ECOTRIM Software
Software is freely available and can be installed onto the desktop
A User Manual that includes case studies and examples
Program is windows based and follows the normal “Windows” layout and feel
The software is easily installed on the desktop via standard windows procedure and the user manual comes in Adobe format.
The system itself is windows based and has the standard menu drop boxes.

10. Simple Example (Univariate)
In this example a monthly OECD MEI Production Index (industry excluding construction) will be benchmarked against the equivalent annual Gross Domestic Product in industry index for the same country. The result is a benchmarked monthly IIP estimate.
First the aggregated series is loaded into ECOTRIM.
Next the related series is loaded into ECOTRIM.
The idea of the example is to demonstrate the ease at which the software can be used. ECOTRIM is also available in SAS and GAUSS.
There are a number of options that can be chosen when selecting data and the methods on which to benchmark. This will depend on the descriptive statistics and the requirements and needs of the NSO. It should be noted that ECOTRIM is a resource for NSOs and one that they would need to adapt to their environments.
This example has been chosen as the OECD receives and calculates these two series from different sources for some or similar variables for which the annual is considered more reliable. The system can undertake benchmarking of flow, index and stock series.

11. Univariate Methods
The next step is to decide on a univariate method. All possible combinations can be tried (simultaneously if wanted).
Possible methods are:
AR (1) MIN SSR or AR (1) MAX LOG
Fernandez
Litterman MIN SSR or Litterman MAX LOG
The system now requests a benchmarking statistical method be chosen.
Boots, Feibes and Lisman and Denton are purely mathematical in nature and perform a pure adjustment procedure.
With this univariate example we have 5 choices.
Ultimately the statistician will be required to run all models and compare the statistical analysis to make an informed choice for their situation.
The developers or ECOTRIM have chosen these statistical benchmarking methods, there are many to chose from and from my understanding the system could be modified to accept and incorporate a new method.

12. Results The results can be displayed in the system graphically
The results can be exported into Excel with or without full information
Software can benchmark using multivariate methods with contemporaneous constraints and preliminary estimates. Software can also run in batch mode.
The graph contains three lines – Blue, Green Red.
The red line are the annual estimates for the GDP series
The green line are the quarterly production series
The blue line is the new monthly benchmarked production series that maintains the constraints of the annual series.
Data can be easily exported into excel either as just the series or with all the statistical results for analysis.
For more complex benchmarking where temporal as well as accounting constraints have to be maintained the system can undertake multivariate analysis. The system can also be run in batch mode where a large numbers of series have to generated.
From this graph you can see that early data was inconsistent but is now a pretty good fit.

13. Possible Future Work Issues for discussion:
Possible area of future work if countries want to trial benchmarking and present results next year?
OECD could coordinate this work and provide comparative analysis of lessons learnt.
Further discussion?
Have a discussion here. See if any countries are interested and/or anyone is willing to try at NSO and present analysis next year?
OECD would offer coordination role and help if required.
Note that some countries provided comments to the effect that there is no interface between their current systems and ECOTRIM or that they would have to design an interface. Need to reinforce that ECOTRIM is a possible resource that countries can use to undertake experimental analysis on possible benchmarking work in their offices first. The main push is to try and get countries undertaking benchmarking first for data consistency and the improved quality of short-term statistics.