1. United Nations Statistics Division
Benchmarking
Training Workshop on the Compilation of Quarterly National Accounts for Economic Cooperation Organization Member Countries
8-11 March 2015
Tehran, Islamic Republic of Iran
United Nations Statistics Division

2. Outline What is benchmarking? Purpose of benchmarking
Benchmarking methods
Benchmarking software for QNA
Conclusions
Questions

3. What is benchmarking?
Benchmarking is a statistical technique to correct inconsistencies between estimates of the same variable obtained from data collected at different frequencies to produce a consistent time series
Example: quarterly and annual value-added estimates which are compiled using different data sources such as quarterly and annual surveys
It is usually done retrospectively as annual benchmark data are available sometime after the quarterly data

4. What is benchmarking?
Reasons for differences between quarterly and annual data sources
Difference in coverage and sample
Annual survey has broader coverage and a more representative sample
Differences in frame and sample size may exist
Difference in definitions and variables
Output replaces value added for growth measures
Accounting methods
Different quarterly and annual accounting methods
Estimation method, non-response treatment, respondent error, sampling error...

5. What is benchmarking? Aspects of benchmarking
Quarterization of annual data to construct time series of historical QNA estimates (back series) and revise preliminary QNA estimates to align them with new annual estimates when they become available
Extrapolation to update the series from movements in the indicator for the most current period (forward series)

6. Purpose of benchmarking
Combines the relative strengths of low-frequency data (say, annual data) and high-frequency data (say, quarterly data) while preserving as much as possible the short-term movements
Creates a coherent high-frequency data series by correcting the difference between benchmark and indicator values (indicator bias)
Ensures benchmark to indicator (BI) ratio becomes 1
Example: Adjust indicator so that

7. Purpose of benchmarking
Ensures, for forward series, that the sum of the four quarters of the current year is as close as possible to the unknown future annual data
Improves quality of national accounts data in terms of comparability and coherence

8. Benchmarking methods Numerical methods Statistical modelling methods
Pro rata distribution method
Proportional Denton method
Statistical modelling methods
ARIMA-model-based methods
General least-squares regression models
Cholette Dagum
Chow-Lin

9. Pro rata distribution method
Benchmarking methods
Pro rata distribution method
For any benchmark year, distribute annual value in direct proportion to the quarterly values
is the level of the QNA estimate for quarter q of year y
is the level of the indicator for quarter q of year y
is the level of the annual benchmark value for year y
is the BI ratio

10. Pro rata distribution method
Benchmarking methods
Pro rata distribution method
Advantages
Easy to compute and interpret
No special software needed
Quarterly estimates can be derived each year independently
Estimates are well aligned to benchmark value and are fairly reliable when BI ratios are stable
Disadvantages
Smoothens quarterly estimates only within a year
Concentrates bias in one quarter and cause abrupt change (“step problem”)
Not recommended for longer time series

11. Example of pro rata distribution method
Benchmarking methods
Example of pro rata distribution method
Yellow cell shows step problem caused by change in BI ratio
Source: IMF QNA manual (2001)

12. Pro rata distribution method – step problem
Benchmarking methods
Pro rata distribution method – step problem
Source: IMF QNA manual (2001)

13. Pro rata distribution method Benchmark-to-indicator ratio
Benchmarking methods
Pro rata distribution method Benchmark-to-indicator ratio
Source: IMF QNA manual (2001)

14. Proportional Denton method
Benchmarking methods
Proportional Denton method
Goal: Find new estimates with minimal deviation from original indicator series
under restriction (for flow series):
is time
is the derived QNA estimate for quarter t
is the level of the indicator for quarter t
is the annual data for year y
is the last year for which an annual benchmark is available
is the last quarter for which quarterly source data are available

15. Proportional Denton method
Benchmarking methods
Proportional Denton method
Avoids the step problem seen in the pro rata distribution method by implicitly constructing from the annual observed BI ratios a time series of quarterly benchmarked QNA estimates-to-indicator (quarterly BI) ratios that
Is as smooth as possible
Average to the annual BI ratios for each year for the back series
Are kept constant and equal to the ratio for the last quarter of the last benchmark year
Ensures quarterly growth rates are adjusted by relatively similar amounts

16. Example of proportional Denton method
Benchmarking methods
Example of proportional Denton method
No big jump in period-to-period change
Source: IMF QNA manual (2001)

17. Proportional Denton method Solution to step problem
Benchmarking methods
Proportional Denton method Solution to step problem
Source: IMF QNA manual (2001)

18. Proportional Denton method Benchmark-to-indicator ratio
Benchmarking methods
Proportional Denton method Benchmark-to-indicator ratio
Source: IMF QNA manual (2001)

19. Proportional Denton method
Benchmarking methods
Proportional Denton method
Method requires that the indicator contains positive values only
For series with zeros but no negative values
Replace zeroes with values infinitesimally close to zero
For series with both negative and positive values and are derived as the difference between two non-negative series (for example, changes in inventories)
Apply method to opening and closing inventory levels, or
Turn the indicator into positive series by
Adding a large constant to all periods
Doing benchmarking
Removing the large constant from the results

20. Proportional Denton method
Benchmarking methods
Proportional Denton method
Impact on back series
QNA quarter-to-quarter growth rates differ from those in the indicator
May introduce new turning points, or
May change timing of turning points
These changes are desirable result of incorporating information in the annual data
Impact on forward series
Quarter-on-quarter growth rates are identical to those in the indicator
But, annual growth rate for the first year of the forward series differs from that of the source data

21. Proportional Denton method
Benchmarking methods
Proportional Denton method
Forward series estimates are of great interest to users
Forward series estimates can be improved to reduce size of future revisions by using enhanced version of proportional Denton method which forecast annual BI ratios by incorporating information on past systematic movements in the annual BI ratios
Example
A study of movements in annual BI ratios show that the indicator on average understates annual rate of growth by 2.0%
Adjust annual BI ratio for forward series by 2.0%

22. Example of enhanced proportional Denton method
Benchmarking methods
Example of enhanced proportional Denton method
Source: IMF QNA manual (2001)

23. Statistical modelling methods
Benchmarking methods
Statistical modelling methods
Take into account any known information about the stochastic properties of the series being benchmarked
Were mostly actually proposed to improve survey estimates where the survey design may provide information about the stochastic (sampling) properties of the series
In QNA, information about the stochastic properties of the series is usually non-existent and non-sampling errors may be more important than sampling errors

24. Statistical modelling methods
Benchmarking methods
Statistical modelling methods
Unlike the Denton method, these methods may over-adjust the series by interpreting as errors, and thus removing, true irregular movements that do not fit the regular patterns of the statistical model
Models may mis-specify the error terms, resulting in inaccurate estimates

25. Benchmarking software for QNA
ECOTRIM
XLPBM

26. Benchmarking software for QNA
Was developed by Statistics Canada
Runs under Unix/DOS operating systems
Allows users to choose among many alternative options offered by the Cholette-Dagum regression-based model
Requires input data to be prepared in pre-formatted text files

27. Benchmarking software for QNA
ECOTRIM
Was developed by Eurostat
Runs in Windows
Provides interactive and batch support
Allows input data to be prepared in text files or spreadsheets
Allows validation of results using graphical tools
Has option for Denton method

28. Benchmarking software for QNA
XLPBM
Was recently developed by IMF Statistics Department
Comprises a Microsoft Excel add-in function with options for the enhanced proportional Denton and Cholette-Dagum methods
Is particularly useful for QNA processing systems based on spreadsheets.

29. Conclusions Benchmarking is needed to derive QNA series that are
Temporally consistent with the ANA benchmarks
Preserve as much as possible the movements in the quarterly indicators
Provide accurate extrapolations for the current year
There are a number of methods to perform benchmarking
The pro rata distribution method is not recommended due to the step problem
The proportional Denton method is robust, simple and well suited for large-scale applications
A number of software are available to do benchmarking

30. Questions
What method is your country using to do benchmarking in the compilation of quarterly GDP estimates? Why?
Is the benchmarking method described in the national accounts methodological notes?

31. Thank you