United Nations Statistics Division Benchmarking Training Workshop on the Compilation of Quarterly National Accounts for Economic Cooperation Organization Member Countries 8-11 March 2015 Tehran, Islamic Republic of Iran United Nations Statistics Division

Outline What is benchmarking? Purpose of benchmarking Benchmarking methods Benchmarking software for QNA Conclusions Questions

What is benchmarking? Benchmarking is a statistical technique to correct inconsistencies between estimates of the same variable obtained from data collected at different frequencies to produce a consistent time series Example: quarterly and annual value-added estimates which are compiled using different data sources such as quarterly and annual surveys It is usually done retrospectively as annual benchmark data are available sometime after the quarterly data

What is benchmarking? Reasons for differences between quarterly and annual data sources Difference in coverage and sample Annual survey has broader coverage and a more representative sample Differences in frame and sample size may exist Difference in definitions and variables Output replaces value added for growth measures Accounting methods Different quarterly and annual accounting methods Estimation method, non-response treatment, respondent error, sampling error...

What is benchmarking? Aspects of benchmarking Quarterization of annual data to construct time series of historical QNA estimates (back series) and revise preliminary QNA estimates to align them with new annual estimates when they become available Extrapolation to update the series from movements in the indicator for the most current period (forward series)

Purpose of benchmarking Combines the relative strengths of low-frequency data (say, annual data) and high-frequency data (say, quarterly data) while preserving as much as possible the short-term movements Creates a coherent high-frequency data series by correcting the difference between benchmark and indicator values (indicator bias) Ensures benchmark to indicator (BI) ratio becomes 1 Example: Adjust indicator so that

Purpose of benchmarking Ensures, for forward series, that the sum of the four quarters of the current year is as close as possible to the unknown future annual data Improves quality of national accounts data in terms of comparability and coherence

Benchmarking methods Numerical methods Statistical modelling methods Pro rata distribution method Proportional Denton method Statistical modelling methods ARIMA-model-based methods General least-squares regression models Cholette Dagum Chow-Lin

Pro rata distribution method Benchmarking methods Pro rata distribution method For any benchmark year, distribute annual value in direct proportion to the quarterly values is the level of the QNA estimate for quarter q of year y is the level of the indicator for quarter q of year y is the level of the annual benchmark value for year y is the BI ratio

Pro rata distribution method Benchmarking methods Pro rata distribution method Advantages Easy to compute and interpret No special software needed Quarterly estimates can be derived each year independently Estimates are well aligned to benchmark value and are fairly reliable when BI ratios are stable Disadvantages Smoothens quarterly estimates only within a year Concentrates bias in one quarter and cause abrupt change (“step problem”) Not recommended for longer time series

Example of pro rata distribution method Benchmarking methods Example of pro rata distribution method Yellow cell shows step problem caused by change in BI ratio Source: IMF QNA manual (2001)

Pro rata distribution method – step problem Benchmarking methods Pro rata distribution method – step problem Source: IMF QNA manual (2001)

Pro rata distribution method Benchmark-to-indicator ratio Benchmarking methods Pro rata distribution method Benchmark-to-indicator ratio Source: IMF QNA manual (2001)

Proportional Denton method Benchmarking methods Proportional Denton method Goal: Find new estimates with minimal deviation from original indicator series under restriction (for flow series): is time is the derived QNA estimate for quarter t is the level of the indicator for quarter t is the annual data for year y is the last year for which an annual benchmark is available is the last quarter for which quarterly source data are available

Proportional Denton method Benchmarking methods Proportional Denton method Avoids the step problem seen in the pro rata distribution method by implicitly constructing from the annual observed BI ratios a time series of quarterly benchmarked QNA estimates-to-indicator (quarterly BI) ratios that Is as smooth as possible Average to the annual BI ratios for each year for the back series Are kept constant and equal to the ratio for the last quarter of the last benchmark year Ensures quarterly growth rates are adjusted by relatively similar amounts

Example of proportional Denton method Benchmarking methods Example of proportional Denton method No big jump in period-to-period change Source: IMF QNA manual (2001)

Proportional Denton method Solution to step problem Benchmarking methods Proportional Denton method Solution to step problem Source: IMF QNA manual (2001)

Proportional Denton method Benchmark-to-indicator ratio Benchmarking methods Proportional Denton method Benchmark-to-indicator ratio Source: IMF QNA manual (2001)

Proportional Denton method Benchmarking methods Proportional Denton method Method requires that the indicator contains positive values only For series with zeros but no negative values Replace zeroes with values infinitesimally close to zero For series with both negative and positive values and are derived as the difference between two non-negative series (for example, changes in inventories) Apply method to opening and closing inventory levels, or Turn the indicator into positive series by Adding a large constant to all periods Doing benchmarking Removing the large constant from the results

Proportional Denton method Benchmarking methods Proportional Denton method Impact on back series QNA quarter-to-quarter growth rates differ from those in the indicator May introduce new turning points, or May change timing of turning points These changes are desirable result of incorporating information in the annual data Impact on forward series Quarter-on-quarter growth rates are identical to those in the indicator But, annual growth rate for the first year of the forward series differs from that of the source data

Proportional Denton method Benchmarking methods Proportional Denton method Forward series estimates are of great interest to users Forward series estimates can be improved to reduce size of future revisions by using enhanced version of proportional Denton method which forecast annual BI ratios by incorporating information on past systematic movements in the annual BI ratios Example A study of movements in annual BI ratios show that the indicator on average understates annual rate of growth by 2.0% Adjust annual BI ratio for forward series by 2.0%

Example of enhanced proportional Denton method Benchmarking methods Example of enhanced proportional Denton method Source: IMF QNA manual (2001)

Statistical modelling methods Benchmarking methods Statistical modelling methods Take into account any known information about the stochastic properties of the series being benchmarked Were mostly actually proposed to improve survey estimates where the survey design may provide information about the stochastic (sampling) properties of the series In QNA, information about the stochastic properties of the series is usually non-existent and non-sampling errors may be more important than sampling errors

Statistical modelling methods Benchmarking methods Statistical modelling methods Unlike the Denton method, these methods may over-adjust the series by interpreting as errors, and thus removing, true irregular movements that do not fit the regular patterns of the statistical model Models may mis-specify the error terms, resulting in inaccurate estimates

Benchmarking software for QNA ECOTRIM XLPBM

Benchmarking software for QNA Was developed by Statistics Canada Runs under Unix/DOS operating systems Allows users to choose among many alternative options offered by the Cholette-Dagum regression-based model Requires input data to be prepared in pre-formatted text files

Benchmarking software for QNA ECOTRIM Was developed by Eurostat Runs in Windows Provides interactive and batch support Allows input data to be prepared in text files or spreadsheets Allows validation of results using graphical tools Has option for Denton method

Benchmarking software for QNA XLPBM Was recently developed by IMF Statistics Department Comprises a Microsoft Excel add-in function with options for the enhanced proportional Denton and Cholette-Dagum methods Is particularly useful for QNA processing systems based on spreadsheets.

Conclusions Benchmarking is needed to derive QNA series that are Temporally consistent with the ANA benchmarks Preserve as much as possible the movements in the quarterly indicators Provide accurate extrapolations for the current year There are a number of methods to perform benchmarking The pro rata distribution method is not recommended due to the step problem The proportional Denton method is robust, simple and well suited for large-scale applications A number of software are available to do benchmarking

Questions What method is your country using to do benchmarking in the compilation of quarterly GDP estimates? Why? Is the benchmarking method described in the national accounts methodological notes?

Thank you