1. Session 7.1 Data Quality 1

2. Introduction
The objective of quality measurement of international merchandise trade statistics is twofold: to provide producers with the appropriate information to monitor and further enhance data quality and to provide users with sufficient information to judge whether the data are adequate for their intended use.
The development of a Quality framework is needed to fulfill properly users needs. It can be considered as a third pillar, in addition to the delivery of data and metadata.
Various Quality frameworks have been developed at national and international level (IMF, EUROSTAT, OECD, ECB,…)
Quality of statistics is a multidimensional concept and is not so easy to measure.
There is no unique structure between the various frameworks but many common or related dimensions.
The ISO 9000 standards gives a frame for quality management but need to be adapted to the specific domain of statistics.

3. Definition and preliminary remarks
Quality is generally defined as “the totality of features and characteristics of a product or service that bear on its ability to satisfy stated or implied needs”
A quality assessment programme should cover:
the quality of outputs (measured on the basis of a “quality model” composed of various dimensions/components)
the analysis of procedures included in the collection/production process in order to ensure the quality of outputs.
The quality assessment programme concerns together:
Individual elements of data (record level)
General issues such as: full coverage, application of international definitions and recommendations,… 3

4. IMTS 2010 recommendations on quality
Outline
Quality models
IMTS 2010 recommendations on quality
Quality measurement of trade statistics : EUROSTAT example
Practical recommendations 4

5. Quality models 1.1 European Statistical System (ESS) Standard for Quality Reports
Provides recommendations for preparation of comprehensive Quality reports
Recommendations are linked to the European Statistics Code of Practice
Recommended standard structure:
Introduction
- brief history of the statistical process
- overview of all outputs
- reference to documentation, especially on methodology
Relevance
Degree to which statistical outputs meet current and potential user needs. It depends on whether all the statistics that are needed are produced and the extent to which concepts used (definitions, classifications,…) reflect user needs
Accuracy
Accuracy of outputs is the degree of closeness of estimates to the true value
Should be covered: (i) coverage errors, (ii) measurement errors, (iii) non response errors, (iv) processing errors

6. Quality models 1.1 European Statistical System (ESS) Standard for Quality Reports
Accuracy
coverage errors: are due to divergences between the target population and the frame population
Example: free zones excluded from the statistical territory, trade below threshold…
What should be included on Coverage Errors:
An assessment, preferably quantitative, on the extent of undercoverage and the bias risks associated with it.
Actions taken for reduction of undercoverage and associated bias risks,
measurement errors: are errors that occur during data collection and cause recorded values of variables to be different from the true ones
Example: Wrong typing by declarants, misclassification of goods,…
What should be included on Measurement Errors:
Identification and general assessment of the main risks in terms of measurement error.
Information on failure rates during data editing.
Efforts made in the collection process on error reduction.
Forms used should be annexed (if very long by hyperlink)

7. Quality models 1.1 European Statistical System (ESS) Standard for Quality Reports
Accuracy
non response errors: Non response is the failure of a sample survey or a census to collect data for all data items in the survey questionnaire from all the population units designated for data collection.
Example: missing declarations, missing data (supplementary units,…)
What should be included on Non response errors:
Non response rates according to the most relevant variables
A qualitative statement on the bias risks associated with non response
Measures to reduce non response
Technical treatment of non response at the estimation stage
processing errors: errors generated during data entry, data editing (checks and corrections), coding or imputation.
Example : errors introduced in the manual coding of declarations
What should be included on Processing Errors:
Identification of the main issues regarding processing errors for the statistical process and its outputs.
Where relevant and available, an analysis of processing errors affecting individual observations should be presented, else a qualitative assessment should be included.

8. Quality models 1.1 European Statistical System (ESS) Standard for Quality Reports
Timeliness and Punctuality
Timeliness of statistical outputs is the length of time between the event or phenomenon statistical process they describe and their availability
Punctuality is the time lag between the release date of data and the target date on which they were scheduled for release as announced in an official release calendar
Accessibility and Clarity
Accessibility and clarity refer to the simplicity and ease with which users can access statistics with the appropriate supporting information and assistance
Coherence and Comparability
The coherence of two or more statistical outputs refer to the degree to which the statistical processes by which they are generated used the same concepts and harmonised methods. Comparability is a special case of coherence, and refers to the aim of combining outputs to make comparisons over time, or across regions, or across domains.
Trade-offs between output quality Components
The report should deal with the trade-offs that have to be made assuming that the budget is limited and that factors contributing to improvements with respect to one component lead to deterioration with respect to another
Examples: trade-off relevance – timeliness; accuracy – timeliness,…

9. Quality models 1.1 European Statistical System (ESS) Standard for Quality Reports
Assessment of users needs and perceptions
The most effective method of evaluating user perceptions is a full scale user satisfaction survey
Performance, Cost and Respondent Burden
Resources must be effectively used. Respondent burden should be proportional to the needs of users and not excessive for respondents. Respondent burden should be measured and targets set for its reduction over time.
Confidentiality, Transparency and Security
The privacy of data providers, the confidentiality of the information and its use only for statistical purposes must be absolutely guaranteed. Statistical authorities must produce and disseminate statistics respecting scientific independence ad in an objective, professional and transparent manner in which all users are treated equitably.
More information on ESS standards and tools on quality reporting:

10. Quality models 1.2 Other Quality Models: IMF DQAF
IMF Data Quality Assessment Framework
First developed in 2001 by the IMF Statistics Department, its aim is to complement the quality dimension of the IMF Special Data Dissemination Standard and General Data Dissemination System and to support assessment of the quality of data provided by countries as background for IMF Reports on the Observance of Standards and Codes
A generic DQAF has been designed in It is based on five dimensions (assurances of integrity, methodological soundness, accuracy and reliability, serviceability, and accessibility) + a set of prerequisites
Several specific frameworks have been defined, including Balance of Payments statistics
IMF Data Quality Reference Site :

11. Quality models 1.2 IMF DQAF : generic model

12. Quality models 1.2 IMF DQAF

13. Quality models 1.2 IMF DQAF

14. Quality models 1.2 IMF DQAF14

15. Quality models 1.2 IMF DQAF

16. Quality models 1.2 IMF DQAF

17. IMTS Recommendations on data quality (Chapter 9)
It is recommended that countries develop quality standards and related good practices covering the institutional arrangements, the statistical processes and outputs.
It is recommended that countries develop a standard for regular quality reports which cover the full range of statistical processes and their outputs.
Such reports can be either producer-oriented with the aim to identify strengths and weaknesses of the statistical process and lead to or contain the definition of quality improvement actions or user-oriented with the aim to keep users informed on the methodology of the statistical process and the quality of the statistical output.
It is recommended that the quality reports of international merchandise trade statistics should be completed or updated at least every five years or more frequently if significant methodological changes or changes in the data sources occur.
It is recommended that countries base their quality reports on a set of quantitative and qualitative indicators for international merchandise trade statistics and on a checklist covering data collection, processing and dissemination to allow for an assessment of strengths and weaknesses in
the statistical process and to identify possible quality improvement actions.

18. Quality report on trade statistics IMTS 2010 Suggested indicators

19. Quality report on trade statistics IMTS 2010 Suggested indicators

20. Quality report on trade statistics IMTS 2010 Annex F: Coherence between IMTS and BOP item goods

21. IMTS Recommendations on data quality (Chapter 9) Cross-country data comparability
Non-comparability may be caused by:
differences in coverage, different treatment of certain goods (e.g., military goods, ship's stores, confidential data), value increases in intermediary countries, differences in classification of goods, time lags in reporting, differences in valuation including CIF/FOB, etc…
Such non-comparability may be substantially reduced by adopting concepts and definitions recommended in IMTS 2010
Nevertheless, a certain amount of non-comparability will remain (due to valuation and partner country attribution rules, reporting errors, fraudulent trade, etc…)
 To make statistics more accurate and useful both for national purposes and for international comparison, countries are encouraged to:
periodically conduct bilateral and multilateral reconciliation studies
Implement data exchanges

22. Quality report on trade statistics Example of Eurostat report
The quality is measured via a set of indicators.
These indicators are listed in the EU legal acts relating to the external trade statistics
 A basic act of the European Parliament and of the Council
 A Commission regulation for the implementing rules

23. Quality report on trade statistics Example of Eurostat report
The Quality Questionnaire and Report
The Quality Questionnaire is the tool used to assess the quality indicators.
It is an annual questionnaire addressed to all Member States and to be replied by the end of October.
The Quality Report is published once by year.
It is based on the information collected through the Quality Questionnaire.
It provides a summary of the main quality indicators for the EU and its Member States: timeliness, accuracy, accessibility, clarity, comparability, and coherence.

24. Quality report on trade statistics Example of Eurostat report
Quality indicators are well defined but not quality targets  Ongoing work
Introduce these quality targets into the EU legislation
 a revision of the EU legislation to be prepared
Monitor more closely the quality of data transmitted by the Member States and react more strongly against the Member States not compliant with the legal requirements
 As soon as the revised legislation comes in force.

25. Quality reports on trade statistics Practical recommendations
1st phase:
Draft an internal quality report (producer-oriented) using at least the quality indicators proposed in IMTS-2010
The report should be updated once a year in order to measure the progress achieved and identify room for improvement in the future
A subset can be produced with a monthly periodicity in order to monitor and improve the statistical process (in particular: results of validity checks should be closely monitored)
2nd phase :
Depending on the national policy (including the others main statistical sectors), the publication of a national Quality report on trade statistics could be envisaged.

26. Quality reports on trade statistics Practical recommendations
Thank you for your attention
Questions and remarks ?