Measuring Data Quality and Compilation of Metadata Second Regional Workshop on the Compilation of Basic Economic Statistics in African Countries: Addis Ababa Thierno Aliou Balde United Nations Statistics Division October 16-19 2007 1

Outline Quality Assessment Frameworks (DQAF) for Basic Economic Statistics (BES) Quality Indicators versus Direct Quality Measures Metadata on BES Recommendations - Conclusion

1. QAF for BES Quality measurement of BES is concerned with providing the user with sufficient information to judge whether or not the data are of adequate quality for their intended use, i.e. to judge their “fitness for use” . For example, data users must be able to verify that the conceptual framework and definitions that would satisfy their particular data needs are the same as, or sufficiently close to those employed in collecting and processing the data.

1. QAF for BES Several statistical organizations and countries have developed definitions of quality and integrated them into QAF. Although the existing quality assessment frameworks slightly differ in their approaches to quality and number/name of quality dimensions, they complement each other and provide comprehensive and flexible structures for the qualitative assessment of a broad range of statistics

1. QAF for BES 1. The IMF quality concept: covers the prerequisites and five dimensions of quality – assurance of integrity, methodological soundness, accuracy and reliability, serviceability and accessibility. 2. The European Statistical System (Eurostat): considers six criteria – relevance, accuracy, timeliness and punctuality, accessibility and clarity, comparability and coherence. 3. The OECD quality concept: quality is viewed in terms of seven dimensions – relevance, accuracy, credibility, timeliness, accessibility, interpretability and coherence.

1. QAF for BES The overall aim of QAFs is to standardize and systematize statistical quality measurement and reporting across countries. They allow an assessment of national practices to be made against internationally accepted statistical approaches for quality measurement. QAFs could be used in a number of aspects, including for: - Guiding countries’ efforts for strengthening their statistical systems by providing a self-assessment tool and for identifying areas of improvement; - Technical assistance purposes; - Reviews of particular statistical domains performed by international organization; - Assessment by other groups of data users.

1. QAF for BES Dimensions of Quality: Prerequisites Relevance Credibility/Assurance and Integrity Accuracy/ Accuracy and Reliability Timeliness/ Serviceability Methodological Soundness/Interpretability Coherence/ Consistency Accessibility

1. QAF for BES Dimensions of Quality (cont.) These dimensions of quality form a complex relationship: - overlapping - interrelated Measurement of BES data quality is not a simple task.

1. QAF for BES Dimensions of Quality (cont.) Prerequisites of quality refer to all institutional and organizational conditions that have an impact on the quality of BES data. Relevance: reflects the degree to which BES data meets the real needs of the users. Credibility: refers to the confidence that users place in the data based on the image of the statistical agency that produce the data. Accuracy: it is the degree to which the data correctly estimate or describe the characteristics they are designed to measure.

1. QAF for BES Dimensions of Quality (cont.) Timeliness: refers to the delay between the end of the reference period to which the data pertain, and the date on which the data are released. The methodological soundness/interpretability: refers to the application of international standards, guidelines and good practices in production of BES. - The metadata provided along with BES play a crucial role for assessing the methodological soundness of data. - Closely related to the interpretability which reflects the ease with which the user may understand and properly use/analyze the data.

1. QAF for BES Dimensions of Quality (cont.) Coherence/consistency: reflects the degree to which the data are logically connected and mutually consistent: - Coherence within dataset - Coherence across datasets - Coherence over time - Coherence across countries. Accessibility: refers to the ease with which data can be obtained from the statistical office.

2. Quality Indicators versus Direct Quality Measures Items that directly measure a particular aspect of quality. Example: the time lag from the reference date to the release of the data. In practice quality measures are difficult or costly to calculate.

2. Quality Indicators versus Direct Quality Measures Summarized quantitative data to provide evidence about the quality of the data. They do not measure quality directly but provide enough information for the assessment of quality. Types of quality indicators: - Key indicators - Supportive indicators - Indicators for further analysis

3. Metadata on BES Statistical data consists generally of: Micro data: data on the characteristics of units of the population. Macro data: derived from the micro data by grouping or aggregation. Metadata: “data about data”, describes the micro data, macro data or other metadata.

3. Metadata on BES Statistical metadata describes or document statistical data. The most fundamental purpose of metadata is to: - help users understand, interpret and analyze the data. - help the producers of statistics to enhance the production and the dissemination of the data.

3. Metadata on BES A bi-directional relationship between metadata and quality: - metadata describe the quality of statistics - metadata are a quality component. Metadata provide a mechanism for comparing national practices in the compilation of BES. A broad spectrum of metadata requirements has to be addressed for the wide range of possible users and uses of metadata.

3. Metadata on BES Six dimensions of metadata: - Data coverage, periodicity and timeliness - Access by public - Integrity of the disseminated data - Quality of the disseminated data - Summary methodology - Dissemination format Each of these dimensions is characterized by a few elements that can be observed or monitored by the users.

4. Recommendations - Conclusion It is recommended that: Countries take into account the above dimensions of quality in the process of producing BES. Countries use the dimensions of quality when measuring and reporting the quality of BES. Countries undertake a quality review of BES every four or five years or more frequently if significant methodological changes or changes in the data sources occur.

4. Recommendations - Conclusion It is recommended that: Countries pay careful attention to maintain a correct balance between different dimensions of quality and use a minimum of indicators. Countries satisfy the following criteria when defining the quality indicators for BES: - Coverage of part or all of the 8 dimensions of quality - Well established methodology for their compilation - Indicators are easy to interpret. Countries adopt the segmentation of users into groups and a layered approach to metadata presentation.

4. Recommendations - Conclusion It is recommended that: Countries adopt a development of a coherent system and a structured approach to metadata across all areas of economic statistics, focusing on improving their quantity and coverage.

4. Recommendations - Conclusion Countries are encouraged to: Use a system of quality indicators and develop their own BES quality framework based on the quality dimensions and the specific circumstances in their economy. Issue regularly, quality reports as part of their metadata. Give a high priority to the development of metadata and to consider their dissemination an integral part of dissemination of BES.

Thank You