Copyright 2010, The World Bank Group. All Rights Reserved. Principles, criteria and methods Part 1 Quality management Produced in Collaboration between World Bank Institute and the Development Data Group (DECDG)

Copyright 2010, The World Bank Group. All Rights Reserved. Definitions of quality/I Quality may be related either to subjective feelings or to objective facts Objectively, the quality of something is the sum of its essential attributes or properties. Quality has to do with user needs and satisfaction. Users can not define the quality standards. A minimum standard is to be transparent about the way in which concepts and methods are used. 2

Copyright 2010, The World Bank Group. All Rights Reserved. Definitions of quality/II Subjectively, something has quality when it meets the expectations of the user The International Organization for Standardization (ISO 8402) defines quality as: ‘Quality is the totality of features or characteristics of a product or service that bear on its ability to satisfy stated or implied needs of customers’. Openness about the data shortcomings and needed data revisions can help in getting understanding and support. 3

Copyright 2010, The World Bank Group. All Rights Reserved. Quality frameworks Some international organizations have developed quality frameworks for official statistics These frameworks generally use the same kind of criteria, but there are slight differences One of the most comprehensive quality framework for statistics is the Data Quality Assessment Framework (DQAF), developed by the IMF Another useful international framework is the European Statistics Code of Practice (ESCP) 4

Copyright 2010, The World Bank Group. All Rights Reserved. DQAF Quality dimensions The DQAF uses the following main dimensions of quality: Prerequisites of quality Assurances of integrity Methodological soundness Accuracy and reliability Serviceability Accessibility 5

Copyright 2010, The World Bank Group. All Rights Reserved. Prerequisites of quality Prerequisites are not quality criteria, but refer to institutional quality elements: Legal and institutional environment - the environment is supportive of statistics Resources - resources are commensurate with needs of statistical programs Relevance - statistics cover relevant information on the subject field Quality management - quality is a cornerstone of statistical work 6

Copyright 2010, The World Bank Group. All Rights Reserved. Prerequisites – the indicators/1 The responsibility for collecting, processing, and disseminating the statistics is clearly specified Data sharing and coordination among data producing agencies are adequate Individual reporters’ data are to be kept confidential and used for statistical purposes only Statistical reporting is ensured through legal mandate and/or measures to encourage response Staff, facilities, computing resources, and financing are commensurate with statistical programs Measures to ensure efficient use of resources are implemented 7

Copyright 2010, The World Bank Group. All Rights Reserved. Prerequisites – the indicators/2 The relevance and practical utility of existing statistics in meeting users’ needs are monitored and reported publicly Processes are in place to focus on quality Processes are in place to monitor the quality of the statistical program and reported publicly Processes are in place to deal with quality considerations in planning the statistical program and reported publicly 8

Copyright 2010, The World Bank Group. All Rights Reserved. Assurances of integrity Explanation: the principle of objectivity in the collection, processing, and dissemination of statistics is firmly adhered to Elements are: Professionalism Transparency Ethical standards 9

Copyright 2010, The World Bank Group. All Rights Reserved. Integrity – indicators/1 Statistics are produced on an impartial basis Choices of sources and statistical techniques as well as decisions about dissemination are informed solely by statistical considerations The appropriate statistical entity is entitled to comment on erroneous interpretation and misuse of statistics The terms and conditions under which statistics are collected, processed, and disseminated are available to the public 10

Copyright 2010, The World Bank Group. All Rights Reserved. Integrity – indicators/2 Internal governmental access to statistics prior to their release is publicly identified Products of statistical agencies/units are clearly identified as such Advanced notice is given of major changes in methodology, source data, and statistical techniques Guidelines for staff behavior are in place and are well known to the staff 11

Copyright 2010, The World Bank Group. All Rights Reserved. Methodological soundness Explanation: The methodological basis for the statistics follows internationally accepted standards, guidelines, or good practices Elements: Concepts and definitions have to be in accord with internationally accepted statistical frameworks Scope must be in accord with internationally accepted standards, guidelines, or good practices Classification/sectorization are in accord with internationally accepted standards, guidelines, or good practices Basis for recording – Flows and stocks are valued and recorded according to internationally accepted standards, guidelines, or good practices 12

Copyright 2010, The World Bank Group. All Rights Reserved. Methodology – Indicators/1 The overall approach to statistics, in terms of concepts and definitions follows internationally accepted standards, guidelines, or good practices, as well as the scope, and the classification system The scope is broadly consistent with internationally accepted standards, guidelines, or good practices Classification systems used are broadly consistent with internationally accepted standards, guidelines, or good practices 13

Copyright 2010, The World Bank Group. All Rights Reserved. Methodology – Indicators/2 Market prices are used to value flows and stocks Recording is done on an accrual basis Grossing/netting procedures are broadly consistent with internationally accepted standards, guidelines, or good practices 14

Copyright 2010, The World Bank Group. All Rights Reserved. Accuracy and reliability/1 Explanation: Source data and statistical techniques are sound and statistical outputs sufficiently portray reality Elements: Source data available provide an adequate basis to compile statistics Assessment of source data Statistical techniques employed conform to sound statistical procedures Assessment and validation of intermediate data and statistical outputs - Correct application of data collection, processing and data analysis. Correct sampling Correct application of the mathemat ical tools. 15

Copyright 2010, The World Bank Group. All Rights Reserved. Accuracy and reliability/2 Revision studies - Revisions, as a gauge of reliability, are tracked and mined for the information they may provide New methods may be needed to be introduced and existing methods may be needed to be revised and changed. Revisions also influence back-data. 16

Copyright 2010, The World Bank Group. All Rights Reserved. Accuracy and reliability – Indicators/1 Indicators of accuracy and reliability include several aspects related to source date: Data collection programs take into account country-specific conditions Source data reasonably approximate the definitions, scope, classifications, valuation, and time of recording required Timeliness Routinely assessment Data compilation employs sound statistical techniques to deal with data sources Other statistical procedures (e.g. data adjustments and transformations) employ sound statistical techniques 17

Copyright 2010, The World Bank Group. All Rights Reserved. Accuracy and reliability – Indicators / 2 Intermediate results are validated against other information where applicable Statistical discrepancies in intermediate data are assessed and investigated Statistical discrepancies and other potential indicators or problems in statistical outputs are investigated Studies and analyses of revisions are carried out routinely and used internally to inform statistical processes 18

Copyright 2010, The World Bank Group. All Rights Reserved. Serviceability/1 Explanation: Statistics, with adequate periodicity and timeliness, are consistent and follow a predictable revisions policy Elements: Periodicity and timeliness - Periodicity and timeliness follow internationally accepted dissemination standards 19

Copyright 2010, The World Bank Group. All Rights Reserved. Serviceability/2 Consistency - Statistics are consistent within the dataset, over time, and with major datasets Revision policy and practice - Data revisions follow a regular and publicized procedure 20

Copyright 2010, The World Bank Group. All Rights Reserved. Serviceability - Indicators Periodicity and timeliness follows dissemination standards Statistics are consistent within the dataset Statistics are consistent or reconcilable over a reasonable period of time Statistics are consistent or reconcilable with those obtained through other data sources Revisions follow a regular and transparent schedule Preliminary and/or revised data are clearly identified Studies and analyses of revisions are made public 21

Copyright 2010, The World Bank Group. All Rights Reserved. Accessibility Explanation: Data and metadata are easily available and assistance to users is adequate Elements: Data accessibility - Statistics are presented in a clear and understandable manner, forms of dissemination are adequate, and statistics are made available on an impartial basis Metadata accessibility - Up-to-date and pertinent metadata are made available Assistance to users - Prompt and knowledgeable support service is available 22

Copyright 2010, The World Bank Group. All Rights Reserved. Accessibility – Indicators/1 Statistics are presented in a way that facilitates proper interpretation and meaningful comparisons (layout and clarity of text, tables, and charts) Dissemination media and format are adequate Statistics are released on a pre-announced schedule Statistics are made available to all users at the same time Statistics not routinely disseminated are made available upon request 23

Copyright 2010, The World Bank Group. All Rights Reserved. Accessibility – Indicators/2 Documentation on concepts, scope, classifications, basis of recording, data sources, and statistical techniques is available, and differences from internationally accepted standards, guidelines, or good practices are annotated Levels of detail are adapted to the needs of the intended audience Contact points for each subject field are publicized Catalogs of publications, documents, and other services, including information on any changes, are widely available 24