Copyright 2010, The World Bank Group. All Rights Reserved. Principles, criteria and methods Part 2 Quality management Produced in Collaboration between World Bank Institute and the Development Data Group (DECDG)

Copyright 2010, The World Bank Group. All Rights Reserved. 15 principles: PROFESSIONAL INDEPENDENCE MANDATE FOR DATA COLLECTION ADEQUACY OF RESOURCES QUALITY COMMITMENT STATISTICAL CONFIDENTIALITY IMPARTIALITY AND OBJECTIVITY SOUND METHODOLOGY Continued: APPROPRIATE STATISTICAL PROCEDURES NON-EXCESSIVE BURDEN ON RESPONDENTS COST EFFECTIVENESS RELEVANCE ACCURACY AND RELIABILITY TIMELINESS AND PUNCTUALITY COHERENCE AND COMPARABILITY ACCESSIBILITY AND CLARITY European Statistics Code of Practice 2

Copyright 2010, The World Bank Group. All Rights Reserved. ESCP questionnaire Provides a tool for self-assessment of quality aspects Designed in a way that most parts can be filled in centrally, e.g. by the quality manager of the organization Follows the structure of the European Statistics Code of Practice - subdivided into 15 principles and 3-7 indicators for each principle For each principle the questionnaire concludes with a follow-up part in which statistical authorities are requested to reflect upon improvement actions and a time frame 3

Copyright 2010, The World Bank Group. All Rights Reserved. Quality management in practice Quality management in the practice of statistical offices has two main aspects: Assessing the quality of statistical outputs – users judge the quality of outputs on aspects relating to their use of the statistics. To meet user quality requirements, a framework for assessing the quality of outputs should be used Managing quality in the ongoing production of statistics - a standard set of indicators for different parts of the production process should be used 4

Copyright 2010, The World Bank Group. All Rights Reserved. Assessing the quality of outputs The key assessment aspects for the users are: Relevance Accuracy Coherence Interpretability Timeliness Accessibility of data and metadata To measure output quality for each of those, indicators can be used 5

Copyright 2010, The World Bank Group. All Rights Reserved. Output quality indicators – accuracy/1 Quality measures (e.g. sampling errors) and indicators (e.g. non- response rates) regularly produced and monitored for source data Definitions of data consistent with both user and provider understanding A revisions policy which balances the need to inform users of improved estimates with possible confusion from insignificant changes 6

Copyright 2010, The World Bank Group. All Rights Reserved. Output quality indicators – accuracy/2 Insignificant and consistent (size and direction) differences between preliminary and final estimates Data rebased regularly Data source samples redesigned or reselected regularly to maintain sample errors within quality standards 7

Copyright 2010, The World Bank Group. All Rights Reserved. Output quality indicators – coherence/1 All source data measured in accordance with standards (frame, statistical units, definitions, classifications, processes) Data presented in a framework, along with other relevant data Long term time series are available, with explanations or adjustments for breaks in the series Input data from different sources are confronted and reconciled Output data is consistent or reconcilable with other sources 8

Copyright 2010, The World Bank Group. All Rights Reserved. Output quality indicators – coherence/2 Concordances available to allow data to be related to previous classification versions or related classifications Consistency between aggregates and components Key classifications are maintained so that comparability over time is maintained 9

Copyright 2010, The World Bank Group. All Rights Reserved. Output quality indicators – interpretability/1 Seasonal and trend analysis and other adjustment techniques are undertaken to enhance the usefulness of the data and reduce problems of interpretation and comparisons Preliminary or early estimates are clearly indicated as such and information provided on expected level of revisions Major changes to the data from revisions, rebasing, etc. are published separately from, and where appropriate ahead of, the release of new information 10

Copyright 2010, The World Bank Group. All Rights Reserved. Output quality indicators – interpretability/2 All revisions are clearly marked and explained Presentation standards used for tables and graphs Information on methods, concepts, data sources, etc. Is up-to-date and readily available to users Information on quality achieved is readily available 11

Copyright 2010, The World Bank Group. All Rights Reserved. Output quality indicators - accessibility Main findings are made widely available (e.g. through press releases, website, public libraries) Information on data availability (published and unpublished) is readily known Catalogues/directories are available for all fields of statistics Information is available in formats and media required by users First release of key data is made available in all media at the same time Release dates are announced in advance Information is affordable New information and communication technology is being used to improve the presentation and accessibility of data 12

Copyright 2010, The World Bank Group. All Rights Reserved. Managing quality of processes The following is necessary: Quality standards specified and known to all process managers and staff Processes and rules documented and the information readily accessible Owners of the outcomes of each stage identified along with their responsibilities Outputs at each stage defined along with standards and tolerances Production and monitoring of indicators to ensure the standards are met and users can be informed of the quality achieved A system for registering process problems and managing action taken A system for change management 13

Copyright 2010, The World Bank Group. All Rights Reserved. Monitoring the quality of surveys/1 Examples of indicators to be monitored are: Sample and non-sample errors Response rates Proportion of proxy interviews Impact of births and deaths in business surveys Impact of rotation 14

Copyright 2010, The World Bank Group. All Rights Reserved. Monitoring the quality of surveys/2 Outliers and imputation Population change Respondent load 15

Copyright 2010, The World Bank Group. All Rights Reserved. Monitoring respondent management/1 Key respondent management issues are: Forms in source data collections tested with respondents Good, up-to-date understanding of respondent information sources Respondents able to provide requested information via preferred method Help/support system available Measures of load produced regularly 16

Copyright 2010, The World Bank Group. All Rights Reserved. Monitoring respondent management/2 Administrative data used wherever possible, for small firms tax data is normally a useful source. Sound security of information Confidentiality checking of releases 17

Copyright 2010, The World Bank Group. All Rights Reserved. Quality support systems Quality management is impossible without adequate support systems Key tools are: Expert service units for standards and techniques for reducing error (e.g. sampling methodologists, questionnaire designers, time series analysts) Project management methodology to manage collections Protocols with associated standards and guidelines for graphs, tables, time series presentation, revisions, releasing data with error, form design, etc Standard frames, frameworks, definitions, questions, classifications, code files Easy access to documentation Peer reviews of the design of various aspects of a collection Registers for logging problems and tracking their resolution 18

Copyright 2010, The World Bank Group. All Rights Reserved. Quality management and leadership High level management support and leadership is needed Important principles are: Output managers actively taking responsibility for the quality for their products Information on quality regularly collected and used Customer orientation Documentation is encouraged Good practices sought out and promoted Extension of solutions/innovations to other processes Regular reviews of performance of systems Staff development and support 19

Copyright 2010, The World Bank Group. All Rights Reserved. Managing staff quality/1 The key to success in quality management is staff with the right skills Important staff skills in this respect are: It starts with a good understanding of the topic they report about. That is subject matter knowledge. Good understanding of key users and emerging new stakeholders Good understanding of public policy directions and issues Ensuring that information is produced at the right frequency to allow timely monitoring of changes. Knowledge of the relevant processes and the methods they need to apply. 20

Copyright 2010, The World Bank Group. All Rights Reserved. Managing staff quality/2 Ensuring that information provides a useful estimate of what key users want to measure Ensuring that questionnaires, definitions and classifications reflect contemporary needs and situations 21