Introduction to Quality Concepts Measurement of the quality of statistics 3-5 October 2012   Marina Signore Istat Division "Metadata, Quality and R&D Projects", Chief

Topics Definition of quality Quality management

The definition of quality The concept of quality multi-faceted concept likely to change over time The definition of quality ESS adopted a common definition Quality is defined by Eurostat with reference to six criteria

Definition of quality for the ESS (ISO 8402, 1986) The totality of features and characteristics of a product or service that bear on its ability to satisfy a given need

Eurostat Quality dimensions Relevance Accuracy Timeliness and Punctuality Accessibility and Clarity Comparability Coherence Eurostat (2003), Definition of Quality in Statistics. Eurostat Working Group on Assessment of Quality in Statistics, Luxembourg, October 2-3.

Quality dimensions Relevance overall statistical activity Relevance is the degree to which statistics meet current and potential users’ needs. It refers to whether all statistics that are needed are produced and the extent to which concepts used (definitions, classifications etc.) reflects user needs overall statistical activity users involvement and feedback from planning to dissemination

Quality dimensions Accuracy old meaning of quality Accuracy in the general statistical sense denotes the closeness of computations or estimates to the exact or true values old meaning of quality effects of both sampling and non-sampling errors several sub-components

Quality dimensions Timeliness and Punctuality increasing importance Timeliness of information reflects the length of time between its availability and the event or phenomenon it describes Punctuality refers to the time lag between the release date of data and the target date when it should have been delivered, for instance, with reference to dates announced in some official release calendar, laid down by Regulations or previously agreed among partners increasing importance trade-off with accuracy release calendar

Quality dimensions Accessibility and Clarity users needs Accessibility refers to the physical conditions in which users can obtain data: where to go, how to order, delivery time, clear pricing policy, convenient marketing conditions (copyright, etc.), availability of micro or macro data, various formats (paper, files, CD-ROM, Internet…), etc. Clarity refers to the data’s information environment whether data are accompanied with appropriate metadata, illustrations such as graphs and maps, whether information on their quality also available (including limitation in use…) and the extent to which additional assistance is provided by the NSI. users needs metadata and quality

Quality dimensions Comparability Comparability aims at measuring the impact of differences in applied statistical concepts and measurement tools/procedures when statistics are compared between geographical areas, nongeographical domains, or over time. We can say it is the extent to which differences between statistics are attributed to differences between the true values of the statistical characteristic Comparability in the National statistical system Comparability in the ESS

Quality dimensions Coherence Comparison between different sources Coherence of statistics is their adequacy to be reliably combined in different ways and for various uses. It is, however, generally easier to show cases of incoherence than to prove coherence Comparison between different sources

Eurostat Quality definition Disadvantages The dimensions are in conflict with each other Not an easy task to assess each dimension No successful attempt to derive an overall index Advantages Harmonised framework at European level for quality measuring and reporting Easier for the users to judge and compare the quality of statistics Can be used as a framework for the assessment of user perception of a statistical product

Quality management systems The commitment for quality is increasing in most NSIs The quality definition sets the requirements that statistical outputs have to meet in order to satisfy users needs Need to have in place a set of coordinated and integrated tools and procedures for assessing and reporting quality but also to improve it Quality management systems To deliver “quality” products and services to users To assess and systematically improve the quality of products, processes and services to users

Institutional environment Elements of a quality management system User needs Management systems & leadership Statistical products Support processes Statistical processes Institutional environment Handbook on Data Quality Assessment Methods and Tools (2007) http://epp.eurostat.ec.europa.eu/portal/page/portal/quality/documents/HANDBOOK%20ON%20DATA%20QUALITY%20ASSESSMENT%20METHODS%20AND%20TOOLS%20%20I.pdf

Product Quality Refers to the quality of statistical output (or products) It is usually expressed as a set of components which have some attributes or requirements that are to be met (e.g. Eurostat quality vector with 6 dimensions) Essential for any assessment of data quality Reporting to users can be organised following quality dimensions

Process Quality Refers to the quality of the operations and instruments used in the data production process The way by which statistical information is produced affects the output quality To monitor and improve the statistical production process is a way to improve data quality Focus on the survey operations and process quality indicators

Process Quality requirements No standard criteria have been defined at ESS level Efficiency - produces the desired outcomes cost efficiently Effectiveness - successful in delivering the desired outcomes Robustness - delivers results against challenging demands Flexibility -readily adaptable to changing needs and demands Transparency - open, visible and easily understood Integration - complementary and consistent, both with other processes, and with meeting business needs Haworth M., Bergdall M., Booleman M., Jones T., Madaleno M. (2001), “LEG chapter: Quality Framework for the European Statistical System”, Proceedings of the International Conference on Quality in Official Statistics, Stockholm, 14-15 May 2001, CD-ROM

How to improve process quality Standardisation of survey operations and tools Use of best practices To decrease the unnecessary variation To adopt practices that have proved to be successful in improving quality or reducing costs Process monitoring - identification, measurement and analysis of key process variables - tools from statistical quality control

Process monitoring Key process variables Examples Those factors that can vary with each repetition of the process and have the largest effect on critical product characteristics, i.e. those characteristics that best indicate the quality of the products* Examples non-response of different types interviewer performance costs and use of time for different processes *Jones and Lewis (2003), “Handbook on improving quality by analysis of process variables”. Eurostat

Continuous Quality improvement the PDCA cycle introduced by Deming plan act do check

Some remarks Quality management is a long term-commitment Support from top management is crucial Design an overall strategy for quality Implementation can be done step by step High initial costs and benefits are not immediate therefore quality work should be seen as an investment May obstacles to face, culture changes might be necessary therefore training staff is important

References Bergdahl M., Japec L., Jones T., Signore M. (2001), " LEG chapter: Tools for Standardising and Improving the Quality of Statistics Production Processes and Other Operations”, Proceedings of the International Conference on Quality in Official Statistics, Stockholm, 14-15 May 2001, CD-ROM Biemer P.P. and Lyberg L. (2003) “Introduction to Survey Quality”, Wiley Blanc M., Lundholm G. and Signore M. (2001), “LEG chapter: Documentation”, Proceedings of the International Conference on Quality in Official Statistics, Stockholm, 14-15 May 2001, CD-ROM Blanc M., Radermacher W. and Koerner T. (2001), “LEG chapter: Quality and Users”, Proceedings of the International Conference on Quality in Official Statistics, Stockholm, 14-15 May 2001, CD-ROM Booleman M., van Brakel R., Jones T. and Tzougas I. (2001), “LEG chapter: Assessment tools”, Proceedings of the International Conference on Quality in Official Statistics, Stockholm, 14-15 May 2001, CD-ROM Eurostat (2003), Definition of Quality in Statistics. Eurostat Working Group on Assessment of Quality in Statistics, Luxembourg, October 2-3. Haworth M., Bergdall M., Booleman M., Jones T., Madaleno M. (2001), “LEG chapter: Quality Framework for the European Statistical System”, Proceedings of the International Conference on Quality in Official Statistics, Stockholm, 14-15 May 2001, CD-ROM Jones and Lewis (2003), “Handbook on improving quality by analysis of process variables”. Eurostat 22 22

References Eurostat (2003), Definition of Quality in Statistics. Eurostat Working Group on Assessment of Quality in Statistics, Luxembourg, October 2-3. Handbook on Data Quality Assessment Methods and Tools (2007), http://epp.eurostat.ec.europa.eu/portal/page/portal/quality/documents/HANDBOOK%20ON%20DATA%20QUALITY%20ASSESSMENT%20METHODS%20AND%20TOOLS%20%20I.pdf Haworth M., Bergdall M., Booleman M., Jones T., Madaleno M. (2001), “LEG chapter: Quality Framework for the European Statistical System”, Proceedings of the International Conference on Quality in Official Statistics, Stockholm, 14-15 May 2001, CD-ROM Lyberg L. et alt. (2001), “Summary Report from the Leadership Group (LEG) on Quality”, Proceedings of the International Conference on Quality in Official Statistics, Stockholm, 14-15 May 2001, CD-ROM Madaleno M., Koerner T., Radermarcher W., Booleman M., van Brakel R. (2001), “LEG chapter: Implemetation of Quality Management in National Statistical Institutes, Proceedings of the International Conference on Quality in Official Statistics, Stockholm, 14-15 May 2001, CD-ROM M.J. Zilhão, M. Madaleno, (2001), "Quality Management in Statistics and Performance Indicators“, Proceedings of the International Conference on Quality in Official Statistics, Stockholm, 14-15 May 2001, CD-ROM WESTAT, (1992) Statistical thinking. Rockville, Maryland 23 23