13 November, 2014 Seminar on Quality Reports QUALITY REPORTS EXPERIENCE OF STATISTICS LITHUANIA Nadiežda Alejeva Head, Price Statistics Division Statistics Lithuania

13 November, 2014 Seminar on Quality Reports CONTENT OF THE PRESENTATION  Basis for quality reports.  The role of quality reporting  Types of the quality reports  Experience of Statistics Lithuania  Quality Report structure 2

13 November, 2014 Seminar on Quality Reports BASIS FOR QUALITY REPORTS  Provisions of the Regulation (EC) No 223/2009 of the European Parliament and of the Council of 11 March 2009 on the European Statistics laid down in Article 12 regarding statistical quality:  To guarantee the quality of results, European statistics shall be developed, produced and disseminated on the basis of uniform standards and of harmonised methods. In this respect, the following quality criteria shall apply:  Relevance,  Accuracy,  Timeliness,  Punctuality,  Accessibility,  Comparability,  Coherence.  European Statistics Code of Practice adopted by the European Statistical System Committee on 28th September 2011  sets the standards for developing, producing and disseminating European statistics.  ESS Handbook for Quality Reports.  Technical Manual of the Single Integrated Metadata Structure (SIMS).  ESS Guidelines for the Implementation of the ESS Quality and Performance Indicators (QPI). 3

13 November, 2014 Seminar on Quality Reports THE ROLE OF QUALITY REPORTING  Quality is a multi-dimensional concept and include all aspects of how well statistics are fit for their purpose.  Quality is of primary importance producing official statistical information.  Producers of official statistics have to guarantee that statistics are developed, produced and disseminated on the basis of uniform standards and of harmonised methods.  Users of statistics are guaranteed assess of appropriate metadata describing the quality of statistical outputs, so that they are able to interpret and use the statistics correctly.  In order to meet the standards of the European Statistics Code of Practice, comprehensive quality reports for the full range of statistical processes and their outputs have to be prepared. 4

13 November, 2014 Seminar on Quality Reports THE ROLE OF QUALITY REPORTING  The quality report is one type of documentation for statistical processes and enable the process quality monitoring.  Quality report should include the information which enable to identify the statistical process and outputs quality problems as well as potential improvements.  Recommendations for quality improvement should be identified and later implemented based on the quality report.  It is recommended that quality reports should be updated annually. 5

13 November, 2014 Seminar on Quality Reports STATISTICAL PROCESSES  Six types of statistical processes:  Sample survey,  Census (for example, population, agricultural),  Using of administrative source(s),  Statistical process involving multiple data sources (for example, structural business survey),  Production of price or other economic index,  Statistical compilation (for example, national accounts, balance of payments). 6

13 November, 2014 Seminar on Quality Reports TYPES OF THE QUALITY REPORT  A quality report may be:  Producer-oriented  producer-oriented quality reports are prepared for the own internal assessment of process;  User-oriented  user-oriented quality reporting is in general less detailed and focuses on the output quality,  in general, user-oriented quality report is a sub-set of the detailed, producer- oriented quality report;  Both.  A quality report can range from very short and concise to very detailed;  A quality report can be prepared: for a specific statistical process (survey); for homogeneous group of processes; for an individual statistical indicator; it may cover only a few specific attributes and indicators.  The European Self Assessment Checklist for Survey Managers (DESAP) –covers all aspects of the statistical process and its outputs, but not in great detail.  Technical reports and the like where the statistical methodology is described in detail.  Quality reports may be prepared for every cycle of the statistical process with the periodicity that is in line with the type and specificity of the statistical process. 7

13 November, 2014 Seminar on Quality Reports TYPES OF THE QUALITY REPORT  In line with latest developments, several metadata structures have been used for the purpose of quality reporting.  The Euro-SDMX Metadata Structure (ESMS) was recommended as reference metadata report structure:  ESMS is considered as the concise, user-oriented format of quality reporting;  ESMS contains a basic level of quality information which is structured along the quality criteria as defined in the European Statistics Code of Practice and Regulation (EC) No 223/2009 – the information focuses more on the statistical output rather than the underlying process itself.  A more detailed quality reporting structure ESS Standard for Quality Reports Structure (ESQRS):  ESQRS is more addressed to the producers of statistics and focuses more on the statistical process side.  All statistical concepts of the 2 existing ESS report structures: ESMS and ESQRS have been included and streamlined in the Single Integrated Metadata Structure (SIMS). 8

13 November, 2014 Seminar on Quality Reports SINGLE INTEGRATED METADATA STRUCTURE (SIMS)  Single Integrated Metadata Structure (SIMS) has been created in order to:  streamline and harmonise metadata and quality reporting in the ESS,  decrease the reporting burden on the statistical authorities by creating the framework for “once for all purposes” reporting, where each concept is only reported upon once and is re-usable for other reporting,  create an integrated and consistent quality and metadata reporting framework where the reports are stored in the same database,  create a flexible and up-to-date system where future extensions are possible by adding new concepts. 9

13 November, 2014 Seminar on Quality Reports EUROPEAN STATISTICS CODE OF PRACTICE  Standard ESS quality reports are built around the fifteen principles of the European Statistics Code of Practice.  Quality is approached along 3 lines:  Quality of characteristics of the institutional environment (6 principles);  Quality of the statistical processes (4 principles);  Quality of the statistical output (5 principles). 10

13 November, 2014 Seminar on Quality Reports EUROPEAN STATISTICS CODE OF PRACTICE  INSTITUTIONAL ENVIRONMENT QUALITY CRITERIA:  Professional independence  Mandate for data collection  Adequacy of resources  Commitment to quality  Statistical confidentiality  Impartiality and objectivity  STATISTICAL PROCESSES QUALITY CRITERIA:  Sound methodology  Appropriate statistical procedures  Non-excessive burden on respondents  Cost effectiveness  STATISTICAL OUTPUT QUALITY CRITERIA:  Relevance  Accuracy and reliability  Timeliness and punctuality  Coherence and comparability  Accessibility and clarity 11

13 November, 2014 Seminar on Quality Reports EXPERIENCE OF STATISTICS LITHUANIA  COMMITMENT TO QUALITY  In 2007, a quality management system conforming to the requirements of the ISO 9001 standard was introduced in Statistics Lithuania and in 2013 it has been certified for the third time.  Introduced and certified quality management system provided a good framework for the implementation of the European Statistics Code of Practice:  management rules, structure, processes and responsibilities are clearly defined and documented,  performance results are planned and pursued purposefully,  quality assessment and improvement methods and tools (audits, self-assessment, quality indicators, user satisfaction surveys, etc.) are implemented in a clear and systematic way.  Quality management system enables the total quality control. 12

13 November, 2014 Seminar on Quality Reports EXPERIENCE OF STATISTICS LITHUANIA  The main trends in activity of Statistics Lithuania aimed at quality management and continuous development in the institution are established in the Quality Policy and Quality Objectives of Statistics Lithuania.  Quality policy reflects the key commitments of Statistics Lithuania, set in the Strategy of Statistics Lithuania for 2013–2017, comprising the Mission statement, Strategic Objective and main trends together with the improvement actions (measures) and expected results.  The guidelines, comprising the description of each stage of statistical production process, their documentation and quality monitoring are published in the process map in which all activities of Statistics Lithuania are detailed using process schemes and supporting documentation. 13

13 November, 2014 Seminar on Quality Reports EXPERIENCE OF STATISTICS LITHUANIA  The system for measuring and monitoring of Statistics Lithuania performance indicators has been introduced; every year a plan for monitoring of performance indicators (internal document) is elaborated.  The set of regularly monitored performance indicators covers ESS quality indicators in relation to the European Statistics Code of Practice and indicators related to different stages of statistical and other processes.  For all performance indicators, critical values are defined. A regular analysis of performance indicators is quarterly made at the survey and institutional levels.  From 2007, inspections of statistical surveys are carried out, the purpose whereof – objective estimation of the quality of preparation of statistics, exchange of experience, indication of good examples.  Starting from 2005 Statistics Lithuania performs several types of regular user satisfaction surveys, assessing user perception from different perspectives: surveys targeted on different user groups; a survey exploring the image of Statistics Lithuania; surveys focusing on the improvement of the services of Statistics Lithuania, etc. From 2007, it has been started to estimate the user satisfaction level. 14

13 November, 2014 Seminar on Quality Reports EXPERIENCE OF STATISTICS LITHUANIA  Periodic quality reporting (serving both producer and user perspectives) according to ESS recommendations for Euro SDMX Metadata Structure are prepared and sent to Eurostat.  Every year, assessment of statistical processes are performed by survey managers using a checklist for quality assessment of surveys (DESAP). Every statistical survey is assessed once per 5 years. Self-assessment exercise is carried out on the initiative of survey managers.  Methodologies of all statistical surveys are standardized, described and publicly available for all users.  Statistical questionnaires are regularly checked and tested in order to reduce measurement errors.  Statistics Lithuania annually prepares the Official statistics releases calendar, which covers all statistics prepared according to the Official Statistics Work Programme. 15

13 November, 2014 Seminar on Quality Reports EXPERIENCE OF STATISTICS LITHUANIA  User-oriented Quality reports are prepared by Statistics Lithuania and made available to the public.  Metadata (Quality report) with integrated quality indicators are documented according to the standardized scheme, consistent with the SIMS.  Metadata (Quality report) includes some background information on the statistical process and outputs that are the subject of the report.  An overview of all outputs associated with the process are given.  References and links are given to other documentation on the methodology of the process and quality of the outputs. 16

13 November, 2014 Seminar on Quality Reports QUALITY REPORT STRUCTURE  NAME OF THE STATISTICAL SURVEY (INDICATOR)  1 CONTACT  1.1 Contact organisation  1.2 Contact organisation unit  1.3 Contact person  1.4 Contact person, position, area of responsibility  1.5 Contact person's postal address  1.6 Contact person's address  1.7 Contact person's phone number  1.8 Contact person's fax number  2 METADATA UPDATE  2.1 Metadata last certified  2.2 Metadata last posted  2.3 Metadata last update (version, check) 17

13 November, 2014 Seminar on Quality Reports QUALITY REPORT STRUCTURE  3 STATISTICAL PRESENTATION  3.1 Description of statistical information (main characteristics, purpose)  3.2 Classification(s), classification system  3.3 Sector coverage  3.4 Definition(s)  3.5 Statistical unit  3.6 Statistical population  3.7 Geographical coverage  3.8 Time coverage  3.9 Base period  4 MEASUREMENT UNIT(S)  5 REFERENCE (REPORTING) PERIOD  6 INSTITUTIONAL MANDATE  6.1 Legal acts and other agreements  6.2 Statistical data exchange  7 CONFIDENTIALITY (LEGAL ACTS PROVIDING FOR STATISTICAL DATA CONFIDENTIALITY)  7.1 Confidentiality policy  7.2 Data confidentiality regulations 18

13 November, 2014 Seminar on Quality Reports QUALITY REPORT STRUCTURE  8 RELEASE POLICY  8.1 Release calendar  8.2 Link to the release calendar  8.3 Release procedure  9 FREQUENCY OF DISSEMINATION  10 DISSEMINATION FORMAT  10.1 Press releases  10.2 Publications  10.3 Databases  10.4 Access to micro-data  10.5 Other  11 METHODOLOGICAL DOCUMENTATION  12 QUALITY MANAGEMENT  12.1 Quality assurance  12.2 Quality assessment 19

13 November, 2014 Seminar on Quality Reports QUALITY REPORT STRUCTURE  13 RELEVANCE  13.1 User needs  13.2 User satisfaction  13.3 Completeness of statistical information  Degree of completeness of required information  14 ACCURACY AND RELIABILITY  14.1 Overall accuracy  14.2 Sampling error  14.3 Non-sampling error  Non-response error  Unit non-response rate  Item non-response rate  15 TIMELINESS AND PUNCTUALITY  15.1 Timeliness  15.2 Punctuality  Percentage of statistical information released on time  16 COMPARABILITY  16.1 Geographical comparability  16.2 Comparability over time  Length of comparable time series 20

13 November, 2014 Seminar on Quality Reports QUALITY REPORT STRUCTURE  17 COHERENCE  17.1 Cross domain coherence  17.2 Internal Coherence  18 ADMINISTRATIVE BURDEN FOR RESPONDENTS (TIME USED BY ONE RESPONDENT TO FILL IN A STATISTICAL QUESTIONNAIRE)  19 REVISION OF STATISTICAL INFORMATION  19.1 Revision policy  19.2 Revision practice  20 STATISTICAL DATA PROCESSING  20.1 Statistical data source  20.2 Periodicity of statistical data collection  20.3 Statistical data collection  20.4 Statistical data validation  20.5 Production of statistical information  20.6 Adjustment  21 COMMENTS, LINKS TO RELATED METADATA 21

13 November, 2014 Seminar on Quality Reports QUALITY INDICATORS  Indicators useful as user-oriented quality indicators and are included in the Quality report:  Data completeness – rate,  Sampling errors – indicators:  coefficient of variation  confidence interval,  Unit non-response – rate,  Item non-response rate,  Time lag – final results,  Punctuality – delivery and publication,  Length of comparable time series,  Data revision – average size.  Quality indicators are published in the Metadata (Quality report), documented according to SIMS and publicly available for all users. 22

13 November, 2014 Seminar on Quality Reports QUALITY REPORT  Survey managers of Statistics Lithuania have to prepare Metadata in line with the Single Integrated Metadata Structure (Quality report)  167 Metadata (Quality reports) have been prepared and are available to users.  Annually, till 1 February, Metadata (Quality reports) have to be revised and updated when methods and sources have changed.  Metadata (Quality reports) including quality indicators is updated according to the approved procedures regulating the preparation, updating and dissemination of Metadata (Quality reports).  Information on quality indicators is updated during 3 days after new statistical information are published. The time of the last update is indicated in the text of the Metadata (Quality reports). 23

13 November, 2014 Seminar on Quality Reports RESPONSIBILITY  Methodology and Quality Division is responsible for setting general requirements for the statistical production, coordination of the introduction of common methodologies and monitoring their implementation.  Strategic Planning and Official Statistics Programming Division is responsible for the organization of definition of performance criteria of Statistics Lithuania and monitoring their implementation.  Information Systems Maintenance Division provides technical support for implementing process quality monitoring. It is responsible for the maintenance of the system for recording quality characteristics of statistical surveys introduced in Statistics Lithuania. 24

13 November, 2014 Seminar on Quality Reports THANK YOU FOR YOUR ATTENTION QUESTIONS? 25