Metadata on quality of statistical information Petr Elias Jitka Prokop Czech Statistical Office

Issues Quality in statistics generally. Quality regarding statistical data and metadata. Producers and users – roles, needs. What should metadata on quality cover? How metadata should be structured? For which purposes can metadata be used? Which tools? Issues

OUTLINE Statistical Quality Focus, Coverage Methods, Tools References

Purposes Producers Users need to => Quality in broad sense Assure quality of statistical data Provide data to users Provide meta-information to users Users need to Know what they need Gain good relevant data (do not care about process) Use data properly => Quality in broad sense

Focus Outputs, Products: Processes, sub-processes, phases (GSBPM): data and metadata, news releases publications, web publishing, ad-hoc products web-databases Processes, sub-processes, phases (GSBPM): needs specification, design, build, data collection, processing, analyses, dissemination, evaluation Input data: Micro-data from survey, registers, secondary data

Quality Statistical Quality Quality Management Statistical outputs (Quality criteria and QPI) Statistical processes (GSBPM; QPI mapping?) Input data, (micro)data Quality Management TQM, EFQM, etc.

Quality Criteria (Principles) Code of Practice: Principles 11 – 15 EU Regulation 223/2009: Quality Criteria Principle 11: Relevance Principle 12: Accuracy and Reliability Principle 13: Timeliness and Punctuality Principle 14: Coherence and Comparability Principle 15: Accessibility and Clarity

Quality Criteria (Principles) Relevance EU Statistics must meet the needs of users. Accuracy and Reliability EU Statistics must accurately and reliably portray reality. Timeliness and Punctuality EU Statistics must be disseminated in a timely and punctual manner. Coherence and Comparability EU Statistics should be consistent internally, over time and comparable between regions and countries; it should be possible to combine and make joint use of related data from different sources. Accessibility and Clarity EU Statistics should be presented in a clear and understandable form, disseminated in a suitable and convenient manner, available and accessible on an impartial basis with supporting metadata and guidance.

Coverage Business statistics Social and demography statistics Price statistics National accounts Administrative data statistics

Methods, tools Monitoring Q Q Reports (internal, external) Self-assessments of a survey Auditing (internal, external), CoP peer reviews Quality reports for producers Quality reports for users Single Integrated Metadata Structure (SIMS) Databases and SW applications

Self-assessments Natural character: Subjective Self-reflection, supportive for audits Form of checklists (eg. DESAP), e-form Can be for public (partly) Includes Quantitative and qualitative info Categorical and textual evaluations Summaries, strengths and weaknesses Suggested actions

Quality reporting Quality and Performance Indicators Applicability: national level, EU level Process and output quality To evaluate quality criteria (dimensions) EU Standard Q Reports & Euro SDMX Structure Single Integrated Metadata Structure For producers For users

Single Integrated Metadata Structure Dynamic inventory of ESS quality and reference metadata statistical concepts Quality reporting 2in1 - for producers, for users Includes Quality and Performance Indicators QPI ESS reference metadata report structure (Euro SDMX Metadata Structure ESMS) Optional tool for production, export of metadata: National Reference Metadata Editor (NRME)

SIMS Dimensions By scope By level By audience Particular survey or a group of surveys A statistical variable (e.g. IPI index) By level National ESS By audience Users Producers

Metadata: stability of values Levels General, group of surveys Survey Reference year A processing (reffers eg. to one news release) Statistical variable Reference period

Types of metadata in Q application Reference metadata Info about process and its phases Schedules, timing Quality performance indicators Calculations Benchmark results Evaluation, (self-)assessments, commentaries Textual, Numerical, Date

Possible categories of Q metadata (content) Basic information (about a survey). User requirements agenda. Methodology info. Time schedules. Timeliness, punctuality. Statistical process phases. Data confidentiality and protection. Data sources; Frame; Sample. Outputs and dissemination. Individual quality criteria (i.e. quality dimensions). Quality performance indicators.

Q metadata application issues Content, structure, links to databases and other apps Metadata updates (items or their parameters, hierarchical structure) Retrieving and updating data (new reference periods...) – i.e. Keep history & update Validity within certain years, batches, reference periods Alerts from the app; decision making in the application. Multi-lingual features. Benchmarking.

ESS quality reporting (references) Technical Manual of the Single Integrated Metadata Structure (SIMS) ESS Handbook for Quality Reports 2014 ESS Guidelines for the Implementation of the ESS Quality and Performance Indicators (QPI) Generic Statistical Business Process Model. Version 5.0. European Statistics Code of Practice Quality Assurance Framework The European Self Assessment Checklist for Survey Managers (DESAP)

Any questions?