CZECH STATISTICAL OFFICE 1 The Quality Metadata System In the Czech Statistical Office Work Session on Statistical Metadata (METIS) March 2010, Geneva Czech Statistical Office Jitka Prokop

CZECH STATISTICAL OFFICE 2 Content 1.Background 2.Quality Metadata System 3.Quality Monitoring 4.Quality Assessment 5.Lessons Learned 6.Conclusions

CZECH STATISTICAL OFFICE Background SMS sub-systems

CZECH STATISTICAL OFFICE Background Links between SMS subsystems SMS-CLASS SMS-VAR SMS-TASKS SMS-QUALITY SMS-USERS SMS-DISSEM SMS-RESP SMS-SERIES Statistical task - a set of statistical activities needed to fulfil a user’s request for statistical information. It can be composed of one or more statistical surveys or similar statistics.

CZECH STATISTICAL OFFICE Quality Metadata System SMS-QUALITY Architecture Based on The ESS concept Quality criteria (e.g. relevance, accuracy, etc.) Quality and performance indicators (respondent burden, cost) The European Self-Assessment Checklist for Survey Managers (DESAP) Statistical Business Process Model Applicable for different types of statistics (in some extent).

CZECH STATISTICAL OFFICE Quality Metadata System Quality Form Map (QFM) Q-attributesModel of Q-attributes in defined structure. Approved by the top management. Includes all proposed Q-attributes. Different types of statistics to be covered. SW application for QFM Defines Q-attributes and/or links into other SMS subsystems Enables inputs of values manually Provides views into DWH Values of Q-attributes are stored in DWH

CZECH STATISTICAL OFFICE Quality Metadata System Structure of the QF Map

CZECH STATISTICAL OFFICE Quality Metadata System Example of QF Map components Group of SectionsSectionsQ-Attributes Design of statistical survey & Relevance Cooperation with users Testing of questionnaire Number of responses in tests of questionnaire – internal, ministries, respondents. Number of testing cycles. Sensitive questions in the questionnaire. Training of staff Scheduled and actual dates Punctuality Feedback form users & their requirements

CZECH STATISTICAL OFFICE Quality Metadata system Levels of Q-attributes 1.Most stable Q-attributes Relates to the whole statistical task, survey, phase of process e.g.: Key users, Information on training of staff, Methodology.. 2.Q-attributes related to processing in a concrete period e.g.: Unit response rate, Extent of sample and frame, Punctuality 3.Key statistical variable In generally defined breakdown e.g.: Coefficient of variation, Item response rate Levels of Q-attributes relate to content and stability of values.

CZECH STATISTICAL OFFICE Quality Monitoring Aspects coversQuality monitoring covers CollectionCollection of data / metadata CalculationsCalculations of Q-attributes ChecksChecks with expected values / scales (future; comparisons) based onQuality monitoring is based on Input variableshistory of changesInput variables, incl. „history of changes“ Variables specially designed for quality monitoringVariables specially designed for quality monitoring Related to questionnaire or interview Related to concrete variables ResultsResults of Q-attributes During a sub-process / phase of statistical process At the end of the sub-process / phase After finishing the whole statistical process

CZECH STATISTICAL OFFICE 11 CollectionCollection of input data for calculation of Q-attributes Calculated Q-attributesCalculated Q-attributes (by SW) e.g. Unit response ratepopulationUnit response rate for the population Unit response ratesstrataUnit response rates for individual strata 3. Quality Monitoring During running of sub-process

CZECH STATISTICAL OFFICE Quality Monitoring After the end sub-process input variablesBased on input variables Editing and imputation rates Distinction of inputs, corrections and deletes Recognition of most edited items specific quality variablesBased on specific quality variables Way (mode) of data collection Type of contact with respondents Aspects of individual imputations (e.g. with or without respondent, technical mistakes)

CZECH STATISTICAL OFFICE Quality Monitoring After the end of the process or After the end of one run of the process Indicators on accuracy Coefficient of variation Item response rates Final unit response rates Imputation rates Revisions Timeliness, Punctuality, Comparability, Coherence

CZECH STATISTICAL OFFICE Quality Assessment The Quality Assessment Guidance (QAG) Purpose, applicability Support for management of process Support for high level decisions Support for quality reporting Self-assessment Auditing Levels - Assessment of Quantitative and qualitative results of Q-attributes Statistical survey / statistical task

CZECH STATISTICAL OFFICE Quality Assessment Ways 4. Quality Assessment Ways of Assessment Categorial assessment Averages of individual results Textual assessment and summary Expert commentaries (suggested issues in QAG) Strong aspects Weaknesses Proposal of concrete actions, priorities

CZECH STATISTICAL OFFICE Quality Assessment Levels of Q Assessment of Q criteria Structure of QA follows the ESS quality criteria and c overs the following levels: statistical variable a.Key statistical variable in particular breakdowns. aggregate b.Statistical variable as an aggregate (average) of the breakdowns or statistical survey. c.Set c.Set of similar indicators, quality of sub-process, quality sub-criterion or criterion. audited statistics d.The audited statistics as the whole.

CZECH STATISTICAL OFFICE 17 Suggestions -> Collection -> Calculations -> Assessment -> Feedback and actions 5.Lessons learned 1. Links to the phases of the SBP

CZECH STATISTICAL OFFICE Lessons Learned Cross-sectional aspects and Time coordination 2.Cross-sectional aspects differenttypes of statisticsEnsure applicability for different types of statistics duplicitieslinksAvoid duplicities, arrange links with other SMS subsystems expertsInvolve experts: SMS-Quality project team quality methodologists subject-matter statisticians ICT experts, members from other project teams 3.Time coordination mutually coordinated – Committee for the Redesign of SIS and SMSDesign and implementation of SMS-QUALITY and other SMS subsystems to be mutually coordinated – Committee for the Redesign of SIS and SMS

CZECH STATISTICAL OFFICE Lessons Learned 4. SMS-Quality project team Appointed by the top-management Suggests schedule of the SMS-QUALITY activities Regularly reports to the top-management Designed QF-map (Architecture of SMS-QUALITY) => Proposes SW application (content, functions) incl. updates Coordinates implementation and cooperation among activities of involved experts

CZECH STATISTICAL OFFICE Lessons Learned 5. Role of subject-matter statisticians and methodologists (a) Testing phase Quality methodologists Define Q-attributes and explanatory notes into SW application Test the SW applications Administrate SMS-Quality, QF Map, QAG Provide scripts for calculations of quality indicators Suggest scales for quality assessment Subject-matter statisticians Provide data for tests Adjust scales for quality assessment Manage routine quality monitoring, assessment

CZECH STATISTICAL OFFICE 21 Full implementation Q-methodologists manage Quality methodology updates (according to the ESS development) Support subject-matter statisticians Administration of SMS-QUALITY, updates Subject-matter statisticians manage Collection of Q-attributes (automatic or manually) Quality assessment, self-assessment Approval of results of assessment 5.Lessons Learned 5. Role of subject-matter statisticians and methodologists (b)

CZECH STATISTICAL OFFICE Lessons Learned 6. Compliance with ESS quality framework SMS-QUALITY should be designed as a flexible tool to ensure easy methodology updates, taking into account development on the ESS level. Application software should have necessary flexibility concerning collection, monitoring and assessment procedures.

CZECH STATISTICAL OFFICE Conclusions Development of SMS-QUALITY has been scheduled with highest priority for the next two years. Further progress in development of SMS-QUALITY will depend On available human and financial resources due to fact the same experts are involved in real production of quality reports and in development of SMS-QUALITY. On progress of other SMS subsystems. Development of quality methodology on both national and international levels shall be taken into account.

CZECH STATISTICAL OFFICE 24 Thank you for your attention.