1. CZECH STATISTICAL OFFICE www.czso.cz 1 The Quality Metadata Subsystem In the Czech Statistical Office European Conference On Quality in Official Statistics 4-6 May 2010, Helsinki Czech Statistical Office Jitka Prokop

2. CZECH STATISTICAL OFFICE www.czso.cz 2 Content 1.Background 2.Quality Metadata System 3.Quality Monitoring 4.Quality Assessment 5.Lessons Learned 6.Conclusions

3. CZECH STATISTICAL OFFICE www.czso.cz 3 Architecture of metadata on qualityArchitecture of metadata on quality Links to other SMS subsystemsLinks to other SMS subsystems Quality MonitoringQuality Monitoring Evaluation of quality, self-assessment and auditing toolsEvaluation of quality, self-assessment and auditing tools Support for quality reporting (ESS and EFQM reports) or auditingSupport for quality reporting (ESS and EFQM reports) or auditing 1. Background SMS-QUALITY features and functions

4. CZECH STATISTICAL OFFICE www.czso.cz 4 1. Background Links between SMS subsystems SMS-CLASS SMS-VAR SMS-TASKS SMS-QUALITY SMS-USERS SMS-DISSEM SMS-RESP SMS-SERIES Statistical task - a set of statistical activities needed to fulfil a user’s request for statistical information. It can be composed of one or more statistical surveys or similar statistics.

5. CZECH STATISTICAL OFFICE www.czso.cz 5 The SMS subsystemsThe SMS subsystems Enable definition of metadata Provide values into SMS-QUALITY views Enable storage of values inserted via SMS-QUALITY 1. Background SMS-subsystems & SMS-QUALITY

6. CZECH STATISTICAL OFFICE www.czso.cz 6 2. Quality Metadata System SMS-QUALITY Architecture Based on ESS concept of qualityThe ESS concept of quality Quality criteria / components / dimensions Quality and performance indicators DESAPThe European Self-Assessment Checklist for Survey Managers (DESAP) Statistical Business Process ModelStatistical Business Process Model Identification of Q-attributes for sub-processes

7. CZECH STATISTICAL OFFICE www.czso.cz 7 2. Quality Metadata System Quality Form Map QFM is Defined structure of Q-attributesDefined structure of Q-attributes (metadata items) BaseBase of SMS-QUALITY different types of statisticsFor different types of statistics (in some extent). SW applicationSW application for QFM Defines Q-attributes and/or links into other SMS subsystems Enables inputs of values manually Provides views into DWH DWHValues of Q-attributes are stored in DWH

8. CZECH STATISTICAL OFFICE www.czso.cz 8 2. Quality Metadata System Structure of the QF Map

9. CZECH STATISTICAL OFFICE www.czso.cz 9 2. Quality Metadata system Levels of Q-attributes A.Most stable Q-attributes Relates to the whole statistical task, survey, phase of process e.g.: Key users, Information on training of staff, Methodology.. B.Q-attributes related to processing in a concrete period e.g.: Unit response rate, Extent of sample and frame, Punctuality C. Q-attributes related to a key statistical variable In generally defined breakdown e.g.: Coefficient of variation, Item response rate Levels of Q-attributes relate to content and stability of values Levels of Q-attributes relate to content and stability of values.

10. CZECH STATISTICAL OFFICE www.czso.cz 10 2. Quality Metadata System Example of QF Map components Group of SectionsSectionsQ-Attributes Design of statistical survey & Relevance Cooperation with users Testing of questionnaire Number of responses in tests of questionnaire – internal, ministries, respondents. Number of testing cycles. Sensitive questions in the questionnaire. Training of staff Scheduled and actual dates Punctuality Feedback form users & their requirements

11. CZECH STATISTICAL OFFICE www.czso.cz 11 3. Quality Monitoring Aspects coversQuality monitoring covers CollectionCollection of data / metadata CalculationsCalculations of Q-attributes ChecksChecks with expected values / scales (future; comparisons) Quality monitoring is based onQuality monitoring is based on Input variableshistory of changesInput variables, incl. „history of changes“ Variables specially designed for quality monitoringVariables specially designed for quality monitoring Related to questionnaire or interview Related to concrete variables ResultsResults of Q-attributes DuringDuring a sub-process / phase of statistical process At the endsub-processAt the end of the sub-process / phase After finishingprocessAfter finishing the whole statistical process

12. CZECH STATISTICAL OFFICE www.czso.cz 12 CollectionCollection of input data for calculation of Q-attributes Calculations of Q-attributesCalculations of Q-attributes (by SW) e.g. Unit response ratepopulationUnit response rate for the population Unit response ratesstrataUnit response rates for individual strata 3. Quality Monitoring During running of sub-process

13. CZECH STATISTICAL OFFICE www.czso.cz 13 3. Quality Monitoring After the end sub-process input variablesBased on input variables Editing and imputation rates Distinction of inputs, corrections and deletes Recognition of most edited items specific quality variablesBased on specific quality variables Way (mode) of data collection Type of contact with respondents Aspects of individual imputations (e.g. with or without respondent, technical mistakes)

14. CZECH STATISTICAL OFFICE www.czso.cz 14 3. Quality Monitoring After the end of the process or After the end of one run of the process Indicators on accuracy Coefficient of variation Item response rates Final unit response rates Imputation rates Revisions Timeliness, Punctuality, Comparability, Coherence

15. CZECH STATISTICAL OFFICE www.czso.cz 15 4. Quality Assessment The Quality Assessment Guidance (QAG) Purpose, applicability management of processSupport for management of process high level decisionsSupport for high level decisions quality reportingSupport for quality reporting Self-assessmentSelf-assessment AuditingAuditing Levels - Assessment of Quantitative and qualitative results of Q-attributes Statistical survey / statistical task

16. CZECH STATISTICAL OFFICE www.czso.cz 16 4. Quality Assessment Ways 4. Quality Assessment Ways of Assessment Categorial assessment Averages of individual results Textual assessment and summary Expert commentaries (suggested issues in QAG) Strong aspects Weaknesses Proposal of concrete actions, priorities

17. CZECH STATISTICAL OFFICE www.czso.cz 17 4. Quality Assessment Levels of Q Assessment Structure of QA follows the ESS quality criteria levels and covers the following levels: statistical variable a.Key statistical variable in particular breakdowns. aggregate b.Statistical variable as an aggregate (average) of the breakdowns or statistical survey. sub-process sub-criteriacriteria c.Set of similar indicators, quality of sub-process, quality sub-criteria or criteria. audited statisticswhole d.The audited statistics as the whole.

18. CZECH STATISTICAL OFFICE www.czso.cz 18 Suggestions -> Collection -> Calculations -> Assessment -> Feedback and actions 5.Lessons learned 1. Links to the phases of the SBP

19. CZECH STATISTICAL OFFICE www.czso.cz 19 5. Lessons Learned Cross-sectional aspects and Time coordination 2.Cross-sectional aspects differenttypes of statisticsEnsure applicability for different types of statistics duplicitieslinksAvoid duplicities, arrange links with other SMS subsystems expertsInvolve experts: SMS-Quality project team quality methodologists subject-matter statisticians ICT experts, members from other project teams 3.Time coordination mutually coordinated – Committee for the Redesign of SIS and SMSDesign and implementation of SMS-QUALITY and other SMS subsystems to be mutually coordinated – Committee for the Redesign of SIS and SMS

20. CZECH STATISTICAL OFFICE www.czso.cz 20 5. Lessons Learned 4. SMS-Quality project team Appointed by the top-management Suggests schedule of the SMS-QUALITY activities Regularly reports to the top-management Designed QF-map (Architecture of SMS-QUALITY) => Proposes SW application (content, functions) incl. updates Coordinates implementation and cooperation among activities of involved experts

21. CZECH STATISTICAL OFFICE www.czso.cz 21 5.Lessons Learned 5. Role of subject-matter statisticians and methodologists (a) Testing phase Quality methodologists Define Q-attributes and explanatory notes into SW application Test the SW applications Administrate SMS-Quality, QF Map, QAG Provide scripts for calculations of quality indicators Suggest scales for quality assessment Subject-matter statisticians Provide data for tests Adjust scales for quality assessment Manage routine quality monitoring, assessment

22. CZECH STATISTICAL OFFICE www.czso.cz 22 Full implementation Q-methodologists manage Quality methodology updates (according to the ESS development) Support subject-matter statisticians Administration of SMS-QUALITY, updates Subject-matter statisticians manage Collection of Q-attributes (automatic or manually) Quality assessment, self-assessment Approval of results of assessment 5.Lessons Learned 5. Role of subject-matter statisticians and methodologists (b)

23. CZECH STATISTICAL OFFICE www.czso.cz 23 5.Lessons Learned 6. Compliance with ESS quality framework SMS-QUALITY should be designed as a flexible tool to ensure easy methodology updates, taking into account development on the ESS level. Application software should have necessary flexibility concerning collection, monitoring and assessment procedures.

24. CZECH STATISTICAL OFFICE www.czso.cz 24 6. Conclusions Development of SMS-QUALITY has been scheduled with high priority for the next two years. Further progress in development of SMS-QUALITY depend On available human and financial resources due to fact the same experts are involved in real production of quality reports and in development of SMS-QUALITY. On progress of other SMS subsystems. Development of quality methodology on both national and international levels shall be taken into account.

25. CZECH STATISTICAL OFFICE www.czso.cz 25 Thank you for your attention. jitka.prokop@czso.cz