Are the Standard Documentations really Quality Reports? European Conference on Quality in Official Statistics Helsinki, 3-6 May 2010 © STATISTIK AUSTRIA Thomas Burg

2 Outline  What are Standard Documentations?  Chapter 6: Quality  Contents: Different product types => Focus on Accuracy  Comparison: EHQR – Standard Documentations  Outlook – Next Steps

3 Standard Documentation (I) 1. Summary, important hints 2. General information 3. Statistical Concepts, Methods 4. Producing of Statistics, processing of data, Quality assurance 5. Publication (accessibility) 6. Quality Standard Documentation is a fixed structure of titles

4 Chapter 6 Quality (I) General Information Statistical Concepts, Methods Producing of Statistics, processing of data, Quality assurance Publication (Accessibility) Relevance Accuracy Timeliness and Punctuality Comparability Coherence DocumentationQuality Summary, Important Hints

5 Chapter 6: Quality (II) Substructure according to Definition of Quality RelevanceAccuracyTimeliness Punctuality ComparabilityCoherence Purpose of the product Sampling effects Nearness to occurrence Comparability over Time Are there external references? User integration Non- Sampling effects Project planGeographical Comparability Are there differences Legal Relevance Comparability across other domains Are the differences explainable

6 Chapter 6 Quality (III) Types of Statistics Primary Statistics (Surveys ) Statistics based on Administrative Sources Accounting Model Based Statistics Registers Indices

7 Chapter 6 Quality (IV) Extent of diversity caused by type of product RelevanceLow - medium AccuracyHigh Timeliness/PunctualityLow ComparabilityMedium CoherenceMedium Accessibility and ClarityMedium

8 Chapter 6 Quality (V) Relevance User integration (Analytic) Purposes Legal relevance Integration into a system of statistical products Closing a gap in the “statistical landscape” Relevant for national accounts

9 Chapter 6 Quality (VI) Accuracy (I)

10 Chapter 6 Quality (VII) Accuracy (II) Sampling Effects => well established methods ! Non-Sampling: Non Response and Coverage Surveys Administrative Sources Aspects are sometimes transferred to data holders Differences between administrative and statistical use Coverage Matching process

11 Chapter 6 Quality (VIII) Accuracy (III) “Two product in one” situation :Subject and weights Model Based effects Indices Accounting Puzzle like situation Model based effects Sensitivity analysis (resources)

12 Chapter 6 Quality (IX) Accuracy (IV) Not very often stand alone Model Based effects Model Based Statistics Registers New as a separate type of statistics Prototype situation Completeness, Coverage Implementation of NACE 2008

13 Chapter 6 Quality (X) Timeliness and Punctuality Different type of statistics is not the decisive factor for the contents: Legal aspects Periodicity Concerned Subject General demand on „information velocity“

14 Chapter 6 Quality (XI) Comparability 3 subtopics: Over time: Indices and Accounting products Periodicity is important Geographical: EU-comparability is most important Some regional aspects (sampling) Other domains: special cases for instance economic branches

15 Chapter 6 Quality (XII) Coherence Nearly every product belongs to a “system” of statistics No information on coherence: no possible choice Three questions: Are there other sources? Are there significant differences? if Yes are they explainable? Economic statistics a little nearer to National Accounts

16 Chapter 6 Quality (XIII) Publication – Accessibility and Clarity Broadness of Publication very similar to all products Revisions are a topic rather in economic statistics => Quantitative effects are addressed as well Confidentiality more interesting for surveys

17 EHQR – Standard Documentation (I) 1. Introduction 2. Relevance 3. Accuracy 4. Timeliness and Punctuality 5. Accessibility and Clarity 6. Coherence and Comparability 7. Trade-offs between Output Quality Components 8. Assessment of User Needs and Perceptions 9. Performance, Cost and Respondent Burden 10. Confidentiality, Transparency and Security 11. Conclusion ESS Handbook on Quality Reports Austrian Standard Documentation 1. Summary, Important Hints 2. General Information 3. Statistical Concepts, Methods 4. Producing of Statistics, processing of data, Quality assurance 5. Publication (accessibility) 6.1 Relevance 6.2 Accuracy 6.3 Timeliness and Punctuality 6.4 Comparability 6.5 Coherence

18 EHQR – Standard Documentation (II) General Overview EHQRStandard Documentation 1. Introduction  Spread across the various chapters of the Standard documentations 2. Relevance  Chapter 6.1 Relevance  Some issues are handled in Chapter 3 ‘Concepts’.  Quantitative Quality Indicators are spread across chapters 3, 4 and 6 of the Standard Documentations. 3. Accuracy  No general statement of Overall Accuracy.  Other aspects as described in this paper covered in a very similar way in Chapter 6.2.  Seasonal Adjustment is not under a separate headline but Model Based effects deal with it.  Revisions found under Chapter 5 (Publication).  Information about Processing and Measurement found in chapters 3 and 4 of Standard Documentation 4. Timeliness and Punctuality  Chapter 6.3 of Standard Documentation 5. Accessibility and Clarity  Chapter 5 Publication  No User feedback integrated  Some information in 6.1 Relevance 6.Coherence and Comparability  6.4 Comparability  6.5 Coherence  Some aspects of ‘internal coherence’ are covered in chapter 5 7. Tradeoffs between Output Quality components  Not directly addressed; sometimes implicitly when talking about preliminary results in chapter Assessment of User Need and Perception  6.1 Relevance  Chapter 2 General Information 9. Performance, Cost and Respondent Burden.  Quality Indicators are spread across documentation  No Information about Cost  Measures to reduce response burden: Chapters 3,4, and Confidentiality, Transparency and Security  Confidentiality and Security: Chapter 5 Publication  Transparency of legal basis: Chapter Conclusion  No concluding remarks,  Some statements in Chapter 1

19 EHQR – Standard Documentation (III) Deficiencies No Statement on overall Accuracy Tradeoffs are not explicitly addressed Quality Indicators are not summarized Concluding remarks are missing

20 Conclusions and Outlook Conclusions: Different product types => different contents Accuracy lies in the center of product quality Research is needed! A huge amount of EHQR information is covered and EHQR is a good benchmark Outlook: Revising the structure of the Standard Documentation Chapter on future outlooks Highlighting the quality chapter Executive summary Building in a table of key aspects

Thank you for your attention !