1. Supervisory and Control Systems for National Accounts Purposes
Wir bewegen
Quality Reports
at Statistics Austria
I n f o r m a t i o n e n
Eurostat Workshop
Supervisory and Control Systems for National Accounts Purposes
Ursula Havel
December 2009
© STATISTIK AUSTRIA
December 2009

2. history objectives Quality reports motivation contents Feedback talks
organisation
December 2009

3. History of Quality at statistics Austria
First considerations in the end of the nineties
Change of legal status in TQM was established
- Several activities started
- Project Controlling
- Considerations about Quality Reporting
- Database QRD
- Staff Opinion Survey
development of detailed quality reports
beginning of “Feedback talks”

4. Standard documentations
Development of a template for standardised
documentations for all statistical products and for
reporting on quality according to the ESS quality dimensions
Start with Standard documentations
Objective: Standard documentations for all relevant products by end of 2007
Start with Feedback talks based on Standard documentations: Statistics Austria in co-operation with the Quality Assurance Committee of the Statistical Council Objective: All statistics reviewed by end of („first round“)

5. Why a system on standard documentations (1)
European
demands
Statistical Council
Statistics Austria as main provider of data
Increasing demand on metadata
Code of Practice

7. Why a system on standard documentations (2)
Documentation for every statistical product
Learning more about our own products
Information policy for users
Looking for possible improvements

8. System of Standard – Documentations Considerations (1)
Standard documentation want to follow two purposes:
The quick reader - somebody who is not very familiar with
the product - should come into the position to have an
overview and some basic understanding of what is going on
Power users with high level of expert knowledge should be provided with detailed information

9. System of Standard – Documentations Considerations (2)
Internal aspects:
Double work should be avoided – standard
documentation serves as quality report
Possibility for auditing products -> Feedback talks
Fixed structure with an eye to a possible database
solution

10. System of Standard - Documentations
1. Summary, important hints
2. General information
3. Statistical concepts, methods
4. Producing of statistics, processing of data,
Quality assurance
5. Publication (accessibility)
6. Quality

11. 1. Summary, Important Hints (1)
The idea behind that is to give a „quick reader” a short informative overview of the product. To achieve this, the summary itself follows a structure similar to the whole report.
A second demand to this introduction is that it should contain all problematic aspects, which are important for the understanding and usability of the data.

12. 1. Summary, Important Hints (2)
Product
Summary
What can be done with the figures – and what not?
where are pitfalls for a user

13. Describing the Product
2. General information
Describing the Product
Type
Organisation
Area
Aim, Purpose
User
Statistics
Legal Basis
History
Orderer
Periodicity

14. 3. Statistical Concepts, Methods
Subject of the statistics
What is it about?
Used units
Survey unit
From where comes it?
Observation unit
Respondents/Data Sources
Survey technique/data transfer
How comes it?
Sample
Participation/Questionnaire
Items/Indicators
How to characterize it?
Classifications
Regional breakdown

15. Data Capture Editing Grossing up Estimation, Models Coding Imputation
4. Producing of Statistics, processing of data, quality assurance
Data Capture
Editing
Grossing up
Estimation,
Models
Coding
Imputation
Data input
Datacleaning
Data processing

16. 5. Publication Time of release Where is it published
Preliminary results
Time of release
Final Results
Where is it published
Published in
Revisions
Usability, clarity
Confidentiality

17. 6. Quality Documentation Quality Relevance General Information
Accuracy
Statistical Concepts, Methods
Timeliness and punctuality
Producing of Statistics,
processing of data,
Quality assurance
Comparability
Publication (Accessibility)
Coherence

18. Standard documentations - Publication
All Standard documentations are to be published on the Internet
Standard documentations since Coverage: currently approx. 87 %

19. Feedback talks on quality
Standard documentations (quality reports)
provide an essential basis for
Feedback talks on quality

20. Content and objectives of the feedback talks (1)
an important approach for the assessment and evaluation of the quality of Statistics Austria's statistical work
Involvement of
Statistical Council - Quality Assurance Committee
external experts/users
Internal experts

21. Content and objectives of the feedback talks (2)
represent a statistical audit function as they aim at:
reviewing statistical methods and the quality of process
identifying quality improvement potentials on the basis of external and internal users‘/experts’ views
improving the standard documentations (quality reports)
defining recommendations and action plans
documentation and follow up of improvement measures

22. Organisation of feedback talks (1)
Feedback talks: monthly (except July, August)
Organised by Statistics Austria in close cooperation with the Quality Assurance Committee of the Statistical Council
Moderator: the Chairperson of the Quality Assurance Committee of the Statistical Council

23. Organisation of feedback talks (2)
5-year plan, in which all statistical products of Statistics Austria are scheduled for a feedback talk
Detailed planning (quarterly): selection of one product – or a set of connected products - for monthly feedback talks
Agreement of Statistics Austria and the Quality Assurance Committee of the Statistical Council on which key users/ external experts should be invited to the feedback talks
Agreement on a date for forwarding of the draft standard documentation (quality report) for internal discussion

24. Organisation of feedback talks (3)
The draft report is evaluated by the quality management and the methods unit, internal discussion of the product and processes with the experts responsible for the product
Revision of the documentation according to the agreements with the experts from the unit responsible for the product
2 weeks before the feedback talk, at the latest, a final version of the standard documentation (quality report) is forwarded to all participants

25. Organisation of feedback talks (4)
Participants invited to feedback talks on quality
Statistics Austria: Director General-Statistics Quality Management Unit, TQM-Board Experts responsible for the product
Members of the Quality Assurance Committee
of the Statistical Council
Main users, external/internal experts

26. Organisation of feedback talks (5)
List of key questions for the feedback talks:
1. Which are the demands/expectations of the user/client/expert with regard to the quality of the product?
2. Are these demands/expectations fulfilled?
Are the used inputs (data sources), processes and applied methods adequate, of sufficient quality and sufficiently documented?
Are there guidelines for quality assurance, are they sufficiently documented and observed?
Are “good/best practices” considered in the production of statistics? (e.g., approved/successful methods/ trouble shooting in other comparable areas/problems, in other countries)?

27. Organisation of feedback talks (6)
Are the burdens imposed on the data providers justified and as low as possible?
7. Which are the strengths of the product that should be retained at any rate?
8. Where should improvements be undertaken?
9. Is it possible to improve information on methods and quality of the statistical product for the user/client (improvement of the standard documentation)?

28. Recommendations for improvement measures (examples) (1)
Short Term Statistics-Services:
Improvement of seasonal adjustment Short Term Statistics-Industries and Construction:
Improvement of imputation
Structural Business Statistics:
Detailed analysis for improving coherence with foreign trade statistics

29. Recommendations for improvement measures (examples) (2)
Transport Statistics (road transport):
Advancement of the dates for press releases Comparisons with administrative data (toll data)
Special quality studies/further quality analysis
Tax Statistics:
Data matching with the Central Population Register and the Business Register in order to improve regional breakdown
E-Commerce Statistics:
Assessment of the effect of voluntariness (no obligation of respondents to give information) on data quality

30. Recommendations for improvement measures (examples) (3)
Micro-census:
Measures for reducing the high percentage of „third party“ information (answers given not by the persons concerned but by another household member)
Setting up a workshop with internal and external experts to deal with imputation methods
Labour Force Survey:
Unemployement: improvement of the coherence with administrative data (registered unemployed)

31. Thank you!