1. “Quality Measurement at Statistics Austria”
Tacis High Level-Seminar Vienna, August 31 – September 2, 2005
Session 5:
Assuring High Quality Products in Official Statistics
by Quality Management
Doc. TACIS
Statistics Austria
Thomas Burg

2. Quality Measurement at Statistics Austria
Thomas Burg

3. Outline of Presentation
Definition of Quality
Instruments for Quality Measurement
Quality Reports
Consequences and user oriented purposes

4. => Different Answers Quality?
What is Quality of a statistical product (I)?
When do you think a statistical product is good?
This question can be asked to a user
citizen
customer
politician
scientist
This question can be asked to a producer
=> Different Answers
Quality?

5. Possible Answers What is Quality of a Statistical Product (II)?
It is good, when the source is reliable
It has to be accurate
It has to be up to date
The sample must be representative
It must be well documented
The variance should be low
The concepts must be clear
I have no idea

6. => A Statistical Product is a complex system
What is Quality of a Statistical Product (III)?
The answers are fuzzy but most of the aspects are justified
One of the most important consequences is that the approach must be multidimensional
What do we need to evaluate the product
Metadata
Experience
Agreed standards and definitions
=> A Statistical Product is a complex system

7. What is important for the definition?
Multidimensionality
Understandability for user
Transparency
Process vs. product quality
Measurability

8. Definition of Quality in general
ISO /1994
Quality is the totality of features and characteristics of a product or service that bear on its ability to satisfy stated or implied needs.

9. What is specific for official statistics
Large data sets
Mandatory surveys (Sometimes)
Official character of figures
Information provider for policy makers
Constraints by public budget

10. Quality of Statistics is defined with reference to the six criteria:
EUROSTAT Definition of Quality of Statistics(I)
Quality of Statistics is defined with reference to the six criteria:
Relevance
Accuracy
Timeliness and Punctuality
Accessibility and Clarity
Comparability
Coherence

11. EUROSTAT Definition of Quality of Statistics(I)
These criteria are covering a good part of quality of a statistical product but there remains a residual which seems to be unexplainable (“White Noise”).
The criteria are not always equal in their ranking or their weights.
=>
Q= W1*Rel+W2*Acc+W3*Tim+W4*Acl+W5*Comp+W6*Coh+є

12. RELEVANCE
Relevance is the degree to which statistics meet current and potential users’ needs. It refers to whether all statistics that are needed are produced and the extent to which concepts used (definitions, classifications etc.) reflects user needs.

13. ACCURACY
Accuracy in the general statistical sense denotes the closeness of computations or estimates to the exact or true values.

14. TIMELINESS AND PUNCTUALITY
Timeliness of information reflects the length of time between its availability and the event or phenomenon it describes
Punctuality refers to the time lag between the release date of data and the target date when it should have been delivered, for instance, with reference to dates announced in some official release calendar, laid down by Regulations or previously agreed among partners.

15. ACCESSIBILITY AND CLARITY
Accessibility refers to the physical conditions in which users can obtain data: where to go, how to order, delivery time, clear pricing policy, convenient marketing conditions (copyright, etc.), availability of micro or macro data, various formats (paper, files, CD-ROM, Internet…), etc.
Clarity refers to the data’s information environment whether data are accompanied with appropriate metadata, illustrations such as graphs and maps, whether information on their quality also available (including limitation in use…) and the extend to which additional assistance is provided by the NSI.

16. COMPARABILITY
Comparability aims at measuring the impact of differences in applied statistical concepts and measurement tools/procedures when statistics are compared between geographical areas, non-geographical domains, or over time.

17. COHERENCE
The coherence between statistics is orientated towards the comparison of different statistics, which are generally produced in different way and for different primary uses.
When originating from a single source, statistics are coherent in that elementary concepts can be combined reliably in more complex ways. When originating from different sources, and in particular from statistical surveys of different frequencies, statistics are coherent insofar as they are based on common definitions, classifications and methodological standards. The messages that statistics convey to users will then clearly relate to each other, or at least will not contradict each other.

18. General aspects of the criteria
They are not measurable by a certain figure
Multidimensional for themselves
Accuracy: Sampling + nonsampling
Offers a wide variety for indicators
Measurement sometimes a problem
Not all criteria really fit exactly for all products

19. Tradeoffs
If you loose quality in one criteria when deliberately improving another criteria we call it a tradeoff between two quality criteria.
Timeliness <-> Accuracy
Relevance <-> Comparability
Timeliness <-> Coherence
Accuracy <-> Coherence
Etc.

20. Frame for quality reporting
Austrian Federal Statistical Law 2000
Obligations Against EUROSTAT
TQM Board was established
Desire of our Statistical Council to learn more about our products

21. Historical background
Quality Reports
Historical background
In 1998 it became clear that the legal status of official statistics will change
Innovative projects: SYSQUAST
Team of well-proofed statisticians discussed concepts of EUROSTAT
Detailed quality reports were requested for some products by EUROSTAT
First prototype of quality report was developed (Set of indicators)

22. Indicators for a statistical product
Numerical
Binary
Textual
Restrictions in MS-Access
Links to more detailed descriptions

23. Detailed Quality Report (I) Historical aspects and motivation
Requirements by EUROSTAT for some products
Desire of Statistical Council to evaluate some key products
Fulfill principles of TQM sufficiently
Compendium of metadata and description of used methods
Achieving a documentation of the whole production process

24. Detailed Quality Reports (II)
Problems
Who is paying for it
Time consuming
Intensive methodological support
Discussion which products should be involved

25. Detailed Quality Reports (III)
Publication Strategy
Which information should be transferred to the user.
Maybe different user classes
Standard documentation will contain several quality aspects
Standard documentation should be placed on the internet

26. Detailed Quality Reports (IV)
Management Model

27. Relation Standard documentation <-> Quality Report
contains:
Detailed Quality
Report
Metadata
Feedback talks
Quality Issues
Internet, publication

28. Detailed Quality Reports so Far
Plan of establishing quality reports for every relevant product in the next 5 years (since 2003)
Structured after subject matters (Directorates)
Detailed schedule for receiving the reports
Feedback Talks

29. Feedback Talks (I)
Every Quarter of a year is planned 2 month in advance
Set of possible participants (suggested by subject matter division)
One product per month is selected

30. Feedback Talks (II)
Quality report is delivered by subject matter division
Quality report is discussed by quality management, methods, responsible statisticians
Quality Report is embedded in a fixed frame
Should contain
Important hints
Description of methods
Quality issues

31. Feedback Talks(III)
Fixed procedure
Stepping through the report
Improvement of the report
Improvement of the product
Minutes containing all proposals
„To do‘s“

32. Thank you for your attention!