European Conference on Quality in Official Statistics 8-11 July 2008 Mr. Hing-Wang Fung Census and Statistics Department Hong Kong, China (

2 A Further Step in Quality Assurance for the Official Statistics of Hong Kong, China

3 Endeavours to Ensure Quality of Official Statistics  Department’s vision statement: “To provide high-quality statistical services, contributing to the social and economic developments of Hong Kong.”  Pre-requisites:  good management of statistical systems  an enabling working environment for continuous quality improvement

4 Quality Assurance Review Programme  Started with the documentation of each process of statistical systems  Next Steps  Development of Data Quality Assessment Framework  Improvement on statistical processes of statistical systems  Key features involve  promoting self-assessment and enhancement of individual statistical systems; and  conduct of in-depth third party review by an independent quality assurance team

5 Recent Developments in Quality Assurance Review  Developed a set of general guidelines on preparation of proper documentation of statistical systems containing:  essential items that should be covered;  salient points that should be noted; and  good examples and practices for each statistical process

6 Recent Developments in Quality Assurance Review (Cont’d)  Conducted self-assessment of documentation by the subject professionals in accordance with the set of general guidelines

7 Recent Developments in Quality Assurance Review (Cont’d)  Developed a central database on good practices in respect of documentation for use within the Department

8 Data Quality Assessment Framework Quantitative  through a set of quality indicators  modelled on the DESAP checklist promulgated by Eurostat Qualitative  through a series of open-ended questions under the Data Quality Assessment Framework developed by the IMF + Existing New Documentation Review Assessment of quality level achieved for statistical products

9 Preliminary Set of Quality Indicators  Criteria for devising the quality indicators:  Manageable in size  Relevant to the statistical products compiled  Easy to compile  Easy to interpret  Allowing for comparison over time  Representative for the quality dimensions being adopted  Adaptable

10 Preliminary Set of Quality Indicators (Cont’d)  Devised 19 quality indicators under 6 quality dimensions  Translated to 19 assessment questions  With pre-defined response categories  15 of them in five-point ordinal scales with brief explanation for each point, reflecting quality in descending order from 5 to 1  Nearly half of these 15 questions (i.e. 7 questions) are under Relevance and Accuracy  4 of them either for assisting assessors in determining rating for some of the 15 questions or giving hints to the subject professionals on area for improvement

11 Preliminary Set of Quality Indicators (Cont’d) DimensionElementIndicator 1. Relevance  Identification of users1. Means to identify user groups and their frequency 2. Information available on the users   Assessment of user satisfaction 3. Means to collect information on user satisfaction 4. Information available on user satisfaction   Monitoring user needs5. Means to identify users’ needs of information 6. Information available on users’ needs of information  2. Accuracy  Assessment of non- responses 1. Unit non-response rate  2. Classification of unit non-responses for adjustments 3. Imputation rate of key variable   Assessment of sampling error 4. Coefficient of variation of the statistics   Assessment of revision between provisional and final statistics 5. Extent of revisions between the provisional and final key statistics   in five-point ordinal scales, reflecting quality in descending order from 5 to 1

12 Preliminary Set of Quality Indicators (Cont’d) DimensionElementIndicator 3. Timeliness  Production time1. Time lag between the reference period and the release date of the preliminary results  2. Time lag between the reference period and the release date of the report  4. Accessibility  Assistance to users on application of statistics 1. Percentage of enquiries meeting the performance standards and targets of C&SD on the waiting time   Metadata accessibility 2. Ease of getting metadata by the users  5. Comparability  Comparability over time1. Extent of comparability of the statistical product over time   Comparability across domains 2. Extent of comparability of the statistical product across different domains  6. Coherence  Coherence of results for different frequencies 1. Extent of coherence of the statistics based on results for different frequencies   Coherence with other related statistical products 2. Extent of coherence of the statistics within the same socio-economic area   in five-point ordinal scales, reflecting quality in descending order from 5 to 1

13 Illustration of Data Quality Assessment Mechanism Step 1:Set a benchmark for each of the 15 assessment questions by respective subject professional, specifically for each of their statistical products e.g. How do you appraise the time lag between the reference period and the release of the report for the statistical product? Name of report: Report on Annual Survey of Industrial Production RatingResponseBenchmark Time lag between the reference period and the release date of the report 1There is a substantial time lag  16 months 2There is a large time lag14 - <16 months 3There is a certain time lag12 - <14 months 4There is a small time lag10 – <12 months 5There is a very small time lag< 10 months

14 Illustration of Data Quality Assessment Mechanism (Cont’d) Step 2: Moderate all the benchmarks by quality assurance (QA) review team Step 3: Moderate and endorse the set of benchmarks of each statistical product by respective senior officer Step 4: Answer the assessment questions by respective subject professional based on the set of benchmarks endorsed by senior officer Step 5: Mark the responses of questions in the assessment diagram and compile the product quality assessment score by QA review team

15 Illustration of Data Quality Assessment Mechanism (Cont’d) Product Quality Assessment Diagram Statistical Product: Statistics on the operating characteristics of quarrying, manufacturing, and electricity and gas sectors

16 A B Illustration of Data Quality Assessment Mechanism (Cont’d) In this example, the score is 49.  Its value ranges from 4 to 100.

17 Illustration of Data Quality Assessment Mechanism (Cont’d) Interpretation of the results of assessment based on the product quality assessment diagram and score:  Quality of the statistical product is not poor but still far from perfect.  Quality level of the statistical product varies a lot among the different dimensions.  Relevance is the weakest dimension, in particular the identification of users.  Accuracy level is assessed to be just on the average and hence some effort should be put in improving this area, in particular the reduction of non-responses.  The performance of the product in other dimensions are above average.

18 Limitations of the Product Quality Assessment Diagram and Score  Modification of the diagram and formula for computing the score necessary if some indicators are not applicable  Equal weights assumed for the 15 indicators  The score is ordinal in nature  Different permutations of the ratings for the 15 indicators may result in same score  Sequence of indicators may affect the value of score

19 Way Forward  To supplement and refine the set of quality indicators of statistical products and conduct the assessment on a regular basis  To further promote a culture of systematic improvement drive within the department  To improve the statistical processes, concepts and methodology of each statistical system through continuous assessment

20 Thank you !

21 Computation Formula of Product Quality Assessment Score 1 24 o A B ai bi supplementary

22 Computation Formula of Product Quality Assessment Score (Cont’d) When the rating of all the 15 questions are equal to 5, A=a 1 =a 2 =…=a 15 =5 and B=b 1 =b 2 =…=b 15 =5 supplementary 24 o ai bi B A

23 Computation Formula of Product Quality Assessment Score (Cont’d) When the rating of all the 15 questions are equal to 1, a 1 =a 2 =…=a 15 =1 and b 1 =b 2 =…=b 15 =1 supplementary ai bi B A 24 o