Quality Review of Key Indicators at Statistics Canada ICES-III, June 2007 Claude Julien and Don Royce

Outline Quality Management at Statistics Canada Quality review process Recommendations Observations on the review process Conclusion

Quality Management at STC

1997 Quality Audit Quality Assurance Framework (QAF) Audit on 4 key surveys Office of the Auditor-General recommended to examine a wider application of enhanced self-assessment

Quality Assurance Framework (QAF) Quality definition Fitness-for-use: relevance, accuracy, timeliness, coherence, accessibility, interpretability Management of each dimension Policies, standards, guidelines and practices Five main systems: users liaison, planning, methods and standards, dissemination and reporting (Brackstone 1999, Survey Methodology)

Integrated Program Reports (IPR) Quadrennial report and biennial follow-up Quality, costs, management of HR and strategic direction Self-assessment and user survey or consultation Focus on quality of outputs Review and feedback from Policy Committee

Quality Review Process

Initiating the review Some serious errors in the preceding 18 months International trade Productivity estimates Consumer Price Index Questions from Finance, other key users and media Policy Committee created Steering Committee of DGs to review quality assurance practices Headed by DG-Methodology and assisted by head of Quality Secretariat

Objectives of the review Identify best practices that should be promoted to other programs Identify specific weaknesses where action is needed

Programs Reviewed Monthly programs Labour Force Survey Retail Trade Survey Manufacturing International Trade Consumer Price Index GDP by industry Quarterly programs Income and Expenditure Accounts Labour Productivity Balance of Payments

Challenges – Types of programs Labour Force Survey Retail Trade Survey Manufacturing International Trade Consumer Price Index GDP by industry Income and Expenditure Accounts Labour Productivity Balance of Payments Traditional Administrative data Mixture of data Derived data Mixture of types

Challenges – Program process Process = complex series of steps Set objectives, define population, create frame, select sample, collect data, etc. Identify a reasonable number of recognizable steps Focus on one or two

Challenges – Aspects of a step Design Implementation Execution Evaluation

Challenges – Dimensions of quality Relevance Accuracy Timeliness Coherence Interpretability Accessibility

Mandate of the review Review the practices during production (execution) to prevent the release of erroneous data (accuracy) Particular focus on the certification and data dissemination of data Review of Dissemination and Communications Program Identify the underlying risk factors and their importance.

Review Teams Identify and volunteer 10 reviewers 10 review teams 3 reviewers + member of Steering Committee per team Lead reviewer

Organization Policy Committee Directors General Assistant Directors Directors Mandate Recruitment Program

Introductory meeting Program director, lead reviewer and head of QS Introduce the review process Introduce the program Identify main steps of program process

Main steps of program process Preparation for certification Data collection Editing and transformation Imputation and estimation Certification Data release Post-release

Gather information Semi-structured questionnaire Cover the topics of the mandate Not to be completed Interviews with review team and program director and managers Available documentation Meta-data Program reports

Assessment Individual reports Report by lead reviewer Reviewed by Steering Committee Vetted by program director One-day debriefing session with all reviewers Summary report Report by Steering Committee Presentation to Policy Committee

Taking action Disposition of recommendations by Policy Committee Funding for corrective actions Follow-up in Integrated Program Reports

Recommendations

Main recommendations Program managers are well aware of the risks factors The programs are well-positioned to identify the improvements Some programs would benefit more than others from additional investment Human resources concerns dominate all other risk factors.

Main recommendations The existence of a research and analysis capacity separate from the production operation is a key factor in assuring quality. There are numerous best practices in all programs that can usefully be shared within the Agency. Statistics Canada should develop a formal Quality Incident Response Plan Proposals to further increase the timeliness of the programs should be regarded with extreme caution Statistics Canada should establish an ongoing program of Quality Assurance Reviews

Observations on the review process

Aspects to bring forward Clear mandate from Policy Committee to focus the review Participation at the DG level to assure legitimacy Strong cooperation from program directors and managers Concurrent reviews to assure attention and team work Knowledge and experience of reviewers Composition of review teams to assure some standardization Excellent learning, development and networking opportunity for reviewers Overall positive experience for the reviewers Collection approach and tools One-day debriefing session

Aspects to improve Better assessment and reporting of level of various risk factors Timeliness 10 reviews in five months (Oct-Mar) Reviewers spent around 15 days More interviews than expected Time to write, review and translate reports More timely process and dissemination More time to schedule meetings around production More standard program reports

Conclusion The review was developed and carried out very quickly The review was a success thanks to all participants The review was not thorough; it complemented other key quality management mechanisms Mandate to propose an on-going review program Build on recent review Criteria for selecting surveys Tool box / Flexible approach Part of learning and development program

Merci For more information, or to obtain a French copy of this presentation, please contact: Pour de plus amples informations ou pour obtenir une copie en français du document, veuillez contacter : Claude Julien / Courriel: Phone number / Téléphone: