Experiences in Developing Statistical Quality Frameworks Michael Colledge, Statistical Consultant 27 June 2018 Session 2

Context for Paper and Presentation Need for and benefits of a statistical quality framework Systematic mechanism for identification of quality problems, Basis for creating and maintaining a quality culture Source of reference material for quality training. Makes transparent the processes by which quality is assured Reinforces image of office as a credible provider of quality statistics. Facilitates exchange of ideas on quality management with other national and international producers of statistics Statistical offices undertake a wide range of quality initiatives with or without an overall framework What happens Do the quality initiative meet their objectives? What are the lessons learned? Based on response by 7 statistical offices I have been personally involved in some of the initiatives and wanted to hear what happened

Structure of Presentation Introducing and maintaining a QAF Support for QAF implementation Quality reviews of processes and outputs Support for other national agencies producing statistics Quality management and other cross-cutting functions General messages

Introducing and maintaining a Quality Assurance Framework QAF at Statistics Canada First released 1999 Objectives Identify quality management and risk mitigation strategies within context of StatCan's priorities and values Reference quality practices in place Agency-wide, and by individual program areas Most recently updated (third version) 2017 Challenge to keep relevant New data sources, technologies, practices SCF=QAF Satcan QAF was the first

Introducing and maintaining a QAF (continued) UN Statistical Quality Assurance Framework (SQAF) Established by Committee for Chief statisticians of UN System Comprises template and guidelines Completed March 2018 For use by UN agencies without QAF, or as basis to revise QAF UN Commission for Trade and Development SQAF UNCTAD has decentralised statistical system Need common understanding of quality across organisation Statistics Branch previously developed own QAF Awaited implementation to have support of UN SQAF

Support for QAF implementation Data quality network Established 2009 Comprised three groups Core team Forum for Data Quality Unit to present strategy Quality circle Subject matter heads Relate quality principles back to staff Quality working group Quality managers from regional offices Only quality working group is still active

Support for QAF implementation Implementation of Code of Good Practice (Peru) Lessons learned An organisational unit must be assigned overall responsibly Need to interpret commitments in terms of measurable indicators Need quality support tools to facilitate implementation Quality Training (Statistics Canada) Pilot course on quality management Converted to self-directed learning

Quality reviews of processes and outputs Quality review program (Statistics Canada) Started 2008 Independent peer review Objectives spread knowledge of quality assurance practices and risks they can mitigate identify where resources needed to reduce risks to quality About 5 reviews per year Discontinued 2014 Sufficient corporate oversight by other means Corporate IT solutions project, financial and human resource management frameworks Raised staff awareness.

Support for other national agencies producing statistics Statistical Regulations (Statistics Sweden) Three regulations Updated quality concept Updated quality declaration/report to accompany outputs Annual self evaluation form Thee sets of guidelines Handbook on quality for official statistics of Sweden (2016) Handbook on evaluation of quality of official statistics in Sweden (2017) What identifies official statistics in Sweden (in preparation) Data Quality Toolkit (Statistics Canada)

Support for other national agencies producing statistics Statistical Clearing House (Australian Bureau of Statistics) Started 1997 Objectives to promote good survey practice and to minimise respondent burden by requiring clearance of any survey of more than 50 businesses by staff of SCH In 2016/17 About 150 surveys reviewed 75 full review Reduced respondent burden by 3900 hours Improved quality Ceased operations mid 2017

General messages for quality initiatives Evolve or expect to discontinue Involve staff on whom activities have direct impact Provide quality tools to support quality initatives Especially to other ONAs Coordinate with other cross-cutting frameworks Quality assurance can be embedded within them

Experiences in Developing Statistical Quality Frameworks Thank you for your attention Michael Colledge, Statistical Consultant michael.colledge@gmail.com