A Basis for Better Decision Making: Improvements to Quality Reviews and Reporting Catherine Bremner and Sarah Tucker Office for National Statistics

Motivation Quality Centre is Centre of methodological expertise in statistical quality and respondent burden Working over the past 12 months to streamline current quality initiatives and drive innovation

Outline Introduction to quality reporting Quality reporting developments Using quality reports to drive quality improvements The new quality review process at ONS

Quality Reporting Overview Why do we report on Quality? Give users enough information to decide on suitable uses for the data Meet obligations under Code of Practice Meet requirements from Eurostat

Quality and Methodology Information Static quality information Updated on annual basis or when there are major methodological changes Includes: history of output details of statistical process reports against European Statistical System Quality Dimensions Extensively quality assured

Quality and Methodology Information

Current format 5 years old – time to review Changes to website – accessibility More information on user personas - Inquiring Citizen, Data Forager and Expert Analyst Need to change the way we present the data – content unlikely to change much as meets our obligations

Reviewing Presentation of Quality Information Aims to extend the reach of quality information to user personas – alongside current QMIs Investigations currently focussing on information that users need so they can’t misuse data First steps: consulting with data producers, methodologists and other stakeholders

Reviewing Presentation of Quality Information One size does not fit all Pool of suggested pieces of information to choose from Fit together to give picture of how not to misuse data Held focus groups/workshops and individual meetings to gain thoughts on what should be considered for inclusion

Internal Consultation Ongoing – Points Raised Four focus groups have been held - 2 each for output managers and quality assurers Individual meetings with output managers and other stakeholders Some themes are coming through consistently: uncertainty – is it an estimate etc, periodicity, coherence, accuracy and health warnings. As expected, emphasis is different across type of outputs – admin data vs. Survey

Next steps Compiling a shortlist of these vital pieces of information Work with data producers to produce pilot User test Review results Propose action

Quality Reviews at ONS Previously carried out Annual Quality Reviews Made use of Quality, Methodology and Harmonisation Tool (QMHT) Responding to feedback, developed a new process during summer 2014

Regular Quality Reviews (RQR) Face to face meeting between Output manager, Deputy Director, G6/G7 Methodologist and Quality Centre representative Makes use of existing information from Value Engineering and Quality and Methodology Information (QMI) reports, which are discussed at the meeting Quality Assurance (DD) walk-through incorporated into the meeting

Regular Quality Reviews (RQR) G6/G7 methodologist produces bespoke recommendations, which will be presented at Management/Project boards Output reviewed once every 3 years Supports the Code of Practice Piloted on 3 outputs during summer 2014 Process rolled out in November 2014; 13 reviews conducted to date

Why are RQRs required? Support requirements of the Code of Practice Assesses the output Adds value to output through bespoke recommendations Allows recommendations to be addressed, improving the output

What do we expect? What we already require - QMI, Quality Assurance (DD) walkthrough, Quality checklist Quality Centre collate all the relevant documents e.g. QMI, Value Engineering, Assessment Reports to be reviewed Output Manager, DD, and team members (as required) to meeting

Recommendations – Adding Value Aim to write up within two weeks of the meeting Assigned a RAG status Prioritised from a Methodological and ONS view Timetable of when/how recommendations are dealt with depend on what they are Range from a quick win – update QMI or longer term plan of getting work onto business plan Role for Quality Centre in monitoring

Relevance for GSS Regular Quality Review approach is a tool that could be used by GSS Expected that senior statistician could carry out review Alternatively, support from Methodology Advisory Service The approach will be included in the GSS quality resources

Contacts Quality Reporting Sarah Tucker Regular Quality Reviews Catherine Bremner,