Justin D. Nguyen Carma R. Hogue Automatically Generated Quality Control Tables and Quality Improvement Programs Justin D. Nguyen Carma R. Hogue 06/2018

Outline Overview of the Current Quality Audit Program (QAP) Challenges Goals and Achievement of Quality Processing Procedures for Improving Quality and Controlling Costs Future Steps

Overview of the Current Quality Audit Programs (QAP) – Results of Current QAP Five-Year Cycle Over 87% of all audited programs found to be largely compliant Only one program with no compliance issues and no recommendations for improvement 38 standards found noncompliant across all 31 audited programs

Challenges What caused these issues? Program areas are not realizing certain activities should be documented Lack of accurate and current program documentation led to managers having difficulty finding copies of documents they believed existed Document management system software by outside vendor

Ultimate Goal of Quality Improvement Assure Data Users that Our Products Are of the Highest Quality the public sponsors joint partners other customers Continually Improve All Processes automate quality control systems conduct research and program evaluation Develop Quality Control Procedures to Monitor and Evaluate All Processes

Continuous Quality Improvement Automate Quality Control Tables for Improvement: Examples: nonresponse bias analysis total quantity response rate unit response rate Monitor Progress Results monitor survey response rates monitor complete, incomplete, and follow-up interview cases Apply Tableplots and Big Data Methods

Continuous Quality Improvement (cont’d) Example of monitoring progress results

Continuous Quality Improvement (cont’d) Applying Quality Statistical Standards throughout the Survey Life Cycle Planning and Development Collecting and Acquiring Data Capturing and Processing Data Producing Estimates and Measure Preparing Report Releasing Information Products

Continuous Quality Improvement (cont’d)

Procedures for Improving Quality

Procedures for Improving Quality (cont’d) Apply Tableplots and Big Data Methods: build more “adaptive” editing process reduce biases during the edit process reduce cost and improve timeliness

Procedures for Improving Quality (cont’d) Tableplot of Services Annual Survey Data through March 31, 2017

Procedures for Improving Quality (cont’d) Tableplot of Services Annual Survey Data through June 30, 2017

Future Steps Automate quality control procedures and implementing quality processes in the Survey Life Cycle Build self-documenting systems to ensure documentation matches what system and programs are doing Build quality control measures for emerging methodologies Automate editing as much as possible to ensure that all editing processes are repeatable Make use of Big Data and tableplots to selectively edit large datasets Make use of paradata and key performance indicators to improve data quality

Questions and Contact Information Questions Contact Information Justin Nguyen, Staff Chief Justin.D.Nguyen@Census.Gov Carma Hogue, Director of Research & Methodology for Econ. Programs Carma.Ray.Hogue@Census.Gov