1. Justin D. Nguyen Carma R. Hogue
Automatically Generated Quality Control Tables and Quality Improvement Programs
Justin D. Nguyen
Carma R. Hogue
06/2018

2. Outline Overview of the Current Quality Audit Program (QAP) Challenges
Goals and Achievement of Quality Processing
Procedures for Improving Quality and Controlling Costs
Future Steps

3. Overview of the Current Quality Audit Programs (QAP) – Results of Current QAP Five-Year Cycle
Over 87% of all audited programs found to be largely compliant
Only one program with no compliance issues and no recommendations for improvement
38 standards found noncompliant across all 31 audited programs

4. Challenges What caused these issues?
Program areas are not realizing certain activities should be documented
Lack of accurate and current program documentation led to managers having difficulty finding copies of documents they believed existed
Document management system software by outside vendor

5. Ultimate Goal of Quality Improvement
Assure Data Users that Our Products Are of the Highest Quality
the public
sponsors
joint partners
other customers
Continually Improve All Processes
automate quality control systems
conduct research and program evaluation
Develop Quality Control Procedures to Monitor and Evaluate All Processes

6. Continuous Quality Improvement
Automate Quality Control Tables for Improvement:
Examples:
nonresponse bias analysis
total quantity response rate
unit response rate
Monitor Progress Results
monitor survey response rates
monitor complete, incomplete, and follow-up interview cases
Apply Tableplots and Big Data Methods

7. Continuous Quality Improvement (cont’d)
Example of monitoring progress results

8. Continuous Quality Improvement (cont’d)
Applying Quality Statistical Standards throughout the Survey Life Cycle
Planning and Development
Collecting and Acquiring Data
Capturing and Processing Data
Producing Estimates and Measure
Preparing Report
Releasing Information Products

9. Continuous Quality Improvement (cont’d)

10. Procedures for Improving Quality

11. Procedures for Improving Quality (cont’d)
Apply Tableplots and Big Data Methods:
build more “adaptive” editing process
reduce biases during the edit process
reduce cost and improve timeliness

12. Procedures for Improving Quality (cont’d)
Tableplot of Services Annual Survey Data through March 31, 2017

13. Procedures for Improving Quality (cont’d)
Tableplot of Services Annual Survey Data through June 30, 2017

14. Future Steps
Automate quality control procedures and implementing quality processes in the Survey Life Cycle
Build self-documenting systems to ensure documentation matches what system and programs are doing
Build quality control measures for emerging methodologies
Automate editing as much as possible to ensure that all editing processes are repeatable
Make use of Big Data and tableplots to selectively edit large datasets
Make use of paradata and key performance indicators to improve data quality

15. Questions and Contact Information
Questions
Contact Information
Justin Nguyen, Staff Chief
Carma Hogue, Director of Research & Methodology for Econ. Programs