1. 1 Data Quality Standards at the U.S. Census Bureau Pamela D. McGovern and John M. Bushery U.S. Census Bureau Quality Program Staff Washington, DC 20233 European Conference on Quality in Official Statistics July 2008

2. 2 Agenda Background Development Process Framework and Structure What Worked Well Next Steps Challenges

3. 3 Background

4. 4 Office of Management and Budget (OMB) Provides direction on Government-wide budget and management issues Issued Information Quality Guidelines (2002) Issued Standards and Guidelines for Statistical Surveys (2006)

5. 5 U.S. Census Bureau Composed of separate program directorates: Economic Demographic Decennial

6. 6 Quality Program Initiated in 1998 Purpose –Advance and maintain the Census Bureau interests in quality of its products and processes Key players –Methodology and Standards (M&S) Council –Quality Program Staff Reference: Landman, et al., (Q2001)

7. 7 Methodology and Standards (M&S) Council Steering Committee for the Quality Program Chiefs of methodology & research divisions and senior statisticians and methodologists Ensure sound statistical and methodological practices

8. 8 Quality Program Staff (QPS) Support staff for the M&S Council Facilitate development and implementation of standards and guidelines Evaluate implementation of standards

9. 9 Development Process

10. 10 Original Approach Informal process –Team formed based on need –No formal deadlines Detailed requirements Development periods lengthy –Took from 1 to 4 years

11. 11 New Approach Structured approach that uses: Dedicated Staff Contractor Assistance Directorate Reviewers Focus Groups

12. 12 Dedicated Staff (QPS) Two Mathematical Statisticians and one Survey Statistician Advantages: Development times decrease Cohesion of content improve Consistency of format improve

13. 13 Contractor Assistance Advantages: Provides expertise in conducting focus groups Brings outside perspective Disadvantage: Lacks familiarity with Census Bureau issues

14. 14 Directorate Reviewers Focus group participants and program area managers Advantages: Allows QPS to obtain directorate input early in process Promotes buy-in Disadvantage: Risk exists that can lose buy-in

15. 15 Focus Groups Seven to ten mid-level managers; Two hours in length Advantages: Forum for discussing important issues across directorates May increase buy-in Participants more attentive during the review Disadvantage: Scheduling difficulties can slow down process

16. 16 Process Steps QPS sends draft standard to reviewers QPS compiles comments and produces discussion issues for focus group Contractor conducts focus group QPS revises standard and conducts walk- throughs with the M&S Council QPS prepares “final draft” standard and sends to reviewers along with responses QPS and quality assurance staff review the requirements to determine if they are auditable QPS submits set for final review and approval

17. 17 Framework and Structure of the Standards

18. 18 Framework Lists standards and their purpose –Based on the OMB standards –Based on our existing Census Bureau standards (See our Information Quality website http://www.census.gov/quality/) http://www.census.gov/quality/ Consists of two types of standards –Process Standards (18) –Supporting Standards (5)

19. 19 Process Standards Reflect the phases of the life cycle of a data product: Planning and Design Obtaining Data Processing Data Analyzing Data Disseminating Products

20. 20 Supporting Standards Apply to all phases of the life cycle: Protecting Confidentiality Managing Data and Documents Ensuring Quality Applying for a Waiver

21. 21 Structure of the Standards High-level requirements in list form: State “what” is required; avoid the “how” Include examples to provide guidance Provide links to reference documents and to related standards

22. 22 What Worked Well

23. 23 Accelerated Development Reduced development times due to: Early input Straightforward requirements Individual walkthroughs with the M&S Council members

24. 24 Program Area Input Interviews with program area experts to get assistance in drafting requirements Focus group approach –Structure the discussion based on comments –Overlap participants on related standards –Record the session

25. 25 Focus Group Evaluation Scale: 1= strongly disagree to 5= strongly agree CharacteristicRating The goal of the focus group was clear.4.2 The participant’s point of view was heard.4.5 The goal of the focus group was accomplished.4.0 The length of the focus groups was sufficient.4.2 The participant’s satisfaction with the outcome.4.0

26. 26 Next Steps

27. 27 Next Steps Review the full set of standards for completeness, consistency, and cohesion Submit the standards to the directorates for final review and approval Expect completion by September 2009

28. 28 Next Steps Implementation of the standards –QPS assists the program directorates –Training Evaluations of the implementation –Identify aspects of the standards that need improvement

29. 29 Challenges

30. 30 Challenges Risk of excessive burden –How compliance will affect costs and schedules High-level standards –Avoid being too vague –Program areas interpret standards differently Verification of compliance –Audit program (Bushery et al., Q2006)

31. 31 Thank you Contact Information: Pamela D. McGovern 4600 Silver Hill Road U.S. Census Bureau Washington, DC 20233 pamela.d.mcgovern@census.gov