1. U.S. Department of Commerce Economics and Statistics Administration U.S. CENSUS BUREAU Using System Paradata to Target and Evaluate Data Collection Operations Stephanie Coffey WSS Seminar 4/18/2016

2. U.S. Department of Commerce Economics and Statistics Administration U.S. CENSUS BUREAU Overview National Survey of College Graduates (NSCG) Survey Background System Paradata Collected 2 Examples of Paradata Use –Web instrument performance –Intelligent Mail Barcoding Wrap-Up 2

3. U.S. Department of Commerce Economics and Statistics Administration U.S. CENSUS BUREAU National Survey of College Graduates Sponsored by NCSES at NSF U.S. college-educated science & engineering population –Education, demographic and employment info for S&E degrees and occupations Recent college graduates Women, minorities, persons with disabilities Longitudinal survey with four rotating panels –New cohort is selected out of ACS for 1 st interview –Three follow-up interviews (every 2-3 years) –Total sample size is approximately 135,000 cases 3

4. U.S. Department of Commerce Economics and Statistics Administration U.S. CENSUS BUREAU National Survey of College Graduates Standard Data Collection Path (New Cohort) Weeks 0 – 6 Initial Invite Phase Prenotice Web Invites #1&2 Weeks 0 – 6 Initial Invite Phase Prenotice Web Invites #1&2 Weeks 7 - 11 1 st Reminder Phase 1 st Paper Q’naire Web Invite #3 Weeks 7 - 11 1 st Reminder Phase 1 st Paper Q’naire Web Invite #3 Weeks 12 - 17 Nonresp. FU Phase CATI NRFU Begins Web Invite #4 Weeks 12 - 17 Nonresp. FU Phase CATI NRFU Begins Web Invite #4 Weeks 18 - 22 2 nd Reminder Phase 2 nd Paper Q’naire Web Invite #5 Weeks 18 - 22 2 nd Reminder Phase 2 nd Paper Q’naire Web Invite #5 Weeks 23 - end Final Reminder Phase Web Invite #6 Weeks 23 - end Final Reminder Phase Web Invite #6

5. U.S. Department of Commerce Economics and Statistics Administration U.S. CENSUS BUREAU National Survey of College Graduates Types of Paradata Collected from Major Operations Outgoing Mailings: Mailed Web Invites, Reminders, Questionnaires –Census – National Processing Center (NPC) Mail-Out Date Undeliverable-as-Addressed (UAA) Status/Reason Upon Check-In at NPC –USPS – Integrated Postal Tracking Service (IPTS) – New in 2015 First Scan into Postal System / Last Scan out of Postal System (out for delivery) UAA Status/Reason Upon USPS Determination Incoming Mailings: Return Paper Questionnaires –Census – NPC Return Date Upon Receipt and Check-In at NPC –USPS – IPTS First Scan Back into Postal System / Last Scan out of Postal System 5

6. U.S. Department of Commerce Economics and Statistics Administration U.S. CENSUS BUREAU National Survey of College Graduates Types of Paradata Collected from Major Operations Web Access from Respondent –Server-side Paradata Log-in Attempt Date/Time Log-in Success Status Time-in-Instrument Log-out Date/Time Submission Date/Time Outgoing Telephone Contact Attempt –Centralized Call Center Case Management Call Attempt Date/Time Time in Instrument Outcome of Call Attempt Other Operations Not Included (Email Reminders, Incoming Telephone) 6

7. U.S. Department of Commerce Economics and Statistics Administration U.S. CENSUS BUREAU Web Instrument Performance Event

8. U.S. Department of Commerce Economics and Statistics Administration U.S. CENSUS BUREAU 2013 Data Collection Web is initial mode offered in NSCG new cohort –Web first used as production mode in 2013 –Prior to 2013, web experiments, mode preference experiments NSCG achieves about 30% response rate in the first month –NSCG has 6 month data collection First invites with usernames and passwords sent on 2/20 –Arrival expected between 2/22-2/24 (Sat or Mon)

9. U.S. Department of Commerce Economics and Statistics Administration U.S. CENSUS BUREAU Web Interruption Afternoon of 2/25 – Morning 2/26 –External DHS simulated attack to check data security (unscheduled, unknown by Census) –IT teams at Census: maintenance & applying updates Resulted in performance issues in the web instrument –No advance warning of external simulations –NSCG was informed by respondents calling TQA line: trouble logging in, delays in navigating through survey Question: Can we figure out who was affected?

10. U.S. Department of Commerce Economics and Statistics Administration U.S. CENSUS BUREAU Evidence in Server-Side Paradata servicetypeidtime (seconds)type2 NSCGparadata108000841361842005login NSCGparadata108000841361845429submitted NSCGparadata108000841361845466logout NSCGparadata108001591361843035login NSCGparadata108001591361846593submitted NSCGparadata108001591361846627logout NSCGparadata108002581361847610login NSCGparadata108002581361847625login NSCGparadata108002581361847959login NSCGparadata108020311361840282failed_login NSCGparadata108020311361840295failed_login NSCGparadata108020311361840349failed_login 57 min 60 min 15 sec 5 min 13 sec <1 min

11. U.S. Department of Commerce Economics and Statistics Administration U.S. CENSUS BUREAU Evidence in Server-Side Paradata servicetypeidtime (seconds)type2 NSCGparadata108000841361842005login NSCGparadata108000841361845429submitted NSCGparadata108000841361845466logout NSCGparadata108001591361843035login NSCGparadata108001591361846593submitted NSCGparadata108001591361846627logout NSCGparadata108002581361847610login NSCGparadata108002581361847625login NSCGparadata108002581361847959login NSCGparadata108020311361840282failed_login NSCGparadata108020311361840295failed_login NSCGparadata108020311361840349failed_login 57 min 60 min 15 sec 5 min 13 sec <1 min Patient respondents!

12. U.S. Department of Commerce Economics and Statistics Administration U.S. CENSUS BUREAU Evidence in Server-Side Paradata servicetypeidtime (seconds)type2 NSCGparadata108000841361842005login NSCGparadata108000841361845429submitted NSCGparadata108000841361845466logout NSCGparadata108001591361843035login NSCGparadata108001591361846593submitted NSCGparadata108001591361846627logout NSCGparadata108002581361847610login NSCGparadata108002581361847625login NSCGparadata108002581361847959login NSCGparadata108020311361840282failed_login NSCGparadata108020311361840295failed_login NSCGparadata108020311361840349failed_login 57 min 60 min 15 sec 5 min 13 sec <1 min 2+ Logins in Short Time

13. U.S. Department of Commerce Economics and Statistics Administration U.S. CENSUS BUREAU Evidence in Server-Side Paradata servicetypeidtime (seconds)type2 NSCGparadata108000841361842005login NSCGparadata108000841361845429submitted NSCGparadata108000841361845466logout NSCGparadata108001591361843035login NSCGparadata108001591361846593submitted NSCGparadata108001591361846627logout NSCGparadata108002581361847610login NSCGparadata108002581361847625login NSCGparadata108002581361847959login NSCGparadata108020311361840282failed_login NSCGparadata108020311361840295failed_login NSCGparadata108020311361840349failed_login 57 min 60 min 15 sec 5 min 13 sec <1 min Can’t Successfully Log In

14. U.S. Department of Commerce Economics and Statistics Administration U.S. CENSUS BUREAU Operational Questions Major Concern: Burden / Respondent Frustration –Time in instrument How long did individuals spend logged in? –Server breakoffs Will sample persons come back to respond? Examine Login Events –Login/Submit = 1 event –Login/Submit/Logout = 1 event –Login/Logout = 1 event –If there was no Submission or Logout: Login alone = Server Breakoff Failed login = Failed Login

15. U.S. Department of Commerce Economics and Statistics Administration U.S. CENSUS BUREAU Increase in the mean time-in-instrument

16. U.S. Department of Commerce Economics and Statistics Administration U.S. CENSUS BUREAU Increase in the mean time-in-instrument More server breakoffs than submissions!

17. U.S. Department of Commerce Economics and Statistics Administration U.S. CENSUS BUREAU 17 Mean time in instrument nearly back to average Server breakoffs still higher than expected

18. U.S. Department of Commerce Economics and Statistics Administration U.S. CENSUS BUREAU Action & Results Determined time period of web issue: 6pm – 2am Obtained approval to send special letter to affected cases –Apology and reminder of sample person’s importance to data quality –Turnaround time: 3 business days from analysis to mailing Sent letter to all breakoff cases: –Not an experiment Response rate of affected cases same as unaffected cases a month after mailing –Would this be the same if this issue occurred late in data collection? (Early respondents = higher engagement?) Encouraged that we could be responsive to a new issue

19. U.S. Department of Commerce Economics and Statistics Administration U.S. CENSUS BUREAU Intelligent Mail Barcoding

20. U.S. Department of Commerce Economics and Statistics Administration U.S. CENSUS BUREAU Mailing Operation in NSCG NSCG heavily relies upon mailing operations –Pre-notice Letter –Web invites –Paper Questionnaires –Reminder Letters & Postcards –Prepaid Incentives A respondent receiving all mailings in sequence: 11 mailings (plus any requested remails) Historically a low-information operation –Was the address mailable? –Did we receive something in return? –Only have date of check-in at NPC 20

21. U.S. Department of Commerce Economics and Statistics Administration U.S. CENSUS BUREAU Integrated Postal System Tracking USPS offers IPTS Data Tracks data through each step of postal delivery –First scan at first post office –Every transfer scan –Last scan before delivery –Scan to reroute due to UAAs –Reasons for UAAs 21

22. U.S. Department of Commerce Economics and Statistics Administration U.S. CENSUS BUREAU IPTS for NSCG Two points of interest in the NSCG –How much earlier can we know about UAAs? Can send sample persons to locating earlier Can save cost through not mailing future mailings –How much earlier can we know about return questionnaires? Can reduce burden by not contacting likely respondents Can reduce cost by not contacting likely respondents –Discuss UAA timeliness today 22

23. U.S. Department of Commerce Economics and Statistics Administration U.S. CENSUS BUREAU Implementation of IPTS Add barcodes to mailing packages –Need outgoing barcodes to track progress and UAAs (letters and questionnaires) –Need return barcodes to track progress of return questionnaires (questionnaires only) Redesign envelope to accommodate barcodes on printed letters and questionnaires –Ensure barcode can be scanned by USPS Provide messaging on questionnaire return envelope to insert questionnaire correctly –Ensure barcode is displayed 23

24. U.S. Department of Commerce Economics and Statistics Administration U.S. CENSUS BUREAU Lag in UAA Identification UAAs can be identified at various points –Delivery Point Verification before mailing (not mailed) –During any scan through postal system –When post office attempts to deliver mail After UAA is identified –Mail package is rerouted through USPS back to NPC –Upon arrival at NPC, UAAs are checked in as resources allow Report in Unified Tracking System (UTS) to track –Each mail piece and its mail date –When the UAA was reported by USPS –When the UAA was checked in at NPC 24

25. U.S. Department of Commerce Economics and Statistics Administration U.S. CENSUS BUREAU Lag in UAA Identification 25 List of web invite letters and reminder mailings sent during Weeks 1 - 13

26. U.S. Department of Commerce Economics and Statistics Administration U.S. CENSUS BUREAU Lag in UAA Identification 26 Number of UAAs recorded by the USPS For each data collection operation.

27. U.S. Department of Commerce Economics and Statistics Administration U.S. CENSUS BUREAU Lag in UAA Identification 27 Number & percentage of USPS records that also have a check-in record at NPC.

28. U.S. Department of Commerce Economics and Statistics Administration U.S. CENSUS BUREAU Lag in UAA Identification 28 Number & percentage of USPS records that also have a check-in record at NPC. Question: How long is the lag from when we could see this information in IPTS to when we see it in check- in records at NPC?

29. U.S. Department of Commerce Economics and Statistics Administration U.S. CENSUS BUREAU Lag in UAA Identification 29 Average, median, and max lag between USPS and NPC

30. U.S. Department of Commerce Economics and Statistics Administration U.S. CENSUS BUREAU Action & Results Clear that data from the USPS scanning –More timely –Cost neutral, if not cheaper Still some questions about UAA data –Understand lag between mailing and USPS UAA –Better understand NPC check-in process Records we cannot currently identify Monitor and use USPS UAA data in 2017 Implement return questionnaire scan data in 2017 –Monitor and evaluate –Place potential respondents on hold in other modes until we determine validity of complete

31. U.S. Department of Commerce Economics and Statistics Administration U.S. CENSUS BUREAU Summary High volume of system-generated paradata –Need to know what is available –Need to know who provides it so you can ask –Possible operational changes to collect or use (envelopes) –May need new systems to manage or track paradata Can be monitored to improve data collection –Efficiency and timeliness Might be able to help answer research questions –May need to combine data across modes to answer research questions –Important to keep in mind limitations of system paradata 31

32. U.S. Department of Commerce Economics and Statistics Administration U.S. CENSUS BUREAU Contact Information stephanie.coffey@census.gov stephanie.coffey@census.gov