1. Electronic Questionnaire Collection at Statistics Canada Milana Karaganis Marc St-Denis

2. Outline  Electronic Questionnaire project  Electronic Questionnaire for business surveys 20/10/2015 Statistics Canada Statistique Canada 2

3. Electronic Questionnaire Project  Build generic EQ service to be used by all surveys EQ platform Standard approach to EQ  Create a critical mass of EQ service users 160+ surveys Business surveys and some social surveys  Reduce collection and operations costs  Timeline: April 1, 2010 -> March 31, 2016 20/10/2015 Statistics Canada Statistique Canada 3

4. Electronic Questionnaire Project  E-questionnaire solution Support all EQ applications for all surveys Use Statistics Canada infrastructure Comply with security and confidentiality standards Automate EQ survey application development -> build an engine to render EQ applications based on provided inputs, E-questionnaire Generation System (EQGS) 20/10/2015 Statistics Canada Statistique Canada 4

5. Electronic Questionnaire Project 20/10/2015 Statistics Canada Statistique Canada 5

6. Electronic Questionnaire Project  Standard approach to EQ – corporate solution Develop EQ applications according to EQ standards Apply standard look and feel Build functionalities required for all surveys Use standard collection strategies EQ as initial collection mode in sequential mode collection Use standard development process Shorten development cycle 20/10/2015 Statistics Canada Statistique Canada 6

7. 20/10/2015 Statistics Canada Statistique Canada 7

8. 20/10/2015 Statistics Canada Statistique Canada 8

9. 20/10/2015 Statistics Canada Statistique Canada 9 Progress To-Date  EQ solution implemented Change Management process First set of EQ standards released Survey development process created Initial collection strategies established  Deployed 39 EQ applications and realized efficiencies  Overall success: Lessons learned Big cultural shift

10. EQ Among Business Surveys  Outline Evolution of EQ collection strategies Results observed to date Survey collection strategy guidelines Results of tests 20/10/2015 Statistics Canada Statistique Canada 10

11. Collection Strategies Among Business Surveys  Moving from conventional collection modes (PAPI & CATI) to EQ (over 100 surveys in 5 yrs)  Evolution of multi-modal collection with EQ as primary mode  Measuring results and improving Respecting data quality, timeliness, budgets  Reviewing collection strategies 20/10/2015 Statistics Canada Statistique Canada 11

12. Reviewing Collection Strategy Quarterly Services Indicators (QSI) Survey 20/10/2015 Statistics Canada Statistique Canada 12 Services Indicator Survey (quarterly ) (QSI) 2011 Q4 2011Q1 2012 Mail out / E-invitationOct 3, 2011Mail out / E-invitationJan 4, 2012 e-reminder / faxOct 17 e-reminderJan 18 Phone reminder (all)Oct 24 e-reminderJan 25 Phone reminder (all)Nov 7 e-reminderFeb 8 e-reminder / faxNov 14Phone reminder (all)Feb 15 e-reminder / faxNov 28e-reminderFeb 29 Fax to EQDec 5Phone and fax (all)Mar 7 End of collectionDec 9, 2011End of collectionMar 9, 2012

13. Q4 2011 EQMailTotal Final Sample1,52598.3%271.7%1,552 Response Rate58.5%50.0%58.0% # of cases in NRFU90559.3%2281.5%927 20/10/2015 Statistics Canada Statistique Canada 13 Q1 2012 EQMailTotal Final Sample1,35196.4%503.6%1,401 Response Rate65.6%46.2%65.6% # of cases in NRFU68350.6%4182.0% 724 Reviewing Collection Strategy Quarterly Services Indicators (QSI) Survey

14. EQ Collection Strategy Guidelines  Developing tools to ensure optimal collection strategies (monthly; quarterly; annual) Follow-up schedule geared to multi-mode collection  Standardise approach towards efficiencies  Considering respondent burden, data quality, response rates, costs 20/10/2015 Statistics Canada Statistique Canada 14

15. DateDays after Mail outType of FU Pre-contact n/a Mail-out / E-invitation send date: 0 Collection Start date: Bounce-backs follow up 2 1 week max to complete then switch to mail for those unable to get correct email 1st Follow up attempt date: Phone 30only for those without e-mail address (mail units) EQ 301st reminder email Fax Xno fax unless requested Mail Xno mail unless requested 2nd Follow up attempt date: Phone 60only for those without e-mail address (mail units) EQ Every 2 nd week2nd reminder email Fax Xno fax unless requested Mail Xno mail unless requested 3rd Follow up attempt date: Phone 90only for those without e-mail address (mail units) EQ Every 2 nd week2nd reminder email Fax Xno fax unless requested Mail Xno mail unless requested 4th Follow up attempt date: Phone 120only for those without e-mail address (mail units) EQ Every 2 nd week3rd reminder email Fax 120 fax reminder + copy of questionnaire to those without email address Mail 120mail only to those without email address or fax number 5th Follow up attempt date: Phone 140all outstanding EQ Every 2 nd week5th reminder email Fax 140 fax reminder + copy of questionnaire to those without email address Mail 140mail only to those without email address or fax number 6th Follow up attempt date: Phone 160all outstanding units EQ 1056th and final reminder Fax Xno fax Mail Xno mail

16. EQ Experimental Design for the UES Mailout Mar 18 E-invitation Mar 28 T1 (Standard) T2 (E-mails only) T3 (phone attempt) T4 (phone attempt + e-mails) RemindersE-mail only 1 st attempt – phone; 3 next – E-mail 1 st attempt – phone; 3 next – E-mail Reminder schedule 1 st April 26 2 nd May 26 3 rd June 23 4 th July 25 1 st April 26 2 nd May 9 3 rd May 26 4 th June 7 1 st April 26-29 2 nd May 26 3 rd June 23 4 th July 25 1 st April 26-29 2 nd May 9 3 rd May 26 4 th June 7 Phone follow-upYesNo (firm appt only) YesNo (firm appt only) Experiment end date August 1June 20 *July 11 August 1June 20 *July 11 Sample size allocated to the treatment 1125110211481114 20/10/2015 Statistics Canada Statistique Canada 16

17. Results of the experimental design T1T2T3T4 %u April 2613%12.5%15%12% May 924%22%24%22% May 2636%41%37%45% June 744%43%46%49% June 2051%46%53%52% August 166%54%68%58% Sept 1277%76%78%75% Oct 1482%81%82%81% Progressive unweighted return rate (%) for each treatment (2011) T1T2T3T4 3.82.84.33.9 Number of call attempts by survey treatment 17 Statistics Canada Statistique Canada 20/10/2015

18. Highlights  Treatments consisting of 4 e-reminders only resulted in 46+% as of June 20th  At end of collection (Oct 14) the 4 treatments had very similar response rates  T2 showed the lowest attempts and lowest TPU  Limited worth conducting telephone NRFU for first few months; still reach similar final response rates with fewer phone attempts 20/10/2015 Statistics Canada Statistique Canada 18

19. Access Code Letter  Piloted for UES 2012 collection  Conducted pre-contact for birth cases in sample  Unable to contact cases = mailed an access code letter providing hyperlink to an EQ  More difficult cases since unable to reach during pre-contact  After 30 days, telephone NRFU 20/10/2015 Statistics Canada Statistique Canada 19

20. Access Code Letters: Results Status Access code letter # units (total) Resp. (EQ)Resp. (Mail) Non-Resp.Out of Scope In Progress Total 765150 (20%)39 (5%)52 (7%)74 (9%)450 (60%) Collection status on July 16, 2012 for units that received an access code letter 20 Statistics Canada Statistique Canada 20/10/2015

21. The Future  Successful in achieving its objectives  Mission critical monthly surveys  UES (60 surveys) + LFS  Add new functionalities to EQ platform  Respondent EQ = Interviewer EQ  Assess respondent’s level of tolerance to edits  Streamline processes and operations  Monitor quality, response rates, costs  Research 20/10/2015 Statistics Canada Statistique Canada 21

22. 20/10/2015 Statistics Canada Statistique Canada 22 milana.karaganis@statcan.gc.ca marc.st-denis@statcan.gc.ca