Electronic Questionnaire Collection at Statistics Canada Milana Karaganis Marc St-Denis

Outline  Electronic Questionnaire project  Electronic Questionnaire for business surveys 20/10/2015 Statistics Canada Statistique Canada 2

Electronic Questionnaire Project  Build generic EQ service to be used by all surveys EQ platform Standard approach to EQ  Create a critical mass of EQ service users 160+ surveys Business surveys and some social surveys  Reduce collection and operations costs  Timeline: April 1, > March 31, /10/2015 Statistics Canada Statistique Canada 3

Electronic Questionnaire Project  E-questionnaire solution Support all EQ applications for all surveys Use Statistics Canada infrastructure Comply with security and confidentiality standards Automate EQ survey application development -> build an engine to render EQ applications based on provided inputs, E-questionnaire Generation System (EQGS) 20/10/2015 Statistics Canada Statistique Canada 4

Electronic Questionnaire Project 20/10/2015 Statistics Canada Statistique Canada 5

Electronic Questionnaire Project  Standard approach to EQ – corporate solution Develop EQ applications according to EQ standards Apply standard look and feel Build functionalities required for all surveys Use standard collection strategies EQ as initial collection mode in sequential mode collection Use standard development process Shorten development cycle 20/10/2015 Statistics Canada Statistique Canada 6

20/10/2015 Statistics Canada Statistique Canada 7

20/10/2015 Statistics Canada Statistique Canada 8

20/10/2015 Statistics Canada Statistique Canada 9 Progress To-Date  EQ solution implemented Change Management process First set of EQ standards released Survey development process created Initial collection strategies established  Deployed 39 EQ applications and realized efficiencies  Overall success: Lessons learned Big cultural shift

EQ Among Business Surveys  Outline Evolution of EQ collection strategies Results observed to date Survey collection strategy guidelines Results of tests 20/10/2015 Statistics Canada Statistique Canada 10

Collection Strategies Among Business Surveys  Moving from conventional collection modes (PAPI & CATI) to EQ (over 100 surveys in 5 yrs)  Evolution of multi-modal collection with EQ as primary mode  Measuring results and improving Respecting data quality, timeliness, budgets  Reviewing collection strategies 20/10/2015 Statistics Canada Statistique Canada 11

Reviewing Collection Strategy Quarterly Services Indicators (QSI) Survey 20/10/2015 Statistics Canada Statistique Canada 12 Services Indicator Survey (quarterly ) (QSI) 2011 Q4 2011Q Mail out / E-invitationOct 3, 2011Mail out / E-invitationJan 4, 2012 e-reminder / faxOct 17 e-reminderJan 18 Phone reminder (all)Oct 24 e-reminderJan 25 Phone reminder (all)Nov 7 e-reminderFeb 8 e-reminder / faxNov 14Phone reminder (all)Feb 15 e-reminder / faxNov 28e-reminderFeb 29 Fax to EQDec 5Phone and fax (all)Mar 7 End of collectionDec 9, 2011End of collectionMar 9, 2012

Q EQMailTotal Final Sample1, %271.7%1,552 Response Rate58.5%50.0%58.0% # of cases in NRFU %2281.5%927 20/10/2015 Statistics Canada Statistique Canada 13 Q EQMailTotal Final Sample1, %503.6%1,401 Response Rate65.6%46.2%65.6% # of cases in NRFU %4182.0% 724 Reviewing Collection Strategy Quarterly Services Indicators (QSI) Survey

EQ Collection Strategy Guidelines  Developing tools to ensure optimal collection strategies (monthly; quarterly; annual) Follow-up schedule geared to multi-mode collection  Standardise approach towards efficiencies  Considering respondent burden, data quality, response rates, costs 20/10/2015 Statistics Canada Statistique Canada 14

DateDays after Mail outType of FU Pre-contact n/a Mail-out / E-invitation send date: 0 Collection Start date: Bounce-backs follow up 2 1 week max to complete then switch to mail for those unable to get correct 1st Follow up attempt date: Phone 30only for those without address (mail units) EQ 301st reminder Fax Xno fax unless requested Mail Xno mail unless requested 2nd Follow up attempt date: Phone 60only for those without address (mail units) EQ Every 2 nd week2nd reminder Fax Xno fax unless requested Mail Xno mail unless requested 3rd Follow up attempt date: Phone 90only for those without address (mail units) EQ Every 2 nd week2nd reminder Fax Xno fax unless requested Mail Xno mail unless requested 4th Follow up attempt date: Phone 120only for those without address (mail units) EQ Every 2 nd week3rd reminder Fax 120 fax reminder + copy of questionnaire to those without address Mail 120mail only to those without address or fax number 5th Follow up attempt date: Phone 140all outstanding EQ Every 2 nd week5th reminder Fax 140 fax reminder + copy of questionnaire to those without address Mail 140mail only to those without address or fax number 6th Follow up attempt date: Phone 160all outstanding units EQ 1056th and final reminder Fax Xno fax Mail Xno mail

EQ Experimental Design for the UES Mailout Mar 18 E-invitation Mar 28 T1 (Standard) T2 ( s only) T3 (phone attempt) T4 (phone attempt + s) Reminders only 1 st attempt – phone; 3 next – 1 st attempt – phone; 3 next – Reminder schedule 1 st April 26 2 nd May 26 3 rd June 23 4 th July 25 1 st April 26 2 nd May 9 3 rd May 26 4 th June 7 1 st April nd May 26 3 rd June 23 4 th July 25 1 st April nd May 9 3 rd May 26 4 th June 7 Phone follow-upYesNo (firm appt only) YesNo (firm appt only) Experiment end date August 1June 20 *July 11 August 1June 20 *July 11 Sample size allocated to the treatment /10/2015 Statistics Canada Statistique Canada 16

Results of the experimental design T1T2T3T4 %u April 2613%12.5%15%12% May 924%22%24%22% May 2636%41%37%45% June 744%43%46%49% June 2051%46%53%52% August 166%54%68%58% Sept 1277%76%78%75% Oct 1482%81%82%81% Progressive unweighted return rate (%) for each treatment (2011) T1T2T3T Number of call attempts by survey treatment 17 Statistics Canada Statistique Canada 20/10/2015

Highlights  Treatments consisting of 4 e-reminders only resulted in 46+% as of June 20th  At end of collection (Oct 14) the 4 treatments had very similar response rates  T2 showed the lowest attempts and lowest TPU  Limited worth conducting telephone NRFU for first few months; still reach similar final response rates with fewer phone attempts 20/10/2015 Statistics Canada Statistique Canada 18

Access Code Letter  Piloted for UES 2012 collection  Conducted pre-contact for birth cases in sample  Unable to contact cases = mailed an access code letter providing hyperlink to an EQ  More difficult cases since unable to reach during pre-contact  After 30 days, telephone NRFU 20/10/2015 Statistics Canada Statistique Canada 19

Access Code Letters: Results Status Access code letter # units (total) Resp. (EQ)Resp. (Mail) Non-Resp.Out of Scope In Progress Total (20%)39 (5%)52 (7%)74 (9%)450 (60%) Collection status on July 16, 2012 for units that received an access code letter 20 Statistics Canada Statistique Canada 20/10/2015

The Future  Successful in achieving its objectives  Mission critical monthly surveys  UES (60 surveys) + LFS  Add new functionalities to EQ platform  Respondent EQ = Interviewer EQ  Assess respondent’s level of tolerance to edits  Streamline processes and operations  Monitor quality, response rates, costs  Research 20/10/2015 Statistics Canada Statistique Canada 21

20/10/2015 Statistics Canada Statistique Canada 22