18/08/2015 Statistics Canada Statistique Canada Responsive Collection Design (RCD) for CATI Surveys and Total Survey Error (TSE) François Laflamme International Total Survey Error Workshop (ITSEW) Quebec, June 2011

18/08/2015 Statistics Canada Statistique Canada 2  Introduction  RCD surveys  RCD strategy  Active management for RCD  Monitoring and decision making tools  Highlights and lessons learned  RCD and TSE  Next Steps Outline

18/08/2015 Statistics Canada Statistique Canada 3 Introduction  Responsive Collection Design (RCD) is an adaptive approach that uses the information available prior and during data collection to adjust collection strategy for the remaining in-progress cases  Trade-off between quality, cost, productivity, responding potential of in-progress cases, survey mode and interaction between surveys

18/08/2015 Statistics Canada Statistique Canada 4 RCD Surveys  Two experiment surveys with RD and control groups  Households and the Environment Survey (HES 2009) ●Dwelling survey with a cross-sectional design ●Canadian Community Health Survey (CCHS 2009) sampling frame  Survey of Labour and Income Dynamics (SLID 2010) ●Longitudinal survey (Complex survey design)  One full RCD survey - SLID 2011  Embedded experiment for the first call  Objectives: proof of concept, improve efficiency and quality

18/08/2015 Statistics Canada Statistique Canada 5 1)Planning phase  Analysis of previous data collection cycle  Data collection phases and strategies ●RCD objectives, staffing plans and response propensity model for each survey  Sample validation  Active management tools and reports ●New key indicators and communication plan  Control group to assess RCD impact RCD Strategy

18/08/2015 Statistics Canada Statistique Canada 6 2)Initial data collection phase  Use strategic improvement opportunities previously identified ●New time slice strategy and intermediate cap  More likely to collect easy cases  Monitor key indicators to identify start of RCD Phase 1 ●Response rate, productivity, cost (proportion of budget spent) and responding potential of in-progress cases ●By Regional Office (RO) RCD Strategy (cont’d)

18/08/2015 Statistics Canada Statistique Canada 7 3)RCD Phase 1 - Daily overnight job  Categorize and prioritize cases to improve overall response rates ●Probability of completion (propensity) - logistic regression model (sampling frame and sequence of calls information)  Monitor key indicators to identify start of RCD Phase 2 ●Representativity indicator and previous key indicators ●By Regional Office (RO) 4)RCD Phase 2 - Daily overnight job  Prioritize cases to improve sample representativity ●Priority to domains of interest with lower response rates RCD Strategy (cont’d)

18/08/2015 Statistics Canada Statistique Canada 8 RCD Strategy for HES and SLID 1) For SLID 2010, a group called “High probability-Tracing” was used during RD phase 1 2) For SLID 2011, another group called “High probability-Refusal” was added during RD phase 1 3) The intermediate and global caps on calls were (20, 25) for HES and (30, 40) for SLID

18/08/2015 Statistics Canada Statistique Canada 9 Active Management for RCD  Set of plans and tools to manage data collection while in progress  By Regional Office (RO)  Paradata and data sources used  Blaise Transaction file (BTH) (i.e. calls and contact information), interviewer payroll hours, budget and target figures, previous and current collection cycle information, response propensity model results  Key indicators  Used to identify when to start RCD Phase 1 and Phase 2  No survey estimates monitoring so far

18/08/2015 Statistics Canada Statistique Canada 10 When to initiate RCD Phases?  Decision based on survey progress in terms of response rate, productivity, proportion of budget spent (cost) and responding potential of in-progress sample

18/08/2015 Statistics Canada Statistique Canada 11 18/08/2015 Statistics Canada Statistique Canada 11 RCD Dashboard - Example for RCD Phase 1  Dashboards are used to identify when to start both RCD phases to facilitate interpretation and objective decision-making  RCD phase 1: 6 conditions, RCD phase 2: 7 conditions  Yellow and red lights signal when many conditions are met

18/08/2015 Statistics Canada Statistique Canada 12 When to initiate RCD Phase 2?  Similar dashboard than RCD Phase 1 is used (7 conditions)  Condition 7: average response rate increase over the last 5 days  Representativity indicator is also used  Representativity indicator is a measure of variability of response rates between domains of interest  National versus regional objectives

18/08/2015 Statistics Canada Statistique Canada 13 Other Active Management Tools for RCD - Examples  Interviewing progress, results and effort  Response and resolved rates, tracing and refusal conversion results and effort, refusal at the first contact, refusal and no contact rates  In-progress cases  Distribution of calls (for cap on calls monitoring), distribution of cases by Blaise group, browser use, intensive tracing, cases with high propensity but few calls  System time (effort)  By period of day, phase and group (RCD, CG)  Expected distribution by Blaise group between phases  Useful for staff planning  Several other ad hoc tools  Used to identify problems or emerging issues

18/08/2015 Statistics Canada Statistique Canada 14 18/08/2015 Statistics Canada Statistique Canada 14 Active Management Challenges  Large amount of information and reports available  In the past not enough info, currently too much  Need to concentrate on major issues (not on good to know info) ●Can spend a lot of time on something not broken ●Some analysis can wait at the end of collection  Analysis and communication  Often require an extra analytical step before the information is communicated ●Reports are not enough ●Real challenge is to analyse, summarize and communicate ●In the case of RCD, only the main reports were distributed ●Other reports were only used when required

18/08/2015 Statistics Canada Statistique Canada 15 Highlights and Lessons Learned  Higher overall response rate when RCD is used  Compared to previous survey cycle  RD group achieved same response rate with less effort (~2%)  Sample representativity generally improved  High probability and priority groups had positive impact  SLID 2011  Better response rate and contact rate (over 2%) for cases for which the first call was forced to be in the same time slice of previous interview  Pilot test demonstrated the technical feasibility of RCD  Active Management and communication  Essential for any RCD  Need timely and accessible paradata  RCD is not a “magic” solution  Need to be used in conjunction with other initiatives

18/08/2015 Statistics Canada Statistique Canada 16 RCD and TSE  RCD objectives in terms of TSE  Reduce variance and non-response bias  Currently, more likely the variance  But might depend on the non-response adjustment strategy  Did RCD introduce a potential non-response bias?  Active Monitoring of the non-response bias requires survey estimates during data collection - but can be done after  Did RCD change the responding propensity of cases?  Soft treatments - More meaningful grouping of cases ●No incentives, no sub-sampling of respondents

18/08/2015 Statistics Canada Statistique Canada 17 Next Steps  Implement RCD for other CATI surveys  Developed with a research rather than a production perspective  Improve current RCD strategy  Propensity models  Gradually phase-in of RCD phase1  Integrate new conditions for decision making ●Representativity, survey estimates, non-response bias  Include cost-efficiency objective  RCD for CAPI surveys  Potential benefits of RCD to improve cost-efficiency  Feasible study : communication flow, potential actions, concurrent surveys, operational and technical constraints.  RCD for multi-mode surveys  RCD theoretical framework

18/08/2015 Statistics Canada Statistique Canada 18 For more information, please contact Pour plus d’information, veuillez contacter François Laflamme