1. An Active Collection using Intermediate Estimates to Manage Follow-Up of Non-Response and Measurement Errors
Jeannine Claveau, Serge Godbout and Claude Turmelle
Statistics Canada
International Total Survey Error Workshop
Québec, June 20, 2011

2. Outline Introduction Quality Indicators (QI)
Measure of Impact (MI) Scores
Future Work 2
Statistics Canada • Statistique Canada
Statistics Canada • Statistique Canada
20/11/2018
20/11/2018

3. Unified Enterprise Surveys (UES)
UES consists of 58 annual business surveys integrated in terms of content, collection and data processing
Collect information on enterprise financial variables
Collection period: February to early October
Telephone pre-contact used for new units in the sample
Mail questionnaires for initial data gathering
Telephone follow-up conducted to collect data from non- respondent and to resolve failed edits 3
Statistics Canada • Statistique Canada
Statistics Canada • Statistique Canada
20/11/2018
20/11/2018

4. Unified Enterprise Surveys (UES)
Score function is used to prioritize telephone follow-up for non-response
Score based on weighted sampling revenue
For most of the UES surveys: no score function used for failed edits follow-up
Collection Processing System: Blaise
Paradata in Blaise Transaction History files 4
Statistics Canada • Statistique Canada
Statistics Canada • Statistique Canada
20/11/2018
20/11/2018

5. Integrated Business Statistics Program (IBSP)
IBSP is under development to redesign and expand UES to integrate other enterprise surveys and sub-annual surveys
Goal:
Reduce operating costs
Enhance quality assurance
IBSP will integrate 120 surveys by 2016 (phase 1: 2014)
Electronic questionnaire (electronic data collection) will be the principal collection mode offered to enterprise 5
Statistics Canada • Statistique Canada
Statistics Canada • Statistique Canada
20/11/2018
20/11/2018

6. Current UES – Processing Model
Collection, processing and analysis are run sequentially
Estimates produced at very end only
Collection ends at set date
Sampling
Collection
Processing
Analysis
Dissemination
Statistics Canada • Statistique Canada
20/11/2018
20/11/2018 6

7. Statistics Canada • Statistique Canada
IBSP – Estimates Model
Collection, processing and analysis will be run in parallel
Estimates will be produced and re-run periodically
Collection could end earlier when pre-specified quality target has been met
Collection
Sampling
Dissemination
Processing
Analysis 7
Statistics Canada • Statistique Canada
20/11/2018
20/11/2018 7

8. Active Collection
Role: Manage follow-up of non-response and measurement errors (failed edits)
Responsive Design (Laflamme and Karaganis, 2010) or Dynamic adaptative approach (Schouten, Calinescu and Luiten, 2011) that uses data available during collection to modify collection strategy
Estimates and quality indicators will be produced periodically throughout collection: e.g. monthly basis
Then scores measuring impact on estimates and on quality indicators are calculated to allocate and prioritize telephone follow-up 8
Statistics Canada • Statistique Canada
Statistics Canada • Statistique Canada
20/11/2018
20/11/2018

9. Basic Collection Strategy
Initial Sample S
Production of Intermediate Estimates
Successive Designs d0 d1 d2
di-2
di-1
NR1
NR2
NR3
NRi-1
NRi
Observed NR and Response R1 R2 R3
Ri-1 Ri
Statistics Canada • Statistique Canada
20/11/2018

10. Parameter and Estimator
Variables of interest: Set of I key variables
Parameters of interest:
Stratified expansion estimators:
Sampling variances: (under a stratified Bernoulli design):
Where i, k and h identify respectively the I variables, the Nh units and the H strata Nh = stratum population size ph = unit sampling probability within stratum nh = the stratum sample size 10
Statistics Canada • Statistique Canada
Statistics Canada • Statistique Canada
20/11/2018
20/11/2018

11. Non-Response Response propensity model: Estimation:
Auxiliary data and paradata would be used to estimate response propensities
Estimation:
In case on non-response, we will either use imputation or reweighting to account for missing data
Response propensities could be used to form imputation or reweighting homogeneous classes for reducing the non- response bias (Haziza and Beaumont, 2007)
Stratified expansion estimators: 11
Statistics Canada • Statistique Canada
Statistics Canada • Statistique Canada
20/11/2018
20/11/2018

12. Quality Indicators (QI)
Role:
Monitor collection progress
Help to allocate and prioritize collection efforts
Can be item-based
Specific to a variable of interest
Variance, CV
Item response rate of a variable of interest
Bias:
MSE: 12
Statistics Canada • Statistique Canada
Statistics Canada • Statistique Canada
20/11/2018
20/11/2018

13. Quality Indicators (QI)
Can be covariate-based
Derived from statistics on the estimated response propensities given the covariates X
Independent from the variables of interest
Examples of covariate-based QIs (Schouten, 2011) :
Mean response propensity:
R-indicator:
Standardized Maximal Bias:
Standardized Maximal Variance:
Standardized Maximal MSE: 13
Statistics Canada • Statistique Canada
Statistics Canada • Statistique Canada
20/11/2018
20/11/2018

14. Measure of Impact (MI) Scores
Types of Scores
Common types: Edit-related and estimate-related score functions
Example: Predicted difference in estimates (Hedlin, 2008)
Proposal: Generalize the MI Score to include quality-related score functions
For an estimated parameter (estimate or quality indicator)
Definition:
Where is the estimated parameter after changing reported values and/or covariates of unit k respectively to and/or and is a scaling factor 14
Statistics Canada • Statistique Canada
Statistics Canada • Statistique Canada
20/11/2018
20/11/2018

15. Measure of Impact (MI) Scores
MI Score for an estimated total:
Requires predicted values to compare to reported values
Proposal: Use imputation to obtain predicted values
Used to prioritize units for failed edit follow-up 15
Statistics Canada • Statistique Canada
Statistics Canada • Statistique Canada
20/11/2018
20/11/2018

16. Measure of Impact (MI) Scores
MI Score for item-based quality indicators
MI Score for estimated sampling variance for expansion estimators
Specific to a variable of interest
Also use imputation to obtain predicted values
Linked directly to quality of output estimates
Prioritize units for failed edit follow-up 16
Statistics Canada • Statistique Canada
Statistics Canada • Statistique Canada
20/11/2018
20/11/2018

17. Measure of Impact (MI) Scores
MI Score for item-based quality indicator
MI Score for covariate-based quality indicator
Used to prioritize units for both non-response and failed edit follow- up 17
Statistics Canada • Statistique Canada
Statistics Canada • Statistique Canada
20/11/2018
20/11/2018

18. Active Collection Management
A large number of variables to monitor
Monitoring all of them will be a challenge
Not all equally important
Identify a limited number of key variables
For each key variable
Quality monitored using item-based QIs and MI Scores
For the non-key variables
Quality controlled using covariate-based QIs 18
Statistics Canada • Statistique Canada
Statistics Canada • Statistique Canada
20/11/2018
20/11/2018

19. Active Collection Management
MI scores for each estimated parameter and quality indicator are considered local scores
In order to prioritize units for telephone follow-up, global score per unit is needed
Derive global MI Score (Hedlin, 2008)
Sum, maximum or Euclidian distance could be used
Some QIs are appropriate for evaluating the impact of non- response and others for the impact of edit failures
Derive one global score for non-response follow-up and one global score for failed edit follow-up 19
Statistics Canada • Statistique Canada
Statistics Canada • Statistique Canada
20/11/2018
20/11/2018

20. Control Quality with Covariated-Based QIs
Goal: Increase the average of the response propensities while improving their homogeneity. 20
Statistics Canada • Statistique Canada
Statistics Canada • Statistique Canada
20/11/2018
20/11/2018

21. Control Quality with Covariated-Based QIs
Goal: Increase the average of the response propensities while improving their homogeneity. 21
Statistics Canada • Statistique Canada
Statistics Canada • Statistique Canada
20/11/2018
20/11/2018

22. Control Quality with Covariated-Based QIs
Goal: Increase the average of the response propensities while improving their homogeneity. 22
Statistics Canada • Statistique Canada
Statistics Canada • Statistique Canada
20/11/2018
20/11/2018

23. Summary Current Approach Proposed Approach Follow-up and editing
A score function with no link with estimates
Prioritization based on frame (static) information
Follow-up and editing for influential units based on estimates and quality
Prioritization based on frame, paradata (dynamic) and estimates
Processing
Results (and quality measures) are known only at the end of the process
Produce results (and quality measures) during collection to manage collection
Cut-off collection
Based on weighted response rate
Based on achieved quality of estimates 23
Statistics Canada • Statistique Canada
Statistics Canada • Statistique Canada
20/11/2018
20/11/2018 23

24. Quality Indicators (QI) Measure of Impact (MI) Scores
Summary
Quality Indicators (QI)
Measure of Impact (MI) Scores
Quality (accuracy) specific to a domain and an estimate
Impact of a unit on an estimate or on a quality indicator
Monitor collection and analysis progress
Allocate and prioritize collection and analysis efforts
Proactively identify problems
Assess quality of produced estimates
Close active collection
Non-response and failed-edit follow-up 24
Statistics Canada • Statistique Canada
Statistics Canada • Statistique Canada
20/11/2018
20/11/2018

25. Summary Covariate-based QIs Item-based QIs
Independent of survey variables
Related to survey variables
Used to all variables
Used with MI Scores to monitor specified key variables
Mean response propensity
R-indicator
Standardized Maximal Bias, Variance and MSE
Other…
Item response rate
Variance, CV
Bias, MSE 25
Statistics Canada • Statistique Canada
Statistics Canada • Statistique Canada
20/11/2018
20/11/2018

26. Non-response follow-up
Summary
Non-response follow-up
MI Scores
Failed Edit follow-up
One global score
Item response rate
Mean response propensity
R-indicator
Variance, CV
Item-based MSE
Standardized Maximal Bias, Variance and MSE
Estimated total
Estimated sampling variance 26
Statistics Canada • Statistique Canada
Statistics Canada • Statistique Canada
20/11/2018
20/11/2018

27. Future Work Methodology development
Response propensity model: development of a model based on data and paradata
Item-based and covariate-based QIs
Validation of the proposed strategy
Conduct simulation studies and develop prototypes using current UES environment
Summer 2011 prototype: response rates, imputation rate, CV and MI scores
Next prototype: Other local and global MI scores and QIs 27
Statistics Canada • Statistique Canada
Statistics Canada • Statistique Canada
20/11/2018
20/11/2018

28. Discussion
What quality indicators are appropriate to measure the risks of potential bias in the estimates?
What is the best way to use quality indicator (e.g. R-indicator) to monitor collection of highly skewed business surveys?
The proposed approach obviously affects the response propensities throughout collection. Although we can adjust the estimator later on to take this into account, is it something we should move away from? Or should we take advantage of it?
In the proposed approach, are there any additional aspects that should be considered? 28
Statistics Canada • Statistique Canada
Statistics Canada • Statistique Canada
20/11/2018
20/11/2018

29. Statistics Canada • Statistique Canada
Merci / Thank You
For more information,  Pour plus d’information,
please contact: veuillez contacter :
Jeannine Claveau
Serge Godbout
Claude Turmelle
Statistics Canada • Statistique Canada
20/11/2018