Modeling and Simulation of Survey Collection Using Paradata Presented by: Kristen Couture Co-authored by: Yves Bélanger Elisabeth Neusy

2 Outline  Motivation for Simulating Survey Collection  Details of Simulation  Modeling using Paradata  Preliminary Results  Conclusions  Future Work

3 Motivation  Ultimate goal: make CATI survey collection more efficient  Recent initiatives in the field Experimentation with call attempts and calling priorities Takes time, lack of control, costly, results not always easy to interpret  Need for a controlled environment, where impact of each experiment can be tested prior to collection

4 Microsimulation  What is microsimulation? A modeling technique that operates at the level of individual units, such as persons, households, vehicles, etc.  For us: microsimulation = a "virtual collection" system  Recreates CATI collection environment with Simulation Software (SAS Simulation Studio)  Allows manipulation of parameters in simulated environment

5 Microsimulation  What are the elements of our microsimulation? Cases Queues Interviewers Rules of the Call Scheduler (flows and priorities) Output Call Transaction File

6 Overview of Microsimulation Paradata Model Call OutcomesModel Call Duration Model Parameters Collection Parameters Simulation Model SAS Simulation Studio

7 Modeling using Paradata  Use existing survey data (Blaise Transaction History)  Call Outcome Multinomial logistic regression  Call Duration Create histograms and fit distributions for each of the outcomes  Output Model parameters Estimated parameters from logistic regression model Fitted distribution and parameters Input into simulation model

8 Modeling using Paradata: Call Outcome  Multinomial Logistic Regression Model  Model probability of outcomes (sum of probabilities = 1)  k+1 outcomes  x i = explanatory variables from paradata  p j = probability of outcome j  = parameters from logistic regression model

9 Modeling Call Outcome: An Example  Paradata: Existing RDD survey  5 outcomes: Unresolved, Out of Scope, Refusal, Other Contact, Respondent  7 explanatory variables entered into the model Time of Call: Afternoon, Evening, Weekend Residential Status: Residential phone number Call history: Unresolved, Refusal, Contact  Estimated parameters from model are entered into simulation

10 Time of Call Call History Modeling Call Outcome: An Example  Calculate probability of each outcome p j values

11 Microsimulation Collection Parameters Simulation Model Paradata Model Call OutcomesModel Call Duration Model Parameters

12 Preliminary Results: Two examples  Investigate how collection parameters impact response rates  Two Examples: Example 1 : Different distributions of interviewers throughout the day Example 2: Different distributions of interviewers throughout the day combined with different time slices  Purpose: Demonstrate how users can manipulate collection parameters to test specific collection scenarios Verify that simulation results reflect collection

13 Example 1  Change allocation of interviewers throughout the time periods  3 Time Periods each 4 hours in length  30 interviewers per day for 30 days  What happens to response rates? Time period # of Interviewers Morning (9h-13h) 4 Afternoon (13h-17h) 4 Evening (17h-21h) 22 Fixed Total30 One Possible Scenario

14 Example 1

15 Example 2  Same setup as Example 1  Add time slices: control maximum number of attempts made at different time periods throughout the day  What happens to response rates? Time period # of Interviewers Max # of Attempts Morning42 Afternoon42 Evening2216 Fixed Total3020 One Possible Scenario

16 Example 2  Response Rates

17 Conclusions  Create simple simulation model using paradata that produces results that reflect collection  Able to test different collection parameters to see impact on response rates without spending a lot of money or time  Approach adaptable to all types of CATI surveys

18 Future Work  Improve logistic model by adding more parameters  Add more complicated collection procedures to the model such as interviewer characteristics  Simulate collection with multiple surveys at a time to see impact  Run simulation for a survey to predict outcome and compare with actual results from field

19  For more information,  Pour plus d’information, please contact:veuillez contacter : Kristen Couture Yves Bélanger Elisabeth Neusy