TOPIC - Page 1 Representativity Indicators for Survey Quality R-indicators and fieldwork monitoring Koen Beullens & Geert Loosveldt K.U.Leuven.

Slides:



Advertisements
Similar presentations
Multiple Indicator Cluster Surveys MICS3 Regional Training Workshop Household Information Panel.
Advertisements

MICS4 Survey Design Workshop Multiple Indicator Cluster Surveys Survey Design Workshop Household Questionnaire: Household Information Panel and Household.
Standardized Scales.
Survey Methodology Nonresponse EPID 626 Lecture 6.
The estimation strategy of the National Household Survey (NHS) François Verret, Mike Bankier, Wesley Benjamin & Lisa Hayden Statistics Canada Presentation.
Quality indicators for measuring and enhancing the composition of survey response Q2008 – Special topic session, July 9 Jelke Bethlehem and Barry Schouten.
SAMPLING.
The current status of fisheries stock assessment Mark Maunder Inter-American Tropical Tuna Commission (IATTC) Center for the Advancement of Population.
Chapter 7 Sampling Distributions
Dynamic Treatment Regimes, STAR*D & Voting D. Lizotte, E. Laber & S. Murphy LSU ---- Geaux Tigers! April 2009.
Documentation and survey quality. Introduction.
Dynamic Treatment Regimes, STAR*D & Voting D. Lizotte, E. Laber & S. Murphy Psychiatric Biostatistics Symposium May 2009.
Aaker, Kumar, Day Ninth Edition Instructor’s Presentation Slides
The Excel NORMDIST Function Computes the cumulative probability to the value X Business Statistics: A First Course, 5e © 2009 Prentice-Hall, Inc
SAMPLING Chapter 7. DESIGNING A SAMPLING STRATEGY The major interest in sampling has to do with the generalizability of a research study’s findings Sampling.
REPRESENTATIVE RESPONSE Some Examples ITSEW Peter Lundquist Statistics Sweden.
08/08/2015 Statistics Canada Statistique Canada Paradata Collection Research for Social Surveys at Statistics Canada François Laflamme International Total.
17 June, 2003Sampling TWO-STAGE CLUSTER SAMPLING (WITH QUOTA SAMPLING AT SECOND STAGE)
18/08/2015 Statistics Canada Statistique Canada Responsive Collection Design (RCD) for CATI Surveys and Total Survey Error (TSE) François Laflamme International.
The new HBS Chisinau, 26 October Outline 1.How the HBS changed 2.Assessment of data quality 3.Data comparability 4.Conclusions.
RTI International is a trade name of Research Triangle Institute 3040 Cornwallis Road ■ P.O. Box ■ Research Triangle Park, North Carolina, USA
1. An Overview of the Data Analysis and Probability Standard for School Mathematics? 2.
The Vocabulary of Research. What is Credibility? A researcher’s ability to demonstrate that the study is accurate based on the way the study was conducted.
Response Rates and Results of the Advance Letter Experiment 2004 RRFSS Workshop Toronto, June 23, 2004 David A. Northrup, Renée Elsbett-Koeppen and Andrea.
Joint UNECE/Eurostat Meeting on Population and Housing Censuses (28-30 October 2009) Accuracy evaluation of Nuts level 2 hypercubes with the adoption of.
1 Performance Evaluation of Computer Networks: Part II Objectives r Simulation Modeling r Classification of Simulation Modeling r Discrete-Event Simulation.
Nonresponse issues in ICT surveys Vasja Vehovar, Univerza v Ljubljani, FDV Bled, June 5, 2006.
Sampling Techniques LEARNING OBJECTIVES : After studying this module, participants will be able to : 1. Identify and define the population to be studied.
Fieldwork efforts  Monitoring fieldwork efforts  Monitoring fieldwork efforts: Did interviewers /survey organisations implement fieldwork guidelines.
Chapter 11.1 Measures of Dispersion and Types of Studies.
Nonresponse adjustment in the European Social Survey (ESS) Ineke Stoop (SCP-Netherlands) Jaak Billiet (Uni Leuven-Belgium) Achim Koch (Gesis-Germany)
S-005 Intervention research: True experiments and quasi- experiments.
A Latent Class Call-back Model for Survey Nonresponse Paul P. Biemer RTI International and UNC-CH Michael W. Link Centers for Disease Control and Prevention.
Quality strategies in cross- national surveys The case of the European Social Survey Ineke Stoop.
Sampling is the other method of getting data, along with experimentation. It involves looking at a sample from a population with the hope of making inferences.
LT 4.2 Designing Experiments Thanks to James Jaszczak, American Nicaraguan School.
Brown, Suter, and Churchill Basic Marketing Research (8 th Edition) © 2014 CENGAGE Learning Basic Marketing Research Customer Insights and Managerial Action.
DTC Quantitative Methods Survey Research Design/Sampling (Mostly a hangover from Week 1…) Thursday 17 th January 2013.
Nonresponse Rates and Nonresponse Bias In Surveys Robert M. Groves University of Michigan and Joint Program in Survey Methodology, USA Emilia Peytcheva.
The Challenge of Non- Response in Surveys. The Overall Response Rate The number of complete interviews divided by the number of eligible units in the.
CHAPTER 12 Descriptive, Program Evaluation, and Advanced Methods.
1 Non-Response Non-response is the failure to obtain survey measures on a sample unit Non-response increases the potential for non-response bias. It occurs.
Market research for a start-up. LEARNING OUTCOMES By the end of this lesson I will be able to: –Define and explain market research –Distinguish between.
A Theoretical Framework for Adaptive Collection Designs Jean-François Beaumont, Statistics Canada David Haziza, Université de Montréal International Total.
SAMPLE SELECTION in Earnings Equation Cheti Nicoletti ISER, University of Essex.
Basic Business Statistics, 10e © 2006 Prentice-Hall, Inc.. Chap 7-1 Chapter 7 Sampling Distributions Basic Business Statistics.
FCS681 Types of Research Design: Non Experimental Hira Cho.
Copyright © 2013, 2009, and 2007, Pearson Education, Inc. 1 In an observational study, the researcher observes values of the response variable and explanatory.
Principles of exp. design Control for effects of lurking variables Randomization to keep personal biases or other preferences out of the study Replication.
1 Quality Control for Field Operations. 2 Overview Goal To ensure the quality of survey field work Purpose To detect and deter interviewer errors and.
Collection of Data Jim Bohan
Chapter 20 Classification and Estimation Classification – Feature selection Good feature have four characteristics: –Discrimination. Features.
Representativity Indicators for Survey Quality Programme: Cooperation Theme: Socio-economic sciences and Humanities Activity: Socio-economic and scientific.
1 European Conference on Quality in Official Statistics - Helsinki. Finland 3-6 May 2010 The use of R-indicators in responsive survey design – Some Norwegian.
Journalism 614: Non-Response and Cell Phone Adoption Issues.
Q2010 – special topic session 33 - Page 1 Indicators for representative response Barry Schouten (Statistics Netherlands) Natalie Shlomo and Chris Skinner.
1 Chapter 11 Understanding Randomness. 2 Why Random? What is it about chance outcomes being random that makes random selection seem fair? Two things:
Chapter 10 (3.8) Marketing Research.  What is Marketing Research? Marketing research is the systematic design, collection, analysis, and reporting of.
Purpose of Epi Studies Discover factors associated with diseases, physical conditions and behaviors Identify the causal factors Show the efficacy of intervening.
Part III – Gathering Data
An Active Collection using Intermediate Estimates to Manage Follow-Up of Non-Response and Measurement Errors Jeannine Claveau, Serge Godbout and Claude.
The European Statistical Training Programme (ESTP)
The European Statistical Training Programme (ESTP)
Chapter 12: Other nonresponse correction techniques
Warm Up Imagine you want to conduct a survey of the students at Leland High School to find the most beloved and despised math teacher on campus. Among.
The European Statistical Training Programme (ESTP)
Sampling for Impact Evaluation -theory and application-
Chapter: 9: Propensity scores
New Techniques and Technologies for Statistics 2017  Estimation of Response Propensities and Indicators of Representative Response Using Population-Level.
Chapter 5: The analysis of nonresponse
Presentation transcript:

TOPIC - Page 1 Representativity Indicators for Survey Quality R-indicators and fieldwork monitoring Koen Beullens & Geert Loosveldt K.U.Leuven

TOPIC - Page 2 Outline  Basic ingredients for R-indicators  Basic ingredients for fieldwork monitoring using R-indicators  R-indicators in action: sample unit re-selection for additional fieldwork efforts Three selection strategies Simple example ESS3 - Belgium

TOPIC - Page 3 Basic ingredients for R-indicators  0 – 1 response indicator  Auxiliary variables, usually (non)respondent related  Determine response propensities ρ i  Determine S( ρ) and R-indicator (and maximal absolute bias)  E.g. ESS3 – Belgium Response rate: 0.62 (ineligibles excluded) R-indicator: 0.79 Maximal absolute bias: 0.17 Auxiliary variables include age, gender, type and condition of dwelling and several indicators on municipality level  But: only a result of a process

TOPIC - Page 4 Basic ingredients for fieldwork monitoring using R- indicators  We need to look inside the fieldwork process  Fieldwork process is layered: Making contact Assess eligibility Is target person able / available? Is target person willing to cooperate?  During these fieldwork phases, many decisions have been made by fieldwork management and interviewers (=treatment variables) Selection, training, allocation and remuneration of interviewers Timing and modes of contact attempts Re-selection of sample units for renewed attempts …

TOPIC - Page 5  Re-attempt  Re-assign  ( STOP ) Noncontact Noncooperation (refusal or other noncooperation)  Re-attempt  Re-assign  ( STOP ) Contact Visit Assignment Gross SampleSample FramePopulation Advance information Cooperation or appointment ( STOP ) EligibleIneligible ( STOP ) Example flowchart of fieldwork ESS3 - Belgium

TOPIC - Page 6 Basic ingredients for fieldwork monitoring using R- indicators  Flowchart inspires to deconstruct the fieldwork into basic building stones Sub-processes Treatment variables Necessity of paradata  Fieldwork can be monitored / evaluated How did the combination of treatment variables affect the quality of the obtained sample? During / at the end of process  Inspire future quality improvements During the current fieldwork (adaptive design) For the next survey

TOPIC - Page 7 R-indicators in action: sample unit re-selection for additional fieldwork efforts  Renewed contact attempt require substantial efforts, usually at lower success rates  Not all cases are reissued  Who should be re-selected in order to improve sample quality Random selection among initial nonrespondents Selection of high propensity cases Selection of low propensity cases  Two applications: Simple imaginary example Real application: ESS3 - Belgium

TOPIC - Page 8 A simple example  Suppose a sample of n= cases  Auxiliary variable X (mean = 0, stdev = 1)  Every unit i has been attempted once  Initial response propensities are determined by:  Sample quality after initial attempt: Response rate = 50% R-indicator = 0.77 Maximal absolute bias = 0.23

TOPIC - Page 9 A simple example  Of the nonrespondents, can be reissued  For each nonresponding unit i we have an expectation about the conversion success, based on the initial attempt  We assume:

TOPIC - Page 10 A simple example Expectations about effect of purposive re-selection Propensity selection Response rateR-indicator Low+++ Random++? High+++--

TOPIC - Page 11 A simple example After 50 replications Initial HighRandomLow Response rate  R-indicator  Maximal absolute bias 

TOPIC - Page 12 A simple example  Variety of quality arrangements, depending on the selection strategy.  Low propensity selection  best return on investment Less effort for same quality Better quality for same effort  Conditions Auxiliary variables should be a good approximation of א Prior knowledge of auxiliary variables Caution: low propensity selection can lead to inverse effects

TOPIC - Page 13 Empirical validation: ESS3 - BE  2927 eligible cases  Auxiliary information, known after the first contact attempt: Age, gender, type of dwelling, housing conditions Information on municipality level: average income, population density, percentage of foreigners  Display evolution of sample quality as a function of total number of contact attempts (efforts)  Then simulate according to three selection strategies Propensities determined after first attempt All other fieldwork conditions constant

TOPIC - Page 14

TOPIC - Page 15

TOPIC - Page 16

TOPIC - Page 17

TOPIC - Page 18

TOPIC - Page 19 Empirical validation: ESS3 - BE  Purposive selection can be cost-efficient, when selecting low propensity cases Less efforts or Better quality  Part of the efforts that are saved can be invested in the collection of more auxiliary variables Area information Ask the neighbours Google streetview … Privacy restrictions!!!

TOPIC - Page 20 Empirical validation: ESS3 - BE  It is possible to disentangle contact and cooperation and evaluate both processes separately Contact: until decision to cooperate / refuse Cooperation: decision to cooperate / refuse  Comparing low propensity selection to observed evolution Both processes seem to leave some space for quality improvement Particularly cooperation / refusal

TOPIC - Page 21

TOPIC - Page 22

TOPIC - Page 23 Fieldwork monitoring – concluding remarks  R-indicator facilitates fieldwork monitoring R-indicator is single-value Response propensity is summary of multivariate distribution  R-indicator inspires fieldwork improvement and efficiency May save a lot of redundant efforts Invest efforts in good auxiliary information  Fieldwork monitoring as such is very complex process Requires complex models Requires good paradata