1. Investigating the Potential of Using Non-Probability Samples Debbie Cooper, ONS

2. Summary Project aims What constitutes non-probability sampling? Advantages of probability sampling Growing interest in non-probability sampling Types of non-probability sampling Key challenges Overcoming challenges When is the use of non-probability sampling justified? Recommendations

3. Project aims 1.Provide a concise review of the types of non-probability samples 2.Highlight the key challenges associated with non-probability sampling 3.Increase awareness of techniques available to potentially overcome these challenges 4.Provide guidance to help inform decision- making on whether a non-probability sample is justified

4. What constitutes non-probability sampling? Non-probability sampling has two distinguishing characteristics:  one cannot specify the probability of selection for each unit that will be included in the sample  it is not possible to ensure that every unit in the population has a nonzero probability of inclusion (Frankfort-Nachmias and Nachmias, 1996)

5. Advantages of probability sampling the ability to calculate selection probabilities allows researchers to create design weights which result in an unbiased estimator allows for representativeness as each unit in the target population has a nonzero probability of selection allows for the estimation of sampling variability BUT non-random nonresponse and undercoverage violate the assumptions of probability sampling, giving them a non-probability element

6. Having said that..... Methods developed to deal with coverage and nonresponse issues in probability sampling:  using multiple sampling frames  adjusting weights for nonresponse and, if relevant, attrition  calibrating weights to population totals

7. Why is there a growing interest in non- probability sampling concerns about increasing nonresponse rates high costs associated with probability sampling ease of carrying out web surveys sometimes no other option available (e.g. hidden population)

8. Types of non-probability sampling Convenience/accidental sampling Purposive sampling Sample matching Chain referral methods

9. Convenience/accidental sampling “Convenience sampling is a form of non-probability sampling in which the ease with which potential participants can be located or recruited is the primary consideration.” (Baker et al., 2013) Types of convenience sampling:  mall-intercept sampling  volunteer sampling

10. Purposive sampling Consists of the researcher approaching people who they decide are most appropriate to participate in the study e.g. a sample of experts on a particular topic

11. Sample matching involves selecting a sample that matches a set of population characteristics of interest most common type of sample matching is quota sampling good estimates of the population characteristics used for matching need to be available

12. Chain referral methods Tend to be used for researching rare or hard- to-reach populations. Types of chain referral methods:  Snowball sampling  Respondent-driven sampling (RDS)

13. Non-probability sampling: key challenges 1. Greater likelihood of selection bias 2. Impossible to utilise unbiased estimators and associated quality measures

14. 1. Selection bias “The error introduced when the study population does not represent the target population” (Delgado-Rodriguez and Llorca, 2004) Some causes:  undercoverage  volunteer bias  interviewer/researcher unconscious bias

15. 2. Unbiased estimators and associated quality measures Standard practice in official statistics is to use probability sampling and design-based estimation Advantages:  resulting estimator is unbiased  sampling variability can be estimated directly These advantages are lost when non-probability samples are used.

16. Overcoming challenges at: 1.Sampling stage 2.Weighting and estimation stage

17. 1. Overcoming challenges at sampling stage Main challenge at this stage is obtaining a representative sample. Two popular non-probability sampling strategies developed to obtain a representative sample are:  Sample matching (quota sampling)  Respondent-Driven Sampling

18. Sample matching (quota sampling) Aim is to obtain responses from a specific number of units that match the target population Quota sampling:  interviewers asked to interview a certain number of people with particular characteristics  final sample ‘mirrors’ the target population in terms of these characteristics

19. Sample matching problems Choice of who to interview still in the hands of the interviewer (unconscious bias) Undercoverage Nonresponse Therefore, recommended to use sample matching along with weighting

20. Respondent-driven sampling (RDS) Used for hidden target populations Two distinct phases: 1.Initial sample (the seeds at Wave 0) selected using convenience sampling 2.Rest of the sample is selected by following links from previous respondents

21. 2. Overcoming challenges at weighting and estimation stage Different methods developed for use with the various sampling techniques.  Estimators and quality measures when using RDS  Using weighting  Additional quality measures

22. Estimation and quality measures for use with RDS “For many years, researchers thought it was impossible to make unbiased estimates from this type of sample. However, it was recently shown that if certain conditions are met and if the appropriate procedures are used, then the prevalence estimates from respondent- driven sampling are asymptotically unbiased.” (Salganik, 2006)

23. Estimation and quality measures for use with RDS Heckathorn (2011) – describes various estimators developed for use with RDS  Caution: these estimators require a number of assumptions to be made and biased estimates may result if assumptions are not met Chow and Thompson (2003) proposed a Bayesian approach for estimation for use with link-tracing designs

24. Estimation and quality measures for use with RDS Bootstrap method to construct confidence intervals around estimates (Salganik, 2006) Recommends sample size twice as large as that required under SRS Interval estimates when using Bayesian approach proposed by Chow and Thompson (2003)

25. Using weighting with non-probability sampling Propensity-score adjustments (PSA) to approximate design-based approach Valliant and Dever (2011) approach:  construct pseudo design weights  use covariates from reference survey to adjust these design weights  Caution: Lee (2006) found that although PSA tends to reduce nonresponse bias it seems to increase variance  Caution: Lee (2006) recommends that covariates highly related to study outcomes should be used in the PSA

26. Additional quality measures Credibility Intervals – popular with opt-in panels Participation rates Response rate = no. of respondents total no. of eligible units Participation rate = no. with useable response total no. of initial invitations

27. Quality measures for non-probability sampling Essential for researchers to report on the quality of their estimates Currently no widely acceptable framework for assessing quality of estimates from non- probability samples – development needed Important to use different terminology to that used for probability sampling

28. When is use of non-probability sampling justified? Fitness for purpose ‘Modellers’ vs ‘Describers’ No single correct approach Decision boils down to desired outcomes and resources Communicate clearly the quality of estimates and issues/limitations

29. Recommendations Fitness for purpose should be used to drive survey design Non-probability sampling does not necessarily equate to lack of quality Transparency is essential

30. Questions? Debbie.cooper@ons.gsi.gov.uk