1. Do We Still Need Probability Sampling in Surveys? Robert M. Groves University of Michigan and Joint Program in Survey Methodology, USA

2. Outline The total survey error paradigm in scientific surveys The decline in survey participation The rise of internet panels The “second era” of internet panels So... do we need probability sampling?

3. Outline The total survey error paradigm in scientific surveys The decline in survey participation The rise of internet panels The “second era” of internet panels So... do we need probability sampling?

4. The Ingredients of Scientific Surveys A target population A sampling frame A sample design and selection A set of target constructs A measurement process Statistical estimation

5. Deming (1944) “On Errors in Surveys” American Sociological Review! First listing of sources of problems, beyond sampling, facing surveys

7. Comments on Deming (1944) Includes nonresponse, sampling, interviewer effects, mode effects, various other measurement errors, and processing errors Includes nonstatistical notions (auspices) Includes estimation step errors (wrong weighting) Omits coverage errors “total survey error” not used as a term

8. Sampling Text Treatment of Total Survey Error Kish, Survey Sampling, 1965 –65 of 643 pages on various errors, with specified relationship among errors –Graphic on biases

9. Sampling Biases Frame biases “Consistent” Sampling Bias Constant Statistical Bias Nonsampling Biases Noncoverage NonresponseNonobservation Field: data collection Office: processing Observation

10. Total Survey Error (1979) Anderson, Kasper, Frankel, and Associates Empirical studies on nonresponse, measurement, and processing errors for health survey data Initial total survey error framework in more elaborated nested structure

11. Total Error Variable Error Sampling Nonsampling Field Processing Bias Nonsampling Observation Field Processing Sampling Frame Consistent Nonobservation Noncoverage Nonresponse

12. Survey Errors and Survey Costs (1989), Groves Attempts conceptual linkages between total survey error framework and –psychometric true score theories –econometric measurement error and selection bias notions Ignores processing error Highest conceptual break on variance vs. bias Second conceptual break on errors of nonobservation vs. errors of observation

13. CoverageNonresponseSamplingInterviewerRespondentInstrumentMode CoverageNonresponseSamplingInterviewerRespondentInstrumentMode Errors of Nonobservation Observational Errors Bias Errors of Nonobservation Observational Errors Variance Mean Square Error construct validity theoretical validity empirical validity reliability criterion validity - predictive validity - concurrent validity

14. Nonsampling Error in Surveys (1992), Lessler and Kalsbeek Evokes “total survey design” more than total survey error Omits processing error

15. Components of ErrorTopics Frame errors Missing elements Nonpopulation elements Unrecognized multiplicities Improper use of clustered frames Sampling errors Nonresponse errors Deterministic vs. stochastic view of nonresponse Unit nonresponse Item nonresponse Measurement errors Error models of numeric and categorical data Studies with and without special data collections

16. Introduction to Survey Quality, (2003), Biemer and Lyberg Major division of sampling and nonsampling error Adds “specification error” (a la “construct validity”) Formally discusses process quality Discusses “fitness for use” as quality definition

17. Sources of ErrorTypes of Error Specification error Concepts Objectives Data element Frame error Omissions Erroneous inclusions Duplications Nonresponse error Whole unit Within unit Item Incomplete Information Measurement error Information system Setting Mode of data collection Respondent Interview Instrument Processing error Editing Data entry Coding Weighting Tabulation

18. Survey Methodology, (2004) Groves, Fowler, Couper, Lepkowski, Singer, Tourangeau Notes twin inferential processes in surveys –from a datum reported to the given construct of a sampled unit –from estimate based on respondents to the target population parameter Links inferential steps to error sources

19. Construct Inferential Population Measurement Response Target Population Sampling Frame Sample Validity Measurement Error Coverage Error Sampling Error MeasurementRepresentation Respondents Nonresponse Error Edited Data Processing Error Survey Statistic The Total Survey Error Paradigm

20. Summary of the Evolution of “Total Survey Error” Roots in cautioning against sole attention to sampling error Framework contains statistical and nonstatistical notions Most statistical attention on variance components, most on measurement error variance Late 1970’s attention to “total survey design” 1980’s-1990’s attempt to import psychometric notions Key omissions in research

21. 5 Myths of Survey Practice that TSE Debunks 1.“Nonresponse rates are everything” 2.“Nonresponse rates don’t matter” 3.Give as many cases to the good interviewers as they can work 4.Postsurvey adjustments eliminate nonresponse error 5.Usual standard errors reflect all sources of instability in estimates (measurement error variance, interviewer variance, etc.)

22. Outline The total survey error paradigm in scientific surveys The decline in survey participation The rise of internet panels The “second era” of internet panels So... do we need probability sampling?

23. Response Rates In most rich countries response rates on household and organizational surveys are declining deLeeuw and deHeer (2002) model a 2 percentage point decline per year Probability sampling inference is unbiased from nonresponse with 100% response rate

24. Recent studies challenge a simple link between response rates and nonresponse error Reading Keeter et al. (2000), Curtin et al. (2000), Merkle and Edelman (2002) suggests response rates don’t matter Standard practice urges maximizing response rates What’s a practitioner to do?

25. Mismatches between Statistical Expressions for Nonresponse Error and Practice

26. What does the Stochastic View of Response Propensity Imply? Key issue is whether the influences on survey participation are shared with the influences on the survey variables Increased nonresponse rates do not necessarily imply increased nonresponse error Hence, investigations are necessary to discover whether the estimates of interest might be subject to nonresponse errors

27. Assembly of Prior Studies of Nonresponse Bias Search of peer-reviewed and other publications 47 articles reporting 59 studies About 959 separate estimates (566 percentages) –mean nonresponse rate is 36% –mean bias is 8% of the full sample estimate We treat this as 959 observations, weighted by sample sizes, multiply-imputed for item missing data, standard errors reflecting clustering into 59 studies and imputation variance

28. Percentage Absolute Relative Bias

29. Percentage Absolute Relative Nonresponse Bias by Nonresponse Rate for 959 Estimates from 59 Studies

30. 30 1. Nonresponse Bias Happens

31. 31 2. Large Variation in Nonresponse Bias Across Estimates Within the Same Survey, or

32. 32 3. The Nonresponse Rate of a Survey is a Poor Predictor of the Bias of its Various Estimates (Naïve OLS, R 2 =.04)

33. Conclusions It’s not that nonresponse error doesn’t exist It’s that nonresponse rates aren’t good predictors of nonresponse error We need auxiliary variables to help us gauge nonresponse error

34. A Practical Question “What attraction does a probability sample have for representing a target population if its nonresponse rate is very high and its respondent count is lower than equally- costly nonprobability surveys?”

35. Outline The total survey error paradigm in scientific surveys The decline in survey participation The rise of internet panels The “second era” of internet panels So... do we need probability sampling?

36. A “Solution” to Response Rate Woes Web surveys offer a very different cost structure than telephone and face-to-face surveys –Almost all fixed costs –Very fast data collection But there is no sampling frame –Often probability sampling from large volunteer groups Internet access varies across and within countries

37. Access/Volunteer Internet Panels Massive change in US commercial survey practice, moving from telephone and mail paper questionnaires to web surveys Survey Sampling, a major supplier of telephone samples over the past two decades now reports that 80% of their business is web panel samples Some businesses do only web survey measurement

38. The Method Recruitment of email ID’s from internet users –At survey organization’s web site –Through pop-ups or banners on others’ sites –Through third party vendors A June 15, 2008, Google search of “make money doing surveys” yields 19,300 hits –“make $10 in 5 minutes” www.SurveyMonster.com

40. 40 There is a new industry –Greenfield Online –Survey Sampling –e-Rewards –Lightspeed –ePocrates –Knowledge Networks –Private company panels –Proprietary panels Inside Research, 2007 Baker, 2008

41. Reward Systems Vary Payment per survey Points per survey, yielding eligibility for rewards Points for sweepstakes

42. Adjustment in Estimation Estimation usually involves adjustment to some population totals Some firms have propensity model-based adjustments –“proprietary estimation systems” abound

43. Outline The total survey error paradigm in scientific surveys The decline in survey participation The rise of internet panels The “second era” of internet panels So... do we need probability sampling?

44. September, 2007, Respondent Quality Summit Head of Proctor and Gamble market research 1.Cites Comscore: 0.25% of internet users responsible for 30% of responses to internet panels 2.Cites average number of panel memberships of respondents of 5-8 3.Presents examples of failure to predict behaviors

45. 45 The number of surveys taken matters. Coen et al., 2005 in Baker, 2008

46. 46 The Practical Indicators of “Quality” Cheating on qualifying questions Internal inconsistencies Overly fast completion “Straightlining” in grids Gibberish or duplicated open end responses Failure of “verification” items in grids Selection of bogus or low-probability answers Non-comparability of results with non-panel sample Baker, 2008

47. 47 Panel response rates are in decline as panelists do more surveys. MSI, 2005 in Baker, 2008

48. Where are we now? An industry in turmoil Active study of correlates of low quality conducted by sophisticated clients Professional associations attempting to define quality indicators

49. Outline The total survey error paradigm in scientific surveys The decline in survey participation The rise of internet panels The “second era” of internet panels So... do we need probability sampling?

50. Access Panels and Inference Access panels have conjoined frame development and sample selection Without documentation of the frame development, assessment of coverage properties are not tractable Many use probability sampling from the volunteer set, but ignore this in estimation

51. A Better Question Not “do we still need probability sampling?” but “can we develop good sampling frames with rich auxiliary variables?”

52. Target Population Sampling Frame Sample Respondents Model- assisted Randomization theory Model- assisted Target Population Sampling Frame Sample Respondents Model- assisted ?

53. The Value of Probability Sampling From Well-defined Frames Randomization theory is the powerful linking tool between the sample and the frame Models of nonresponse adjustment are enhanced by auxiliary variables measured on respondents and nonrespondents

54. The Role of Probability Sampling in this Context Probability sampling has low marginal costs within a defined sampling frame Probability sampling offers stratification benefits A sampling frame with rich auxiliary variables can improve stratification effects Access panels should strive for well-defined frame development

55. Speculation As adjustment for nonresponse becomes more important, –Richness of auxiliary variables is primary –Coverage of population becomes relatively less important Hence, frame data and field observations on nonrespondents and respondents are valued

56. Outline The total survey error paradigm in scientific surveys The decline in survey participation The rise of internet panels The “second era” of internet panels So... do we need probability sampling?