1. A Latent Class Call-back Model for Survey Nonresponse Paul P. Biemer RTI International and UNC-CH Michael W. Link Centers for Disease Control and Prevention

2. Outline Motivation for the study Early cooperator effects (ECE) in the Behavior Risk Factor Surveillance System (BRFSS) Call-back models –Manifest and latent Model extensions Application to the BRFSS Results Summary and conclusions

3. Terms and definitions Cooperators = units that will eventually respond at some request or call-back Non-cooperators (also called hardcore nonrespondents) = units that will not respond to any call-back

4. Terms and definitions (cont’d) Early cooperator = Cooperators that respond at early calls (say, 5 or less) Later cooperators = Cooperators that respond at later calls (say, 6 or more) Early cooperator effect (ECE) = expected difference in estimates based on early vs. early + later cooperators (say, )

5. Response rates as a function of number of call attempts Number of call attempts

6. Illustration 1- Have you ever been told by a doctor, nurse or other health professional that you had asthma? Number of call attempts 1-51-15+ Percent “yes”13.813.4 Small ECE maximum of 5 calls is adequate

7. Illustration 2- During the past 12 months, have you had a flu shot? Number of call attempts 1-51-15+ Percent “yes”38.335.8 Larger ECE max of 5 call attempts may be biasing Could consider other definitions of “early cooperator.”

8. Why study ECE? Effort (and costs) could be saved if ECE is small If ECE is not small, adjustments may be applied to reduce it May need to adjust for HCNRs, not only later cooperators

9. What adjustments can be applied to reduce the ECE? Nonresponse adjustments –Requires characteristics of nonrespondents –Lack of information a limitation for some surveys Post-stratification adjustments –Requires known target population totals within adjustment cells –Variables limited to those available externally Call-back model adjustments –Assumes response propensity is function of level of effort required to obtain a response and grouping variables –Related work of Drew and Fuller (1980), Politz and Simmons (1949), others

10. ECE in the BRFSS Survey details One of the largest RDD surveys in the world Estimates the prevalence of risk behaviors and preventive health practices Monthly, state-based, cross-sectional survey Target population is adults in telephone hh’s Data source: 2004 survey with ~300,000 interviews

11. ECE in the BRFSS (cont’d) Early cooperator defined as responding with 5 fewer call attempts Examined differences in –demographic characteristics –10 selected health characteristics overall and by demographic domain ECE estimated by Data weighted by base weights only

12. Typical Values of ECE General Health - ExcAsthma Drink AlcoholFlu Shot Prevalence21%13%53%36% Total1.20.3-2.22.6 Male1.30.1-1.72.9 Female1.10.4-2.02.2 White, non- Hispanic 1.60.1-2.72.4 Black, non- Hispanic 2.50.6-2.21.1 Hispanic-0.71.5-0.91.4

13. Typical Values of ECE (cont’d) Education General HealthAsthma Drink Alcohol Influenza Shot < High school1.11.4-2.62.9 High school1.40.1-2.52.7 > High school1.00.3-1.72.4 Number of adults One2.10.1-2.93.1 Two1.10.2-2.02.5 Three or more0.50.9-1.81.4

14. Summary of the Results Early cooperators are different from later cooperators on many dimensions For most characteristics ECE is relatively small –Less than 3 percentage points at aggregate level –Rarely more than 3 points for domains For some characteristics, ECE may be important Other definitions of ECE also considered

15. Hardcore Nonresponse Bias Hardcore Nonrespondents = Units that will not respond under the current survey protocol no matter the number of call-backs ECE does not include the bias due to hardcore nonrespondents Total nonresponse bias = Bias due to cooperators who did not respond + bias due to hardcore nonrespondents Adjusting for ECE may not remove bias due to HCNR

16. Call-back Models for Adjusting for ECE and HCNR Bias General idea –Estimate the response propensity for subgroups of the population –Response propensity is modeled as a function level of effort (LOE) to obtain a response Two models are considered –Manifest model (MM) – Ignores HCNR –Latent class model (LCM) –Includes HCNR Includes a latent indicator variable to represent the HCNR’s in the population Why latent?

17. Illustration for 5 Call-backs Group AGroup B 111113311133332 3111133333 331113331133331 1111133332 333313111133333... 1 = interview; 2 = noninterview; 3 = noncontact

18. Illustration for 5 Call-backs Group A High response propensity Group B Medium response propensity Group B Low response propensity 111113311133332 3111133333 331113331133331 1111133332 333313111133333... 1 = interview; 2 = noninterview; 3 = noncontact

19. Potential Advantages over Post-Stratification Post-stratification adjustments (PSA’s) depend upon the availability of external benchmarks or auxiliary data –Selection of control variables is quite limited –Target populations also quite limited –Adjust for “ignorable” nonresponse only

20. Potential Advantages over Post-Stratification Call-back model can rely only on internal variables –Weighting classes can be defined for any variables collected in the survey –Can be applied for any target population –Greater ability to selected variables that are highly correlated with response propensity –Adjust for “ignorable” and “nonignorable” nonresponse

21. Modeling Framework Simple random sampling Survey eligibility is known for all sample members No right censoring –(i.e., all noncontacts received maximum LOE) Extensions to relax these assumptions are described in the paper

22. Incorporating the Model-based Weights Unadjusted estimator of the mean Adjusted estimator of the mean Based on the sample distribution Estimated from the model

23. Two Models for Estimating MM (Manifest Model) Assumes all nonrespondents would eventually respond at some LOE (i.e., all nonrespondents have a positive probability of response) LCM (Latent class model) Incorporates 0 probability of response for the hardcore nonrespondents (HNCR’s)

24. Technical Details

25. Notation Levels of effort (LOE) Outcome of LOE l where 1=interview, 2 = noninterview, 3=noncontact LOE associated with state S=1 or 2 Grouping variable (weighting class variable)

26. Notation Number of sample persons in group g interviewed at LOE l* Number of sample persons noninterviewed at LOE l* Number of sample persons never contacted after L (max LOE) attempts Probability person in group g is interviewed at LOE l* Probability person in group g is noninterviewed at LOE l* Probability person in group g is never contacted

27. General Idea –Outcome Patterns for 5 Call-backs Cooperator HCNR 11111 0 31111 0 33111 0 33311 0 33331 0 22222 32222 33222 33322 33332 33333

28. Likelihood for the Manifest Model This model is appropriate when (a)Every sample member has a positive probability of responding at some LOE, or (b)Adjustment for ECE only is desired

29. Likelihood for the Latent Class Model Introduces a latent variable X where X = 1, if HCNR and X = 2, if otherwise Appropriate when some sample members have a 0 probability of responding and adjustment for total nonresponse (Later Cooperators + HCNR’s) is desired

30. Model Restrictions The key assumptions regard the form of the probabilities: where, and and

31. Model Restrictions (cont’d) (i), say, for l = 3,...,L for L<10 Manifest Model and (ii) i.e., no HCNR,s and response probabilities are equal for LOE 3,..., 9. The probabilities for LOEs 1, 2 and 10 (if present in the model) were not restricted Latent Class Model (i)is replaced by... (i′) to represent the HCNR group in the model.

32. Results

33. Four Estimators were Considered Unadjusted estimator Estimator using MM estimates of Estimator using LCM estimates of Estimator using CPS estimates of –i.e., usual PSA estimator –treated as the “gold standard”

34. Comparison of the ECE for a Maximum Five Callbacks Strategy Before and After MM Adjustment GENHLTH Estimate % Unadjusted ECE Manifest Model ECE Excellent20.7-0.9-0.6 Very good33.1-0.4 Good29.6 0.1 ALCOHOL52.8-2.2-1.8 ASTHMA13.4 0.3 0.5 DIABETES 8.8 0.7 0.3 FLUSHOT35.8 2.5-0.8 HLTHCOV86.0 0.8-1.3 PHYMO18.7 2.2 0.9

35. Differences between PSA and Unadjusted and Adjusted Estimates for a Maximum Five Callbacks GENHLTH PSA Estimate Diff Unadj Diff MM Diff LCM Excellent20.7-0.6-0.3-1.2 Very good33.1 0.3-0.1-1.6 Good29.6-0.1 1.0 ALCOHOL52.8-0.6-0.3-0.8 ASTHMA13.4-0.3-0.1 0.1 DIABETES 8.8 0.9 0.4 0.6 FLUSHOT35.8 4.2 0.9-0.1 HLTHCOV86.0 2.7 0.6-2.5 PHYMO18.7 1.9 0.7-0.2

36. Estimating the Potential Bias Reduction BRFSS data do not exhibit very large nonresponse biases Therefore, consider a variable, Y, that has maximum nonresponse bias given the BRFSS nonresponse rates To do this, we form Y g BRFSS response rate for group g Compute the relative difference between unadjusted and adjusted estimates and the PSA estimate of the mean of Y

37. A Measure of Potential Bias Reduction Define the relative difference operator for any maximum LOE, L as Gold standard estimator

38. Absolute Relative Differences (|RD L |) for Unadjusted and Adjusted Estimators as a Function of Number of Call-backs No. of Call-Backs|RD U,L | (%)|RD MM,L | (%)|RD LCM,L | (%) 58.85.21.4 76.94.02.5 95.83.42.9 118.83.42.9 144.52.62.8 154.02.32.4

39. Conclusions ECE for 5 call-backs is generally small, but can be moderately high for some characteristics The Manifest Model can be employed to reduce ECE The Latent Class Model can be employed to reduce total nonresponse bias (Later Cooperators + HCNR bias) Future research should focus on –Variable selection –Comparisons of MSEs of the estimators –Small/medium size sample properties –Integration with other post-survey weight adjustments