1. Rapid Risk Factor Surveillance System Conference
Assessing Validity of the Behavioral Risk Factor Surveillance System in an Era of Declining Response Rates
Michael W. Link, Ph.D.
Centers for Disease Control and Prevention
Atlanta, GA, USA
Rapid Risk Factor Surveillance System Conference
Toronto, Canada, June 21, 2007

3. Problems facing random digit dialed telephone surveys in the U.S.
Growing Nonresponse
Frame coverage issues:
Households with no phone (2-3%)
Cell phone only households (12-13%)
Frame efficiency issues:
Proliferation of telephone numbers
Number portability:
Erosion of geographic specificity at state and substate levels

4. What? People don’t like us calling?

9. Behavioral Risk Factor Surveillance System (BRFSS)
Monthly state-based RDD survey of health issues
50 states, District of Columbia, Puerto Rico, Guam, and Virgin Islands
350,000+ adult interviews conducted in 2006
From 2002 to 2006:
completed 1,517,000 interviews
Dialed 14,381,000 telephone numbers

11. Under-representation by 2010: Percentage point difference from Census projections

12. What do we mean by “validity”?
The closeness of our survey estimates to the “true value”
Ideally there is no difference
Potential survey bias is minimized
“Bias” in survey estimates results from product of:
Level of nonresponse
Difference between respondents and nonrespondents on measures of interest

13. Ensuring validity of BRFSS Estimates
Monitoring data collection process
Refining post-survey adjustments
Benchmarking to other studies
Testing alternative ways of collecting data
Cell phone interviewing
Address-based sampling (ABS)

14. Monitoring the Data Collection Process

15. Monitoring 54 monthly surveys
BRFSS data collection process is semi-centralized
States:
In charge of own data collection
Conduct front-line monitoring
Centers for Disease Control (CDC):
Provides sample
Weighting
Quality reports

16. Web-based systems are key
Data transfer via upload/download site
Automated quality control programs
State level and CDC level
Monthly detailed reports to states:
Key quality indicators
Deviations from norm and/or past trends within state
Year-end quality report
Comparison across states
Newest tool: Simplified web-based / color coded system

17. 11/24/2018

19. What did we learn?
Estimates are only as valid as the process in which the data were collected
Tools for monitoring the quality of data collection and collecting valid data are only good:
… if they are actually used
… and if they are understood
… and initiate follow-up action

20. Refining post-survey adjustments

21. Goals and limits of weighting
Weighting and other post-survey adjustments are used to correct for imbalances in the data due to issues of:
Coverage
Sampling
Nonresponse
Weighting methodology affects the estimates produced
Can only weight data you have
Assumes no difference between respondents and nonrespondents on variables of interest
Can only weight to external standards that exist
Typically limits weighting to a handful of demographic variables, not “substantive” variables

22. Current BRFSS Weighting System
Use poststratification (cell-based approach)
Controls for:
Age by sex
Race/Ethnicity (in some states)
Region (in some states)
Problems:
Small sample cells produce highly variable weights and require collapsing
No factor to account for socioeconomic status

23. Conducted detailed analysis to identify key demographic correlates of BRFSS Measures
Age
Race/ethnicity
Gender
Education
Marital status

24. New weighting system Controls for:
Uses “Multi-Dimensional Raking” (Sample Balancing)
Controls for:
Age by sex
Race/ethnicity (2.5% rule)
Region (as necessary)
Education level
Marital status
Telephone service interruption

25. Does It Represent an Improvement?

26. Changes in estimate of health status

27. Implications for users of BRFSS data
Break in time series
Plan to release both classic (old) and new weights
Full changeover in 2010
Health condition and risk factor estimates will likely be higher

28. What did we learn?
Modifications to post-survey adjustments can improve the quality of the estimates produced
Sometimes need to be innovative in the use of external data in developing population estimates

29. Benchmarking to external standards

30. Importance and challenges of benchmarking
True standards rarely exist in health surveys – relative standards
Better coverage, response
No two studies are identical
Populations
Modes / procedures
Wording / question order
Post-survey adjustments / population standards

31. Benchmark surveys for BRFSS
National Health Interview Survey (NHIS):
In-person interviews with adults 17+
2004: 94,460 adults in 36,579 households
Household-level response rate = 86.9%
National Health and Nutrition Examination Survey (NHANES):
In-person survey with physical measures at mobile lab
: 10,122 adults
Household-level response rate = 91.0%

32. Comparison across 15 key health variables
Cigarette smoking
Diabetes
Height
Weight
Body mass index
Health status
Asthma
HIV testing
Alcohol consumption
Medical coverage
Influenza vaccination
Pneumonia shot

33. Summary of findings BRFSS vs NHIS estimates:
Significantly different on 10 of 15 variables
Relative difference:
Asthma = +35%
HIV testing = +26%
BRFSS vs. NHANES estimates:
Significantly different on 5 of 6 variables
Current smoking = -12.2%
Body mass index = -2.1%

34. Ever smoke cigarettes
48.0%
44.0%
42.4%

35. Ever told had diabetes
8.1%
8.0%
6.1%

36. Body Mass Index
27.6
27.0
27.0

37. Percentage of 18 – 34 year olds
41.5%
31.5%
31.2%

38. Percentage of Males
50.9%
48.4%
48.4%

39. Percentage of whites
74.9%
71.7%
70.4%

40. What did we learn? There are no “gold standards” in health statistics
All comparisons are relative
Surveys can vary in terms of backend processing just as much as on front-end design and operational issues
Determining if BRFSS compares favorably with other surveys is a matter of perspective

41. Finding new ways of collecting data: Cell phones & Address-based sampling

42. The Plague of Cell Phones!!!

43. Cell phones and telephone surveys
Reliance on cell phones increasing:
Nearly 70% of households in US have a working cell phone
In late 2006, 12.8% of households were cell phone-only
Conducting surveys via cell phones can be operationally challenging:
Cell phone frame very inefficient
Cannot use autodialers
Charges for incoming calls/minutes used
Safety concerns
Potential mode effects / measurement errors

44. 2007 BRFSS cell phone pilot Conducted in 3 U.S. states
Target: 600 cell & landline / 600 cell-only
Abbreviated BRFSS core interview:
66 questions
15-17 minutes (on average)

45. Response rates
29.7%
14.8%
14.1%

46. Landline and Cell phone populations and framesB A C
CELL PHONE

47. Percent male 51.1 46.6 37.9 38.2 Landline survey Cell phone survey
State equalized design weight applied

48. Percent 18-34 years 51.4 24.0 14.5 19.6 Landline survey
Cell phone survey
State equalized design weight applied

49. Percent Hispanic 21.4 15.2 16.8 12.2 Landline survey Cell phone survey
State equalized design weight applied

50. Percent high school or less education
60.3
48.5
39.8
33.6
Landline survey
Cell phone survey
State equalized design weight applied

51. Comparison of key survey estimates

52. Percent any kind of health care coverage
89.0
86.0
78.7
70.1*
Landline survey
Cell phone survey
State equalized design weight applied

53. Percent currently smoke cigarettes
31.1*
24.8
19.7
17.3
Landline survey
Cell phone survey
State equalized design weight applied

54. Percent ever tested for HIV
54.2*
43.6
37.5
36.6
Landline survey
Cell phone survey
State equalized design weight applied

55. Percent binge drink past 30 days
23.5
21.1
13.0
11.0
Landline survey
Cell phone survey
State equalized design weight applied

56. What did we learn?
The part of the population we are missing due to cell phones is different from those we interview --- and we cannot ignore them
Missing critical information needed to integrate landline and cell phone samples at the sub-national level
No reliable external standards denoting telephone usage at subnational level

57. A New Direction and an Old Friend: Address-based Sampling (ABS) and Mail Surveys

58. Address-based sampling (ABS)
Improvement in computer databases and commercial list-building makes sampling addresses feasible in the U.S.
Best source (to date): U.S. Postal Service delivery file
Matching addresses to phone numbers facilitates mixed-mode approaches

59. BRFSS 2006 ABS mixed-mode pilot
Six U.S. states participated
Sample frame: USPS Delivery File
Mixed mode data collection:
Initial mail survey
Postcard reminder
Second mail survey (to nonrespondent)
Telephone survey follow-up (of nonrespondents)
75 questions from BRFSS core questionnaire

60. Response Rates State RDD telephone survey ABS multimode Survey CA 25.4
36.1*** FL
32.8
37.4*** MA
26.3
42.4*** MN
48.5
54.1*** SC
49.1
41.8*** TX
28.7
35.3***
Mean
35.1
41.2

61. Percent male
48.6
40.9
39.3
RDD and ABS data weighted by state equalized design weight only.

62. Percent years old
31.1
19.0
17.6
RDD and ABS data weighted by state equalized design weight only.

63. Percent 65 years and older
23.9
22.7
16.9
RDD and ABS data weighted by state equalized design weight only.

64. Percent some college or more
67.2
60.8
55.7
RDD and ABS data weighted by state equalized design weight only.

65. Percent Hispanic
16.2
14.9
8.9
RDD and ABS data weighted by state equalized design weight only.

66. Type of household telephone access
National Health Interview Survey1
(%)
BRFSS ABS mixed-mode survey
Land line
85.0
88.4
-- Landline only
---
14.5
-- Landline and cellular phone
73.9
Cellular phone only
12.8
10.5
No telephone
2.2
1.1
1 SJ.Blumberg and JV Luke (2007). “Wireless substitution: Early release of estimates based on data from the National Health Interview Survey, July – December 2006.” National Center for Health Statistics E-States.

67. Comparison of Survey Estimates
Health condition /
risk factor
RDD
telephone survey
ABS
multimode survey
Health care coverage
81.9
81.3
Asthma
12.4
13.4
Diabetes
9.3
10.8
Cardiovascular disease
8.3
8.7
Obese (BMI > 30)
22.9
26.7***
Current smoker
20.1
19.9
Binge drinking
15.1
18.1***
Tested for HIV
36.7
36.1
[n]
[21,743]
[4,871]

68. 18-34 year olds: Binge drinking & HIV test by household phone access
Note: Data weighted for sample design and post-stratified to sex, age, and race totals for each state. Final weights were ratio adjusted to equalize the number of cases across states.

69. What did we learn?
Need to begin work now on the workhorse approach of the future
Collection of valid health data likely to involve:
New ways of identifying sample members
Multiple modes of data collection

70. Concluding thoughts
Producing valid survey estimates is a multi-phase / multifaceted process
Assessing validity is often quite difficult, involving a mix of scientific rigor and subjective judgment
Ensuring validity is a necessity for the long-term survival of any health surveillance system

71. Contact: Michael Link MLink@cdc.gov For more information on BRFSS: www.cdc.gov/brfss