Challenges to Surveys Non-response error –Falling response rates over time Sample coverage –Those without landline telephones excluded –Growing concerns about sample coverage due to wireless only households

What is Known about Non- Response Error Response rates are falling for all types of survey –RDD, mail, in-person, federal and state Attributed to: –Growth in non-contact rate Largely due to introduction of telephone screening devices and more active telephone numbers –Growth in refusals Perhaps spurred by growth in unsolicited calls (fund-raisers, market research and survey research generally)

Is there a relationship between response rates and response bias? Asked another way: Do surveys with lower response rates actually produce more absolute response bias? Literature indicates:  Keeter et al. (2000) and (2006) showed that a survey with a very low response rate obtained very similar estimates on key variables as a survey with a much higher response rate (20-30 versus 50-60)  Most of the items examined focused on public opinion issues rather than health  A recent study completed at the U of MN by Davern et al. (2008) showed  Those we work hardest to obtain responses from (initial refusals and those who answer the survey after being contacted several times) are different from others with respect to some socio- demographic characteristics  However, after controlling for socio-demographic factors used in weighting there are few significant differences on key health outcome measures

Response rates are not a good indicator of quality or response bias  Response bias remains a serious problem in high and low response rate surveys alike  The same survey can have some items with a large response bias and a low response bias  “Blind pursuit” of a high response rate is not wise (Groves 2006)  Calling people 50 times and trying to convert refusals many times increases respondent burden  It reduces the ultimate number of respondents that a survey can obtain  Reduces the statistical power of a study  Some evidence has shown that those people we work the hardest to get data from also provide the least accurate information  We should be more concerned with response bias from various estimates from the same survey and between surveys than response rates

What is Known about Sample Coverage Error Percent of households with no phone service is stable over time at between 1-2% –Handle through post-stratification weighting adjustments; considered a conventional adjustment Wireless only households have grown from approximately 4% of people in the US in 2004 to approximately 16% in 2007 –No conventional adjustment has surfaced as yet –Handle through post-stratification weights: age by education, home ownership

Characteristics of Wireless Only Households Wireless only households are different: –Younger (age 18-29) –Adults living with unrelated adults –Renters vs homeowners –Men –Living in poverty –Hispanic and Non-Hispanic Black Relation to health measures: –Better self-reported health, more active –Higher rates of uninsurance –More likely to experience financial barriers to health care –Less likely to report usual source of care (Blumberg & Luke, 2008)

Income questions in 2004 and 2008 Both seek annual income of TARGET and their family –2004 Question 1 refers to family income and Question 2 refers to household income Income is defined within the Question 2 stem (e.g., this includes money from jobs, net income from business, farm or rent, pensions, dividends, interest, social security payments and any other money income received by members of this FAMILY who are 15 years or older) –In 2008 Question 1 introduces what is meant by family income and Questions 2 -4 define income E.g., salary and wages, dividend or interest income, SSI, income from other sources such as self-employment, alimony, child support, contributions from family or others, unemployment compensation, worker’s compensation or veteran’s payments, Social Security or pensions, or anything else Although similar in content, and potentially similar in the way the respondent “hears” and “responds” to these questions, they are not directly comparable