Nonresponse issues in ICT surveys Vasja Vehovar, Univerza v Ljubljani, FDV Bled, June 5, 2006

Introduction –Survey research as scientific discipline –Notion of probability samples –Specific aspect of nonresponse in ICT surveys where nonresponse is often linked to ICT characteristics

Probability samples These are the only samples that enable statistical inference from sample to population and the construction of confidence intervals Definition: We have to know the probability of each unit in the target population (i.e. Sample frame) in advance and it has to be positive for all units These are a very difficult requirements: We have problems with sample frames We have problems with nonresponse Other problems

Nonresponse: Definition Nonresponse occurs when an eligible unit, which was already included into the sample, failed to cooperate in the fieldwork stage of the survey. Major types of nonresponse: –Refusal –Non-contacts (e.g. absence)

Mere cost inflator or a mortal thread for sample surveys Experience shows that with increased efforts (incentives, more contacts, mixing modes, refusal conversions, interviewer training etc.) dramatic improvement in response may occur. But this is not always the case; sometimes serous bias remains even after substantially increased response rates. On the other hand, the increased efforts for lowering nonresponse rates are much more often in vain, as they bring no improvements (or very little) to the estimates.

The nonresponse danger Wn is the share of nonrespondents Yr is the estimate (e.g. share of internet users) among respondents Yn is the estimate of target variable among nonrespondents Bias = Wn*(Yr-Yn) Example:Wn=0.5, Yr=0.9, Yn=0.5 Bias=0.2 (i.e. 0.9 instead of 0.7)

Example I RIS survey among companies

RIS survey I RIS regular surveys on ICT among companies Response rates from 1996 (80%) to 2002 (57%) and 2005 (43%). Telephone surveys with up to 10 follow-ups (n=1000). RIS 2002 response rate (including also the partial respondents) among eligible units in the sample was 73%, with considerable differences in response rates: –largest companies (500+ employees) 90%, –smallest (up to 5 employees) 40%.

RIS survey II The initial nonrespondents (refusals) were asked four key questions: –number of PCs in the company, –number of employees, –Internet access, –Web site presence. Typically, majority of initial refusals do cooperate. At least 40% of all nonrespondents (including noncontacts) provide answers.

RIS survey III With complete and partial respondents (57+16=73%) and nonrespondent (27%) we thus have: 1. Respondents to full questionnaire: 57 % 2.Respondents to short questionnaire: 16 % 3.Unit nonrespondents: 27 % Total 100 %

RIS survey IV

RIS survey V The resulting bias is not ignorable, as a quarter of the difference between the groups enters into the bias. Of course, in addition, there were also the total nonrespondents (27%), for which we may assume that they are even less equipped with ICT If the nonrespondents are at least similar to partial respondents, half of the difference move into the bias, i.e. the true values are between the values of the two groups.

Example II State of the CIO Survey (Chief Information Officer)

State of the CIO I For the needs of CIO (Chief Information Officer) conference (May 2006) a survey among Slovenian CIOs was performed on-line to obtain some insight. Out of over 3000 invitations, 57 responded. The data were then compared with a similar on- line US study (State of the CIO 2006), n=500.

State of the CIO II For the majority of variables, the results were comparable to RIS representative (expensive) telephone survey. In particular, the relations and the ratings (on 1-5 scale) were correctly estimated. Discrepancies existed only with respect to shares arsing from the size structure of the Web survey (to few small companies), e.g. share of ICT expenditures in turnover, share of companies using ROI to evaluate ICT investments The comparisons with US study were also very reasonable.

Conclusions I How is it possible that results usually came out to be so good? Why would anybody pay for an expensive survey with high response rate, if cheap and fast survey data collection will do? The statisticians would say that nothing but probability sample can save us from unpredictable risk of being wrong. The market researchers would say that “sampling is dead”, because things work anyway How to optimise costs vs. data quality?

Conclusions II There are over 50 strategies to handle nonresponse. Besides usual weighting and imputation, the following are the most promising although relatively rarely used: Easy additional strategies: –Apply the key questions for refusals –Analyse the effect of the number of contacts Advanced (i.e. expensive) strategies: –Subsampling of the nonrespondents –Increased number of contacts –Variation of the mode of contacts