The impact of using the web in a mixed mode follow-up of a longitudinal birth cohort study: Evidence from the National Child Development Study Presenter: Joel Williams(TNS-BMRB) Co-authors: Matt Brown and Alissa Goodman (Centre for Longitudinal Studies, UCL Institute of Education)

Year Age Prior sweeps all took place in person (with exception of 2004 telephone survey) Age 55 Survey: Mixed Mode – Web followed by Telephone Firsts for NCDS (and CLS cohorts): Use of mixed mode approach AND use of web TEL National Child Development Study (1958-)

Why go online? Cost Increasingly being done on other surveys To give cohort members a choice Confident that this would deliver high quality data: –Short questionnaire (25 minutes) –(Mostly) factual questions – less susceptible to mode effects –Evidence from other surveys (LSYPE, British Crime Survey, HRS) –Evidence from NCDS – high rates of computer/web use

Key concerns Web take-up rate / overall response rate Data quality / mode effects e.g. item non-response, satisficing, social desirability bias (TEL), positivity bias on rating scales (TEL) Extensive user testing / piloting Careful design of contact strategy / data collection instrument Experimental design

Address and UK telephone numberAddress onlyNon UK residents ONLINE ONLY ONLINE>TELEPHONETELEPHONE ONLY ONLINE>TELEPHONE Other! ONLINE>TELEPHONE 28

83%

Who is most likely to respond online? Participated in prior survey Regular computer / internet users White British Professional / managerial occupations Homeowners Higher than average net pay Higher than average (self- reported) health Non-smokers Do not drink heavily Those with a partner Those with higher than average cognitive function

Mixed modeTelephone onlyMM-TO Expected value of property12.9%7.9%5.0% Amount to pay off on property15.5%11.1%4.5% Gross weekly income14.8%10.6%4.2% Number of cigarettes a day usually smoked2.3%0.0%2.3% Derived weight - kg6.2%5.2%1.0% Whether voted in last general election0.8%0.6%0.2% Units of alcohol consumed in last 7 days1.7%1.6%0.1% Likelihood of working at the age of 602.2%2.0%0.1% Frequency of alcohol consumption0.1% Likelihood of working at the age of 662.2%2.4%-0.2% Party voted for in 2010 general election7.7%8.0%-0.4% Whether employer provided pension scheme is of Type A or Type B 5.5%13.1%-7.6% Item non-response rates?

p<0.05 p>0.05

p<0.05 p>0.05

p<0.05 p>0.05

p<0.05 p>0.05

Self-rated financial situation

Self-rated health

Guidance to users User guide will shortly be available All variables affected by mode effects will be listed Different users will account for mode effects in different ways Options (if developing substantive linear models) –+Mode indicator: combines measurement effect with mode selection effect »+(Estimated) mode selection probability: helps separate these effects out –+Experiment group indicator: isolates measurement effect (but understates it) First option simplest and will control for the ‘noise’ of the MM design But descriptive statistics are mode-specific so need measurement effect

Impact on regression coefficients Tested addition of mode/experiment group indicator to a small number of regression models where mode influences the response level –Units of alcohol per week –Self-rated health Addition of mode indicator: 0-10% change in coefficients (mean = 2.7%) Addition of ex. group indicator: 0-3% change in coefficients (mean = 0.3%) Mode / ex. group coefficients sometimes have different signs

Conclusions I MM had a higher response rate than TO and so more interviews Convenience of completing online seemed popular –Almost no dropout once the questionnaire was opened (despite 30m length) –Generally positive feedback from cohort members Most – but not all - variables unaffected by mode of data collection Inclusion/otherwise of mode indicator(s) has little impact on regression model coefficients

Conclusions II (but…) Higher rates of item non-response (esp. on items where item non- response already high – e.g. income) Important differences in some key items - and could be more of a problem if the survey contained more opinion items, esp. with response scales Accounting for mode specific measurements is an extra complication for data users (guidance being prepared) Long-term impact on study participation is unknown