Analysis of the characteristics of internet respondents to the 2011 Census to inform 2021 Census questionnaire design Orlaith Fraser & Cal Ghee

Overview 1.The census and modes of response 2.Census Quality Survey 3.Propensity score analysis I.Census quality survey results II.2011 Census results 4.Conclusions

The census and modes of response Mode effect = Same respondent gives different response when using different modes X Y 2011 Census: Paper by default, Internet option - 81% responded by paper, 19% by internet 2021 Census: Internet by default, possible paper option.

Census Quality Survey (CQS) What is your date of birth? Face-to-face CAPI sample survey. Sample of census respondents asked majority of census questions again. Answers compared to calculate agreement rates. CQS answers assumed correct as face-to-face likely to be more accurate than self-completion.

Census Quality Survey Stratified Sample  region, hard to count, mode No adult proxy responses Individuals weighted  age, sex, ethnic group, mode Household representative interviewed 5170 matched households 9,650 matched usual residents 5170 matched households Paper/CQS agreement rate Internet/CQS agreement rate Compare agreement rates

Internet agreement rates significantly higher than paper

Paper agreement rates higher than internet

Possible reasons for differences AgeScanning errors? Marital status Social desirability bias? Disability Recall error? Group most likely to change answer Religion Possible reasons for other differences: Use of radio buttons Help information Scrolling distance Paper format easier to look ahead

Propensity score method Direct comparison between internet and paper responders difficult because of differences in respondent characteristics Distribution of internet responders matched to that of paper responders

Proportion of internet responders Propensity Score Method Propensity score = Propensity towards exposure to a treatment (responding by internet) given a set of observed characteristics Proportion of paper responders Adjustment factor for each subgroup = 1.Individual propensity scores derived from logistic regression model 2.Respondents split into ten subgroups based on propensity scores 3.Internet distribution standardised by applying adjustment factors STEPS

Propensity Score Analysis of CQS data VARIABLE CQS/Census Agreement Paper % Internet % Internet - Paper UnadjustedAdjusted Unpaid CareAgree DisabilityAgree Workplace addressAgree Address one year agoAgree Variables included in logistic regression model: Sex, student status, disability, English as a main language (English or Welsh in Wales), good health and whether working

Propensity Score Analysis of CQS data VARIABLE Internet - Paper UnadjustedAdjusted Unpaid Care Disability Workplace address Address one year ago Variables included in logistic regression model: Sex, student status, disability, English as a main language (English or Welsh in Wales), good health and whether working Easier to check postcodes online

Limitations Household responses may not be independent A highly educated young person may respond online for an older less well educated person Proxy effect Proxy may not have responded in the same way as the individual they were representing Chicken and egg dilemma Any mode effects included in the model may be considered as actual predictors Small CQS sample Can’t restrict sample to one response per household

Propensity Score Analysis of Census data  More robust analysis using 10% microdata household sample of census data - stratified by output area  Only household reference persons included  Mainly household level variables included as model predictors  Direct comparison of paper/internet proportions for variable categories rather than CQS/Paper and CQS/Internet agreement rates

Propensity Score Analysis of Census data Variables included in new logistic regression model: Age of household reference person Country of birth Deprivation indicators of a household Ethnic group Household language Household reference person social grade Living arrangements Number of cars and vans in household Region Size of household Tenure Urban Rural classification

Results VariableCategoryPaper %Internet % Internet - Paper UnadjustedAdjusted Pensionable age indicatorOf pensionable age (65+) SexMale DisabilityDay-to-day not limited at all HealthVery good health HoursWorks hrs/wk Activity Last WeekNot working Main Language English as main language (or Welsh in Wales) Marital statusWidowed Marital StatusSingle Level of highest qualifications Level 2: 5 GCSEs (A* - C) or 1 A level or equivalent All internet/paper differences negligible after adjustment – all explained by differences in population characteristics

Conclusions Propensity score analysis can be a useful tool for identifying true mode effects Most differences in internet / paper responses attributable to characteristics of respondent group Useful tool for testing mode effects prior to 2021 census  New mode effects: tablet/mobile/desktop...  More detailed knowledge of internet/paper respondent profiles will help to target support for digitally excluded / reluctant internet respondents

Further information Please contact: Statistical Design, Census Transformation Programme, Office for National Statistics