1. Analysis of the characteristics of internet respondents to the 2011 Census to inform 2021 Census questionnaire design Orlaith Fraser & Cal Ghee

2. Overview 1.The census and modes of response 2.Census Quality Survey 3.Propensity score analysis I.Census quality survey results II.2011 Census results 4.Conclusions

3. The census and modes of response Mode effect = Same respondent gives different response when using different modes X Y 2011 Census: Paper by default, Internet option - 81% responded by paper, 19% by internet 2021 Census: Internet by default, possible paper option.

4. Census Quality Survey (CQS) What is your date of birth? 01 1977 Face-to-face CAPI sample survey. Sample of census respondents asked majority of census questions again. Answers compared to calculate agreement rates. CQS answers assumed correct as face-to-face likely to be more accurate than self-completion.

5. Census Quality Survey Stratified Sample  region, hard to count, mode No adult proxy responses Individuals weighted  age, sex, ethnic group, mode Household representative interviewed 5170 matched households 9,650 matched usual residents 5170 matched households Paper/CQS agreement rate Internet/CQS agreement rate Compare agreement rates

6. Internet agreement rates significantly higher than paper

7. Paper agreement rates higher than internet

8. Possible reasons for differences AgeScanning errors? Marital status Social desirability bias? Disability Recall error? Group most likely to change answer Religion Possible reasons for other differences: Use of radio buttons Help information Scrolling distance Paper format easier to look ahead

9. Propensity score method Direct comparison between internet and paper responders difficult because of differences in respondent characteristics Distribution of internet responders matched to that of paper responders

10. Proportion of internet responders Propensity Score Method Propensity score = Propensity towards exposure to a treatment (responding by internet) given a set of observed characteristics Proportion of paper responders Adjustment factor for each subgroup = 1.Individual propensity scores derived from logistic regression model 2.Respondents split into ten subgroups based on propensity scores 3.Internet distribution standardised by applying adjustment factors STEPS

11. Propensity Score Analysis of CQS data VARIABLE CQS/Census Agreement Paper % Internet % Internet - Paper UnadjustedAdjusted Unpaid CareAgree82.5080.462.04-0.26 DisabilityAgree85.6891.99-6.31-1.72 Workplace addressAgree43.9749.31-5.34-5.91 Address one year agoAgree25.5132.17-6.66-3.39 Variables included in logistic regression model: Sex, student status, disability, English as a main language (English or Welsh in Wales), good health and whether working

12. Propensity Score Analysis of CQS data VARIABLE Internet - Paper UnadjustedAdjusted Unpaid Care-2.040.26 Disability6.311.72 Workplace address5.345.91 Address one year ago6.663.39 Variables included in logistic regression model: Sex, student status, disability, English as a main language (English or Welsh in Wales), good health and whether working Easier to check postcodes online

13. Limitations Household responses may not be independent A highly educated young person may respond online for an older less well educated person Proxy effect Proxy may not have responded in the same way as the individual they were representing Chicken and egg dilemma Any mode effects included in the model may be considered as actual predictors Small CQS sample Can’t restrict sample to one response per household

14. Propensity Score Analysis of Census data  More robust analysis using 10% microdata household sample of census data - stratified by output area  Only household reference persons included  Mainly household level variables included as model predictors  Direct comparison of paper/internet proportions for variable categories rather than CQS/Paper and CQS/Internet agreement rates

15. Propensity Score Analysis of Census data Variables included in new logistic regression model: Age of household reference person Country of birth Deprivation indicators of a household Ethnic group Household language Household reference person social grade Living arrangements Number of cars and vans in household Region Size of household Tenure Urban Rural classification

16. Results VariableCategoryPaper %Internet % Internet - Paper UnadjustedAdjusted Pensionable age indicatorOf pensionable age (65+)29.328.95-20.37-1.11 SexMale58.7366.347.612.09 DisabilityDay-to-day not limited at all73.5886.0612.48-1.99 HealthVery good health32.1943.9611.770.86 HoursWorks 30-49 hrs/wk63.5267.323.801.30 Activity Last WeekNot working39.4819.24-20.23-1.08 Main Language English as main language (or Welsh in Wales)93.8389.60-4.23-0.38 Marital statusWidowed12.664.09-8.571.32 Marital StatusSingle24.8330.025.19-1.19 Level of highest qualifications Level 2: 5 GCSEs (A* - C) or 1 A level or equivalent12.4239.7827.37-0.57 All internet/paper differences negligible after adjustment – all explained by differences in population characteristics

17. Conclusions Propensity score analysis can be a useful tool for identifying true mode effects Most differences in internet / paper responses attributable to characteristics of respondent group Useful tool for testing mode effects prior to 2021 census  New mode effects: tablet/mobile/desktop...  More detailed knowledge of internet/paper respondent profiles will help to target support for digitally excluded / reluctant internet respondents

18. Further information Please contact: orlaith.fraser@ons.gsi.gov.uk cal.ghee@ons.gsi.gov.uk 2021 Statistical Design, Census Transformation Programme, Office for National Statistics