1. Managing Multi Mode Collection Instruments in the 2011 UK Census Frank Nolan, Heather Wagstaff, Ruth Wallis Office for National Statistics UK

2. IAOS20082 Contents What I plan to cover Electronic Questionnaires Challenges Outcomes

3. IAOS20083 Context Traditional Census –High Profile –Involves ALL citizens –Enumerators – face-to-face Increased Diversity –Lower response rates Modern Government / NSI - eforms Options for Multi Mode

4. IAOS20084 1. Electronic Questionnaires

5. IAOS20085 Cost / Benefits overview Intuitively we might think that there were both costs and benefits Two dimensions –Respondent Burden –Quality Graphically

6. Interventions Burden Electronic X Paper

7. IAOS20087 Respondent Burden Reductions –Burden of logging in – security,.. –Skipping not applicable questions Increases –Repeated entry for edit failures

8. Interventions Quality Electronic X Paper

9. IAOS20089 Quality Increases –Key entry – no recognition issues –Handwriting / Within box marks /.. –Prompt for non response / …… Reductions –Constraints on response –Repeated entry –False response (to get around edits)

10. IAOS200810 Improved Timeliness (of response) Internet completions can arrive prior to Census day (quality?) More dynamic survey taking?

11. IAOS200811 Reduction in Cost Key entry completed by respondent –So no scanning / recognition IF QUALITY IMPROVES –Less back office correction Still cost of internet software set up etc.

12. IAOS200812 2. Challenges

13. IAOS200813 Interventions How can we improve the electronic questionnaire – balance mode effects Introduce interventions / edits Similar to Interviewer controlled, but less variability (machine not human driver?)

14. IAOS200814 Spectrum of Interventions Spectrum from Low impact / sensible / simple / expected ……. High Impact / burdensome / complex

15. IAOS200815 Spectrum of changes 1.Repeated data automatically entered 2.Skip sections not applicable (filter) 3.Constrain when only ONE response required 4.Range check 5.Highlight item non response 6.Within and between record checks Voluntary or Mandatory check (hard / soft)

16. IAOS200816 Model Theoretical models developed –To be followed with field work –Current draft of questionnaire Simplified with –Working adults (35 questions) –Children (20 questions) Measurements –Time to complete = Burden –Non response = Quality (Inverse)

17. IAOS200817 Model - Burden T = KS + B + L + I + C KS - Key Strokes B – Tick Boxes L – Lines for write in questions I – Instruction lines C – Cognition levels – understanding and recall

18. IAOS200818 Model – Non Response NR = ∑ NR q A simple score function NR q is the 2001 question non response rates NR qi is applied for each intervention i

19. IAOS200819 3. Outcomes

20. IAOS200820 Response (Time) v Interventions

21. IAOS200821 Time to Complete Time reduced with (small effect) –Repeated questions –Skipping –One response (radio button) Time Increased (larger effect) –Range check –Non response - soft edit –Non response – hard edit

22. IAOS200822 Time to Complete More effect for working adults than children –Resulting from more questions Is there an effect for perceived saving in addition to the actual saving?

23. IAOS200823 Non Response v Interventions

24. IAOS200824 Non Response Only small effect for first five interventions –One repeated question –Few skips –Radio button (does not effect non response) Gains –Improve item non response –see Williams 2006 Australian Census

25. IAOS200825 Where is the Balance? YES to –Repeated responses –Filters to skip –Radio Buttons (one tick) Positive for response and burden

26. IAOS200826 Where is the Balance? BUT –Range checks –Item non response soft check Positive for improved response and Negative for response burden

27. IAOS200827 Where is the Balance? NO to –Item non response hard check Negative for response and burden

28. IAOS200828 4. Conclusion

29. IAOS200829 Drivers Meet user needs –Multi Mode: Duality of delivering / collecting data in a range of formats Tension between consistency and relevance –Mode Effect: Same questionnaire versus advantages from electronic questionnaires

30. IAOS200830 Internet Offers Significant cost advantages of high Internet take up for Census –High field and processing costs BUT internet not universal AND Internet respondents are not representative of population –Younger, larger household, mobile, studying and in labour force, higher income, higher qualifications, … [Williams 2007]

31. IAOS200831 Electronic Questionnaires Improve quality of response and save respondent costs with some checks Net reduction in quality of response with too many check Quality and Respondent Burden balance depends on number of applicable questions Good quality improvements with soft edits for range and non response checks

32. IAOS200832 Thank You