1. How do we assess whether we are improving instrument design? Alice McGee

2. How do we assess whether we are improving instrument design? Using multiple methods to evaluate whether a re-designed travel record was 'better' than the existing one

3. Presentation outline Background to NTS and Travel Record Rationale for re-design Research stages  A - Evaluate the existing record  B - Record re-design  C - Evaluate the new record Findings Conclusions

4. Background to study National Travel Survey (NTS) Measures travel patterns and behaviour  Face to face interview - household level  Seven-day Travel Record - individual level Adult and child versions Nature of information - very detailed

5. Existing Travel Record

6. Rationale Concerns surrounding data quality  High proportions of item missing  Anecdotal evidence suggesting measurement error International evidence supporting alternative designs  Axhausen: Column based approach

7. Research stages A - Evaluate the existing record B - Re-design the record C - Evaluate the new record  to assess whether the new was ‘better’ than the original

8. A - Evaluation methods Quantitative 1. Secondary analysis of survey data Qualitative 2. Workshops with interviewers 3. Cognitive interviews 4. Respondent suggestions 5. Data editor comments

9. B - Re-designing the record Two underpinning elements  Existing literature: theoretical basis for how people approach self-completion documents  Empirical evidence from Stage A Existing literature  Jenkins et al: Readers and Skimmers typology Principles of good questionnaire design

10. B - Aims of the re-design Make the task easier and more straightforward for all types of respondent through:  providing navigational guides to help respondents find their way around the record  improve the way information was organised and displayed NB: unable to address the more conceptual problems

11. B - Main changes to record Colour and layout more user-friendly Instructions brought together Visual guides (arrows) and clearer signposting Pictorial images Tick boxes Column heading posed as questions

12. Re-designed Travel Record

13. C - Evaluation methods Quantitative 1. Comparison of sample of data from existing Record and pilot data (100 respondents) 2. Respondent debriefing questionnaires Qualitative 3. Cognitive interviews 4. Pilot interviewer comments 5. Data editor comments

14. Hypotheses All evidence sources reviewed and triangulated Hypotheses tested at Stage C:  Cognitive respondents would experience narrower range of problems  Problems relating to information organisation and navigation would diminish among cognitive respondents  Systematic review of respondent error would show the new Record produced lower error rates

15. Findings and conclusions All three hypotheses proved correct Triangulation of quantitative and qualitative evaluation methods showed new Record to be ‘better’ than the original Study demonstrates how a combination of evaluation methods can be brought together to substantively conclude the best approach

16. Re-designed Travel Record

17. Future work Evaluation of Event History Calendar approach (EHC)  England Longitudinal Study of Ageing (ELSA)  Retrospective interview following Wave 3  Interactive calendar administered through CAPI Package of work involving multiple evaluation methods (quantitative and qualitative)

18. EHC - Evaluation methods Quantitative 1. Behaviour coding 2. Secondary analysis of survey data 3. Respondent debriefing questionnaires (closed responses) Qualitative 4. Interviewer diary 5. Respondent debriefing questionnaires (open responses) 6. Quasi transcripts 7. Behaviour coding (supporting notes)

19. References Axhausen, K (1995) “Travel Diaries: An annotated catalogue, 2nd edition” Jenkins, C.R., Ciochetto, S. and Davies, W. (1992) 'Results of cognitive research on the public school 1991-92 field test questionnaire for the schools and staffing survey'. Unpublished Jenkins, C.R. and Dillman, D.A. (1995) 'Towards a theory of self- administered questionnaire design' in Lyberg et al (1997) 'Survey Measurement and Process Quality' Wiley McGee, A, Gray, M and Collins, D (2006) “NTS Travel Record Review Stage 1” DfT, web only McGee, A, Gray, M, Andrews, F, Legard, R, Wood, N and Collins, D (2006) “NTS Travel Record Review Stage 2” DfT, web only