1. Designing and Using a Behaviour code frame to assess multiple styles of survey items Alice McGee and Michelle Gray

2. Presentation outline Background to study Aims of research Methodology Designing a behaviour code frame Using the behaviour code frame Analysing the data Lessons learned

3. Background to study English Longitudinal Study of Ageing (ELSA) Dependent Interviewing (DI) Two types of data item  Feed Forward (DI)  Non-Feed Forward (non DI) Little evaluation of the impact of DI on data quality conducted to date

4. Aims of research Research aims:  To assess how DI affects data quality  To explore how Rs react to feed-forward phrases  To find whether this varies by nature and sensitivity of topic Methodological aim:  To explore the combination of CARI and Behaviour Coding as methodological tools

5. Methodology Computer Assisted Recorded Interviewing (CARI)  Computer acts as a sophisticated tape recorder  Unobtrusively records interaction Behaviour Coding  Codes systematically applied to interviewer- respondent behaviours  Uncover and assess problems with questions Two methods combined for this study

6. Designing the code frame

7. Principles for good design Code frame adapted from Cannell et al (1989) Short and straightforward Few, easy to apply codes Discrete Broad rather than specific

8. Behaviours coded Question asking behaviour for interviewers Immediate response behaviour for respondents Whether partner intervened (concurrent interviews) Final outcome of the entire exchange

9. Two behaviour code frames Two code frames designed:  DI (feed-forward) items  non DI (non feed-forward) items First level exchange (initial utterance) Code what occured before other person speaks

10. Code frame

11. Behaviours coded Interviewer/Interviewer feed-forward Respondent/Respondent feed-forward Whether partner intervened Final outcome One code per behaviour

12. Interviewer codes Exact Wording/Slight Change01 *Major change02 *Omission03 *Question became a statement04 *Inaudible Interviewer/Other05 Not applicable 99 *denotes where notes must be made

13. Interviewer feed-forward codes FF item read as worded/slight change01 *FF statement became a question02 *FF question became a statement03 *Other major change04 *Omission05 *Inaudible Interviewer/Other06 Not applicable 99 *denotes where notes must be made

14. Respondent codes Adequate Answer 01 *Inadequate Answer/Elaboration 02 *Clarification 03 Question Re-Read 04 Don't Know 05 Refusal 06 *Inaudible Respondent/Other 07 Not applicable99 *denotes where notes must be made

15. Respondent feed-forward codes *Affirmed FF item - adequate 01 *Disputed FF item - adequate 02 *Inadequate Answer/Elaboration 03 *Clarification 04 Question Re-Read 05 Don't Know 06 Refusal 07 *Inaudible Respondent/Other 08 Not applicable 99 *denotes where notes must be made

16. Partner intervention codes *Yes 01 No 02 Not applicable (no partner present)99 *denotes where notes must be made Code used for where the respondents partner intervened and subsequently answered for the respondent

17. Final outcome codes Adequate Answer 01 *Inadequate Answer 02 Don't Know 03 Refusal 04 *Inaudible/Other99 *denotes where notes must be made Coding whether the final answer meet the objective of the question

18. Technical details

19. CARI equipment Equipment testing  External microphones CARI built into Blaise program Recording switched on and off at relevant items Sound files automatically generated and saved Sound files removed from interviewer laptops  Macro run  Data sticks (USB)

20. Behaviour coding system Conducted within Blaise Coding program designed for this purpose  Weststat testnote software Three windows displayed simultaneously  Blaise interviewing screen  Coding entry screen  Sound file (.wav) Automatically routed through interview Tags to skip to relevant data items

21. Using the code frame

22. Sound file

23. Blaise interviewing screen

24. Coding program

26. Data preparation and analysis

27. Organising the data Two types of data  Behaviour codes (quantitative)  Coder notes on non-standard behaviours (qualitative) All data automatically stored in Excel tab delimited file One Excel file produced for each coder Excel files amalgamated Exported into SPSS

28. Data preparation More cleaning than expected Two main problems:  Duplicate files (limitations of system used)  Incorrect code frame used at interviewer and respondent behaviours (DI and non DI items)

29. Analysing the data SPSS Frequencies and crosstabulations Coder notes provided additional context Very small base sizes at some items due to routing

30. Advantages and disadvantages of our approach and lessons learned

31. What worked CARI  Unobtrusive in nature  Minimal impact on interviewers and respondents Behaviour coding  Able to run statistical analyses  Able to draw conclusions  Method of coding easier than paper (routing)

32. What didn’t worked CARI  High number of inaudible or hard to hear cases (1/3 of respondents)  Purchased speakers to help Next time…  Fully re-test microphones  Probe respondents reasons for not giving consent to being recorded

33. What didn’t worked Behaviour coding  Lengthy and costly process Coding (approximately 45 mins per interview) Data cleaning  Over complex code frame  Coding method found cumbersome, limited and error prone  Coder judgement not measured

34. Next time... One code frame only Build in sufficient time for each stage Clear rationale for behaviour coding Inter-coder reliability test (Kappa score) Adequate sample for uncommon questions Create more sophisticated, less error prone coding system

35. Discussion & Questions...