Greg Griffiths for: Rob Burnside Emma Farrell ICES 3 Montreal June 2007 A Coherent Approach to Questionnaire Design Standards, Trends, and Implementation at the ABS

 How ABS enterprise questionnaire design standards and guidelines are:  implemented organizationally  created and maintained;  scope and content of the standards  trends - changes over time:  role of form design methodologists  new modes and questionnaire structures; and  the Total Respondent experience Overview

Mandatory use of agreed standards for in-scope instruments detailed in Corporate policy manual. Common conventions, look and feel. Bringing about use of best practice. Organization arrangements give responsibility to areas for different aspects of final forms Form development process is supported by:  Standards manual containing guidelines and examples available on the ABS intranet;  Organizational structures incl survey-specific meetings, an annual review of core questions; and  A forms image database with workflow/ sign-off process; Holds images and linked discussion for interim, current and historical forms back to 1998 Big Picture

Methodology Division Forms Consultancy Group (FCG):  maintain and extend standards and guidelines;  understand mode effects and try to minimize;  R&D into form design (all modes) Operational sections - manage survey process and produce output Economic Statistics Data Centre - includes a specialised area constructing forms to standards and coordinating testing Conceptual sections - establish user needs Economic Standards Section - consistency of core economic questions and classifications National Accounts Branch - to ensure inputs for Australian National Accounts Responsibilities of areas involved

FCG Despatch and Collection Unit (DCU) Economic Standards and Classifications Section (ESCS) Collection Management Unit (CMU) Provider Contact Unit (PCU) Business Statistics Centres (BSCs) Administrative Data Acquisition Unit (ADAU) Household Statistics Centres (HSCs) Population Survey Operations (PSO)

Form standards cover: format, structure and question wording principles, common components; all modes - paper mail out, fax, CATI, Excel, TDE/IVR detailed specifications and examples for different types of questions and forms; contact letters and s Development and evaluation standards cover: observational studies, usability testing, cognitive testing... suitability of study design for answering development questions. Scope and content

Relies on maintaining knowledge of ABS and international theoretical and applied research and experience; Maintenance and planned updating of standards based on: Advice on development methods and approaches Emerging issues Involvement in field testing Organizing experiments to examine aspects of design form length effects, scanning design, effectiveness etc. Urgency of development need Creation and maintenance of standards

Trends: 1 - Evolving role of ABS form design methodologists (FCG) ctd Pre 2000Policing layout standards 2000 Methodology standards established. Need to examine broader measurement error recognized More modes and more development methods 2002/03Total Approach Management and a wider view - every interaction with the respondent is significant - appropriate, consistent and effective contact 2007 Multimodal Data Collection project funded 2007 Move of household based collections into same branch as FCG. New structure under consideration.

More modes:  structure, presentation and wording changes to keep things the same conceptually  implies more, different testing and development techniques;  increase testing and creation cost  form design increasingly intertwined with IT capability and applications More question structures:  Complex  Subjective (i.e. non-economic)  More like household questionnaire questions Trends - 2: more modes and question types

More development techniques  right technique to explore the right thing  more use of cognitive and exploratory methods  Post Enumeration Studies, behavior coding, card sorting, interviewer debriefing  Expert review  Classic usability testing for eform and CATI  Classic experimental design where appropriate Trends: 3 - More development techniques

Every interaction with a data provider is important. The form is more than the form.  pre approach and covering letters,  brochures and explanatory material,  Telephone, fax and contacts The response experience is more than filling in the form:  Response enhancing techniques - social influence theories  mode of choice, incentives …. Trends: 4 - Total Approach Management

Strong senior management recognition of the importance of form design which is reflected in organizational structure. Formal form development process with defined roles and sign-off. Corporate systems ensuring communication of standards. Visibility of deliberations & decisions. Summary

A specialized group of methodologists exists outside operational areas – Responsible for standards and development methods – Involved in day-to-day form development as well as R&D into respondent related measurement error – Centralization of methodology for all modes and methods ensuring consistency – Maintains critical mass of expertise – Maintains a strong commitment to research and the communication of results / new developments to operational areas