Copyright 2010, The World Bank Group. All Rights Reserved. Questionnaire Design Issues Section B Disclaimer: The examples used are not necessarily good or recommended but are used for illustrative purposes only.

Copyright 2010, The World Bank Group. All Rights Reserved. Structure and Format Layout can: Facilitate data collection, transformation and capture Differentiate between questions, responses, sections and clarify skip patterns Encourage better response rates 2

Copyright 2010, The World Bank Group. All Rights Reserved. Structure and Format 1.Abbreviations and punctuation a.Spell out abbreviations b.Do not use commas or periods inappropriately c.Abbreviated wording only with trained interviewers d.Underline or bold critical words 2.Response categories a.Margin-justify pre-coded and self-coded responses b.Margin-justify code boxes for write-in responses 3.Space a.Avoid cluttered appearance b.Provide enough space for write-in responses c.Avoid too many thick black lines - try thin gray lines 3

Copyright 2010, The World Bank Group. All Rights Reserved. Structure and Format 4.Answer spaces a.Should be clearly identifiable for each question b.Use columnar or tabular format for answer spaces c.Ensure each response category box is clearly identifiable 5.Interviewer working space a.Provide space and instructions on the questionnaire for any calculations to be performed 6.Differentiation a.Differentiate between the types of answers required and provide guides to skip patterns whenever possible. 4

Copyright 2010, The World Bank Group. All Rights Reserved. Structure and Format 7.Color a.Drop-out colors for image data capture b.Differentiate between sections and clarify skip patterns c.Colorful graphics can help motivate respondents to complete self-enumerated questionnaires 8.Question numbering a.Each question should be numbered sequentially (1, 2, 3) b.Each section should be numbered sequentially (I, II, III, A, B, C…) 5

Copyright 2010, The World Bank Group. All Rights Reserved. Structure and Format 9.Instructions a.Should be placed (in order of preference): –Above the question –At the beginning of a questionnaire section –On the front of the questionnaire –On a separate sheet or in an instructional booklet b.Instructions should have a differentiated and standardized font, location or separation (box) 10.Language a.Various languages may affect spacing and layout b.Translations should be rigorously checked 11.Consistency a.Formatting decisions must be consistent throughout. 6

Copyright 2010, The World Bank Group. All Rights Reserved. Processing Considerations Keyed data entry 1.Appropriate number of code boxes per response 2.One box per character 3.Answer boxes margin justified 4.How many responses should be marked for each multiple choice question? 5.How will “not stated” be marked? 6.How will questions not applicable to a particular respondent be marked? 7

Copyright 2010, The World Bank Group. All Rights Reserved. Processing Considerations Scanned data capture 1.Forms must be kept in good condition 2.If multiple pages, must be separated (bar codes, cutters) 3.Paper color, printing clarity, drop out colors 4.Sufficient white space around answer boxes 5.Proper writing instrument used to complete forms 6.More rigorous testing of questionnaires, early enough to make changes to software, equipment, and/or forms. 7.Compare against a keyed sample 8

Copyright 2010, The World Bank Group. All Rights Reserved. Questionnaire Quality Issues with Scanned Data Capture Questions should be pre-coded as much as possible.  The major areas requiring quality control are : 1.Paper (color, weight and size) 2.Printing (color and clarity) 3.Format (page layout, response position(s), and clear zones) 9

Copyright 2010, The World Bank Group. All Rights Reserved. Questionnaire Quality Issues with Scanned Data Capture Interviewers, and respondents for self-enumeration, must be trained, instructed in how to complete the form correctly. 1.OMR bubbles 2.OCR characters 10

Copyright 2010, The World Bank Group. All Rights Reserved. Methods for Drafting Questionnaires 1.Unstructured Individual Interviewing 2.Qualitative Group Interviews 3.Participant Observation 11

Copyright 2010, The World Bank Group. All Rights Reserved. Unstructured Individual Interviewing A discussion of the proposed survey topics Guided by a topic outline Used to gain insights into questionnaire structure Time required: weeks, Cost: Salary and per diems 12

Copyright 2010, The World Bank Group. All Rights Reserved. Qualitative Group Interviews Informal discussions of selected topics Aids in developing conceptual framework and data specification Can also be used post survey to interpret the data. Time: months, Cost: Salary and per diems 13

Copyright 2010, The World Bank Group. All Rights Reserved. Participant Observation Research Anthropological technique for gathering information Field researchers live with the persons of interest. Can be used to ensure questionnaire will provide enough information and help phrase questions Time: 6 months - 1 year, Cost: Support of researcher and incidentals 14

Copyright 2010, The World Bank Group. All Rights Reserved. Procedures for Testing the Questionnaire Draft 1.Informal Testing - subjective evaluations of the questionnaire, 2.Formal Testing - relies on statistical evaluations a.Pilot Studies b.Split sample tests 15

Copyright 2010, The World Bank Group. All Rights Reserved. Informal Testing A questionnaire field test with a small number of interviews Depends on subjective information provided by interviewers and observers. Not designed to be evaluated on a rigorous statistical basis. Usually done as a first test of the draft or as a final step to ensure that revisions work well. 16

Copyright 2010, The World Bank Group. All Rights Reserved. Informal Testing: personnel, cost and time Coordinator. Interviewers Personnel to evaluate the results. 50 to 300 respondents. Respondents are selected purposively rather than randomly. Can evaluate entire questionnaire or a portion. More than one questionnaire evaluated with split-sample testing. Time: Dependent on questionnaire and sample but operational aspects around 4 months. Cost: also variable but generally low. 17

Copyright 2010, The World Bank Group. All Rights Reserved. Formal Testing Relies on probability sampling and results are evaluated on a rigorous statistical basis. 1.Pilot study - a prototype of the survey conducted to observe all of the proposed survey operations working together. 2.Split-Sample Test - conducted to evaluate two or more alternate versions of the questionnaire. 18

Copyright 2010, The World Bank Group. All Rights Reserved. Formal Testing - Pilot Study Reduces time between final survey and results. Often little time for a pilot test, evaluation and refinement of survey tools and procedures. Large, complicated, or repetitive surveys need a pilot study 19

Copyright 2010, The World Bank Group. All Rights Reserved. Pilot Study – personnel, time and cost Probability sample necessitates larger numbers of personnel, including a sampling statistician, data processing staff and analysts. Different respondents should be used for pilot and survey. Not finished until data analysis is completed. Evaluation plan must be decided in advance. Time and costs are variable but are likely greater than for an informal test. 20

Copyright 2010, The World Bank Group. All Rights Reserved. Formal Testing - Split-sample Experimental design incorporated into the data collection process. Used as methodological studies to advance knowledge of questionnaire design and the survey research process. Personnel requirements are similar for a pilot test but require more sophisticated statistical and analytical expertise. Time and cost are similar to pilot test but the time may be distributed differently for a split-sample with more planning and less editing and coding. 21

Copyright 2010, The World Bank Group. All Rights Reserved. Techniques for Evaluating a Questionnaire 1.Investigating Respondents Interpretations of Questions 2.Observation and Monitoring of Interviews 3.Learning from Interviewers 4.Using Record Checks 5.Response Analysis Surveys 22

Copyright 2010, The World Bank Group. All Rights Reserved. Questionnaire Review Worksheet Exercise Questionnaire organization Question wording Question grouping Skip patterns Coding Questionnaire length 23