Copyright © 2015 Inter-American Development Bank. This work is licensed under a Creative Commons IGO 3.0 Attribution-Non Commercial-No Derivatives (CC-IGO BY-NC-ND 3.0 IGO) license ( and may be reproduced with attribution to the IDB and for any non-commercial purpose. No derivative work is allowed. Any dispute related to the use of the works of the IDB that cannot be settled amicably shall be submitted to arbitration pursuant to the UNCITRAL rules. The use of the IDB’s name for any purpose other than for attribution, and the use of IDB’s logo shall be subject to a separate written license agreement between the IDB and the user and is not authorized as part of this CC-IGO license. Note that link provided above includes additional terms and conditions of the license. The opinions expressed in this publication are those of the authors and do not necessarily reflect the views of the Inter-American Development Bank, its Board of Directors, or the countries they represent.

Questionnaires and Other Instruments Beatriz Godoy, Mario Navarrete Sistemas Integrales

Questionnaires 3 Everyone thinks that designing a good questionnaire is very easy

Questionnaires Producing a good questionnaire is a critical part of the design of the survey. The quality of the measurements obtained by the survey is directly affected by the design of the questionnaire. Even small errors or inaccuracies in the wording or the flow of the questions can lead to unwanted biases. 4

Questionnaires Identifying the right people that should be involved in the development of questionnaires to ensure:  That the objectives of the impact evaluation study are covered  There is comparability with pervious or similar surveys, within and/or outside of the country  There is an appropriate physical format of the survey, to facilitate:  Its application by the enumerators  The design of the data entry program 5

Questionnaires Identifying the relevant people…  Ministries  International agencies  Academics and other experts  Non-governmental organizations  Others 6

Questionnaires Adaptation of standards questionnaires…  They must be adapted or changed without losing technical validity and respecting the established protocols  Language  Visual material  Instruments (physical material) 7

Stages of Questionnaire Development First version Pre- test Pilot test Final version 8

Development of a First Version

Does it Cover All the Topics of Interest? Discussion with the relevant members of the study Identification of indicators Review of previous surveys Review or relevant qualitative studies Review of standard tools that will be applied Biometric measurements 10

Too Much Ambition Can Be a Bad Thing It is important to capture all the information necessary to perform the impact evaluation analysis, however, the questionnaire must be as short as possible to avoid respondent fatigue. 11 Recommendation: To avoid respondent fatigue, each individual interview should not take more than 90 continuous minutes

Selection of the Respondent Each question should be directed at the person who is best informed about the topic Individuals: Each responds for themselves, proxy respondents should be avoided (except for minors) Households Doctors Patients … More than one person in the household (health center, etc.) may be interviewed 12

Ensure Understanding of the Questions They must be asked as they are written Avoid ambiguous wording One question at a time Clarity about whom the question refers to Explicit period of reference 13

Ensure Understanding of the Questions Explicit units of measure Avoid academic phrases that are incomprehensible to the interviewee Use simple language (avoid double negatives) The questions must be culturally sensitive and appropriate 14

Example – Difficult Understanding 15

Examples – Clarity About Whom the Question Refers To 16

Examples – Clarity About Whom the Question Refers To 17

Flow of the Questions They must be organized in a logical order They must only be asked when they are applicable. This is achieved with a good structure of jumps or skips The decision of when and to whom to ask a question must never be left to the interviewer 18

Physical Design of the Questionnaire Types of questions With predefined options Numeric Open response (text) 19 Recommendation: avoid perverse incentives

Physical Design of the Questionnaire Numbering of the questions 20

Physical Design of the Questionnaire Graphic organization: Free form 21

Physical Design of the Questionnaire Graphic organization: Grid 22

Physical Design of the Questionnaire Flap 23

Physical Design of the Questionnaire Editorial standards Jumps or skips 24

Physical Design of the Questionnaire Editorial standards Jumps or skips 25

Physical Design of the Questionnaire Editorial standards Use of uppercase/lowercase 26

Physical Design of the Questionnaire Use of uppercase/lowercase 27

Physical Design of the Questionnaire Editorial standards Use of uppercase/lowercase 28

Physical Design of the Questionnaire Excel or similar is much better than Word or similar Use of formulas for numbering Automatic translation 29

Example of Dictionary for the Automatic Translation 30

Questionnaire Identifier It is very important to define the identifier that will uniquely identify each of the households (health centers, etc.) that participate in the survey. This should appear very explicitly as the first datum on the cover of the questionnaire. 31

Questionnaire Identifier 32

Questionnaire Identifier 33

Questionnaire Identifier 34

Panel Surveys Design the instruments taking this into consideration GPS coordinates Phone numbers of members of the household Phone numbers of relatives, neighbors, or friends References to the address of the home 35

Pre-test

Verification with Reality: Pre-test  Questionnaire The analysts and designers must participate The sample of households does not need to be large or randomly chosen Emphasize the quality of the exercise more than the quantity It is an iterative process 37

Verification with Reality: Pre-test 38 First questionnaire version Household surveys Discussion about results Modification of the questionnaire

Verification with Reality: Pre-test 39 Household surveys Discussion about results Questionnaire modification First questionnaire version Version after the pre-test

Pilot Test

Verification with Reality: Pilot Test  Expected results Training program validated Field procedures validated Instruments validated Data entry program tested and validated 41

Verification with Reality: Pilot Test  Language While it is true that the the pre-test is done with the questionnaire in the language that the research team understands, the pilot test requires that this be translated: At least to the official language(s) of the country 42 Recommendation: Verify the translation by doing it in both directions by independent translators

Verification with Reality: Pilot Test While it is true that the objective of this field exercise is to test all of the survey instruments and procedures, one of the important points is to re-test the questionnaire 43

Manuals

Interviewer manual Supervisor manual Supervision procedures 45

Interviewer Manual  It must be short but exhaustive  Advice about what to do in certain complicated situations  Relation with the supervisor  Include examples in the case of very complicated modules  Do not put explanations for trivial questions such as sex 46

Supervisor Manual  Relationship with the surveyors under their supervision  Relationship with the central team  Instructions about how to make the lists of enumeration areas  Selection of households for the survey  How to read the associated mapping  Relationship with the data entry operator  Interpretation of the error messages of the program 47

Other Instruments  Supervision procedures  Questionnaires to collect test information  Specific protocols  Etc. 48