Training course on developing and using questionnaires for agricultural surveys Field Testing Field Testing Marco Ballin Istanbul, 18-21 July 2011

Field Testing Field testing methods are those methods used to asses the questionnaire in real conditions They are usually implemented by a larger number of units than the pre-field tests. These method can be implemented either in ad hoc test, in a pilot survey or during in ongoing surveys Istanbul, 18-21 July 2011 2

Interviewer debriefing Respondent debriefing Field Testing The following are some of the techniques available to the questionnaire designer to test the questionnaire: Behavior coding Interviewer debriefing Respondent debriefing Intense interviewers or follow-up interviews Istanbul, 18-21 July 2011 3

It can be both for CATI and face to face surveys Field Testing Behavior coding It consists in classifying the interviewer/respondent interaction in order to evaluate the quality of the questionnaire. It can be both for CATI and face to face surveys With this method the behavior and reaction during the interviews have to be classified in a predetermined coding system The analysis of the frequencies of reactions to each question can highlight the occurrence of problems due to administration, wording, questionnaire flow, etc To implement this method you need to train a “coder”; that is a person that analyze the interview and classify the behavior using given scheme Istanbul, 18-21 July 2011 4

Interviewer question–reading codes Field Testing Examples of coding scheme Interviewer question–reading codes Indicator description Exact Interviewer reads the question exactly as printed Slight change Interviewer reads the question changing a minor word that does not alter question meaning Major change Interviewer changes the question such that the question meaning is altered. Interviewer does not complete reading the question. Istanbul, 18-21 July 2011 5

Respondent behaviour codes Field Testing Examples of coding scheme Respondent behaviour codes Indicator description Interruption with answer Respondent interrupts initial question-reading with answer. Clarification Respondent asks for repetition or clarification of question, or makes statement indicating uncertainty about question meaning Adequate answer Respondent gives answer that meets question objective. Qualified answer Respondent gives answer that meets question objective, but is qualified to indicate uncertainty about accuracy. Inadequate answer* Respondent gives answer that does not meet question Don’t know Respondent gives a “don’t know” or equivalent answer. Refusal to answer Respondent refuses to answer the question. Istanbul, 18-21 July 2011 6

To carry out a statistical analysis of the problem met; Field Testing This method allows: To reveal a sizeble number of question problems (the number depends on the coding list categories) To carry out a statistical analysis of the problem met; To be implemented in real conditions Main difficulties and limits are: It is quite expensive and time consuming It describe the problems but doesn’t help in finding out the underlying reasons Istanbul, 18-21 July 2011 7

Interviewer debriefing Field Testing Interviewer debriefing It is a testing method applied to interviewers in order to have their feedbacks on the performance of the questionnaire It is useful to capture the weakness from the point of view of interviewers point of view Interviewers can report their feeling about respondets difficulties too Istanbul, 18-21 July 2011 8

Interviewer debriefing Field Testing Interviewer debriefing The most common debriefing method is the group discussio after the field work The discussion can be driven by some outlines prepared to cover the most important issues This method can be easily applied in case of CATI surveys where interviewers are grouped in the same place In case of face to face interviews, interviewers could be spread all over the country. In this case the debriefing could be carry out using a questionnaire Istanbul, 18-21 July 2011 9

Respondents debriefing Field Testing Respondents debriefing It is analogue to the Interviewers debriefing method, but it is applied to respondents Usually this method allow to find out probles caused by: Wording Definitions Length Clarity Availability of data sources (important for business surveys) Istanbul, 18-21 July 2011 10

Respondents debriefing Field Testing Respondents debriefing Most of times this method consists of a self-administered interview It is necessary that the questionnaire designer are fully aware of the potential problems to develop a set of debriefing questions Istanbul, 18-21 July 2011 11

Intense Interviews or follow-up interviews Field Testing Intense Interviews or follow-up interviews It is a method that shoul hel in providing information about the hidden processes involved in answering the questions: This method can complement the behavior code method because it help in understanding the reasons of the problems that it has identified The method consists in a second semi –structured interview. The objective is to help the respondent to remind : how he/she interpret and understood the questions How he/she arrived to the answer The importance he/she gave to the question … Istanbul, 18-21 July 2011 12

Field Testing Discussion How can we apply some of the hints belonging to these methods in case of lack of resources ? Istanbul, 18-21 July 2011 13