**This instructor is slightly customized for each lesson. Designing Your Survey

By the end of this lesson you will be able to: Define the 6 goals of the questionnaire survey. Differentiate between a biased and an unbiased survey question. Develop survey questions from preliminary objectives.

Design a survey that: has appropriate sample sizes and strategy measures the impact of your Pride campaign on SMART objectives for each target audience measures exposure to your Pride campaign activities

SESSION TOPICS AND TIMING Learning to Detect Bias and Poorly written questions Reading Activity Writing the best possible survey questions Thurs Creating Questions from Preliminary Objectives Drafting Your Stages of Behavior Questions Determining Sample Sizes and Calculating Your Sites Sample Size Assignment, Rubrics and Research Plan Review Friday SESSION TOPICS AND TIMING

Activity 1: Learning to Detect Bias and Poorly Written Questions (Basics) PURPOSE: To practice recognizing “bias” on questions and identify what makes a question biased 45 minutes Defining impact is key here – provide example tied to smart objective

Revisit the Theory of Change Defining impact is key here – provide example tied to smart objective

INSTRUCTIONS: ACTIVITY 2: READING Read the Designing Your Survey Lesson Content (pp 321-336) – 20 minutes Proceed to your workshop group (3-4 persons per groups) and answer the guiding questions in pages 295-298 – 10 minutes Record your groups responses and report your results (5 minutes) ACTIVITY 2: READING

Why Conduct an Interview Survey Surveys are research tools that provide quantitative information about people in your target area. The systematic sampling will make your respondents “representative” of the population of interest. Surveys are ideal tools to measure change in your target audiences because you can measure the “baseline” and post-campaign levels. The difference between baseline and post-campaign survey results provides an estimate of the impact of your campaign on that variable.

Questionnaires and your Pride Campaign You will implement 2 surveys during your campaign: During the planning phase (baseline) At the conclusion of your camp impact.

Goals for your survey 1. Describe your Target Audience(s) “ 90 % of all fishers surveyed are male and 60 % of those are over the age of 40”

Goals for your survey– pre campaign 2. Assign Respondents to their Appropriate Stage-of-Behavior-Change “Among fishermen surveyed, 5 percent are currently setting rattraps on their boats, 5 percent are currently thinking about setting rattraps...”

Goals for your survey– pre campaign 3. Establish a Baseline for your SMART Objectives “30% of Fishers know that rats get to Serena on fishing boats” can be the basis for setting the SMART Objectives: “By 2013, the percentage of Fishers who know that rats travel to Serena on their boats will increase from 30% to 50% Insert an associated knowledge or attitude objective appropriate for your site

Goals for your survey– post campaign 4. Measure the impact of your PRIDE campaign in achieving your SMART Objectives Control site IC Interpersonal communications A Attitude powerful way to do this, is to use a control group will be discussed in detail below. K Knowledge

Goals for your survey– pre campaign CARE FOR THE LASTING PRIDE 5. Measure the Audience Exposure to your Pride Campaign and Attribute Effects to your Pride Campaign A campaign to protect Tagbilaran City’s underwater kingdom

Goals for your survey– pre campaign 6. Ground Truth – test your assumptions about community, strategy, and ToC "99% of community members know about the importance of keeping rats off of the island”… what does this tell you about your ToC? If you thought knowledge was low in your target audience about rats, but you find that it is very very high as in the above example, then this means your campaign may need to be more focused on attitude or IC changes and not knowledge.

DESIGNING YOUR QUESTIONNAIRE

A Note of Caution… Developing and implementing quantitative surveys is a lot of work bias questions ask questions of the wrong people, or at least not enough of the “right” ones, of your critical target audience(s) ask too many questions make mistakes in choosing your samples - your baseline and post-campaign samples are not comparable make mistakes during data entry  you enter the wrong data into the wrong variable and end up with meaningless results We are going to provide the tools you need to design but there are a number of key challenges. Will practice these challenges so you feel comfortable dealing with them bias questions—a great deal of care is needed when crafting questions and when asking them ask questions of the wrong people, or at least not enough of the “right” ones, of your critical target audience(s) ask too many questions make mistakes in choosing your samples, so that your baseline and post-campaign samples are not comparable to each other make mistakes during data entry so that you enter the wrong data into the wrong variable and end up with meaningless results

Biased Questions The most important challenge of survey questions is that they can be BIASED: Biased questions mean: they make respondents tend to pick one response over another because of the way the question is worded or the way the response options are offered. Refer to the lesson plan for examples and talk through each of the questions and the alternatives for each bold faced section on the slide in the lesson content.

Biased questions: Biased Questions Try to educate Assume what they ask (are leading) Are double-barreled, asking 2 questions at once Are ambiguous or confusing Refer to the lesson plan for examples and talk through each of the questions and the alternatives for each bold faced section on the slide in the lesson content.

Key Elements of Good Survey Questions? Unbiased – not leading or pushing respondents towards a specific answer Tied to Objectives Time bound Clear Not Judgemental Refer to the lesson plan for examples and talk through each of the questions.

Writing good survey questions Other important criteria for good survey questions: Include a relevant time frame Use filter questions as needed Don’t overlap (e.g. age ranges) Use appropriate language for the respondents Refer to the lesson plan for examples and talk through each of the questions. Include a relevant time frame – “In the last six months, have you seen anyone fishing inside the MPA?” Use filter questions as needed - Don’t overlap (e.g. age ranges) Use appropriate language for the respondents

Activity 3: Writing the Best possible Survey Questions Purpose: To Practice writing “good” survey questions and appropriate response choices Instructions: (Turn to pages 299-310) Work in pairs Read through each of the survey questions assigned to you and the corresponding responses found For each of the question, do the following: Identify what makes this as poorly written questions

Survey Questionnaire #1 Original Survey Question/Poorly Written Question Do you know what the Ministry’s law on legal size of fish that can be killed is about? YES NO Not Sure Rewritten Survey Question Does the Ministry have a law about the legal size of fish catch? If yes, please tell me what you think the legal size of fish catch is? YES NO Not Sure The question is biased and educates the respondent as to what the law is about. The “good” survey question is rewritten so it is unbiased. It is clear with either a target audience response being acceptable if they response “yes” or “no”. Finally, the rewirtten question does not educate the respondent

Survey Questionnaire #2 Original Survey Question/Poorly Written Question Do you agree that people should no longer slaughter elephants just for their tusks? YES NO Not Sure Rewritten Survey Question I am going to read you a statement and I would like for you to decide if you strongly agree, agree, disagree, strongly disagree or if you have no opinion: “Should people be allowed to kill elephants for their tusks”? Strongly Agree Agree No Opinion Disagree Strongly Disagree The poorly written question is biased and (1) suggests that the target audience response should agree with the statement, it also uses the emotional language of “slaughter” and “just”. The rewritten question is unabiased and does not lead the respondent to answer. It is also void of emotional language.

Survey Questionnaire #3 Original Survey Question/Poorly Written Question For what reasons have you gone into the Bungo Protected Area? Tell me all that Apply Hunting NTFP Farming Other Rewritten Survey Question I am going to read you a list of possible reasons that people sometimes go into Bungo Protected area. I would like you to tell me for each reason whether you have gone into the Bungo PA for that reason during the past 6 months. Reasons Yes No Not Sure Hunting Gathering plants for medicines or food Farming Any other reason, please specify The poorly written question is biased in three ways (1) it is not time bounded, (2) assumes that they have gone into the protected area, and finally with the target audiences responses, the survey will (3) likely end up with a lot of “other” responses. As a result, it introduces bias because if they don’t mention one of the reasons you are interested in, you don’t know if they didn’t do it, or just didn’t think to mention it. The rewritten question is unbiased, time bounded (in the past six months) provides an option for respondents to say they have not gone into the Bungo PA for any reason.

Survey Questionnaire #5 Original Survey Question/Poorly Written Question Have you purchased bushmeat? YES NO Not Sure Rewritten Survey Question Have you purchased or bartered for bushmeat, or meat from the wild animals, at any time in the past 6 months? YES NO Not Sure The poorly written question is not time bound, so you would not be able to distinguish between someone who purchased it a year ago, but not from someone who purchased it recently (or during the timeframe of your campaign). It is also not very specific, as bushmeat can mean different things to different people and can be be bartered. The rewritten question is unbiased, precise and timebound.

Survey Questionnaire #6 Original Survey Question/Poorly Written Question How many times would you say that you have gone into the Jamu Protected Area? Never 1 to 4 times 4 to 10 times 10 t0 20 times Rewritten Question In the last six months, how many times would you say that you have been in this area here (show them the map of the protected area) Never 1 to 4 times 5 to 10 times More than 10 times There are three elements of this question that can be biased. First is that it is not bound and so you wont be able to distinguish between someone who entered the protected area 5 years ago vs. last month (or within the timeframe of the campaign). Second, the protected area language can make people like they area not supposed to be in the area and then may feel not comfortable saying they had been in there. And third, the responses overlap so if they had been in the protected area 4 times, which category would they go in? The rewritten survey is unbiased, time boun, and provided a map of the protected area instead of just saying “protected area”. It also provides mutually exclusive categories for the target audiences responses. It does, however, assume that in this case, the respondents area able to read maps, which may not always be the case.

Survey Questionnaire #7 Original Survey Question/Poorly Written Question Have you talked to anyone about Jamu? YES NO Not Sure Rewritten Survey Question In the last 6 months, have you spoken to anyone in your community about the importance of protected Jamu for the health of the people who live here? YES NO Not Sure The poorly written question is biased. It is not time bound and it is not clear what you want to test regarding interpersonal communication. They could have talked to someone about Jamu, and said that they hate it! The rewritten question is unbiased, time bound, and specifically asks about the type of communication regarding Jamu. In the survey, if this type of question is asked, you may want to follow up with a question about whom in the community they spoke to.

Survey Questionnaire #8 Original Survey Question/Poorly Written Question What are the threats to the Jamu Ecosystem? Fire Drought Poaching Develop-ment Rewritten Question What types of activities and conditions threaten the health and sustainability of important wildlife in Jamu? Fire Drought Poaching Develop-ment Other The poorly written question is biased because it uses language that may not translate to people in rural areas (e.g. ecosystem). It also doesn’t include an “Other” column in the target audiences responses to capture threats that may be included in the list. The rewritten question is unbiased and uses clear language that is relevant for the target audience.

Survey Questionnaire #9 Original Survey Question/Poorly Written Question In the past 6 months, have you ever fished illegally? YES NO Not Sure Rewritten Survey Question In the last 6 months, have you ever used dynamite fishing? YES NO Not Sure The poorly written question is biased and will be sensitive to target audiences as it assumes “illegal” fishing practices. Not many fisherman will agree to tell you that they are fishing illegal for fear of retribution. The rewritten question is unbiased and still asks about illegal fishing behavior but does not identify it as illegal. However, using this same language will be subject on a case to case basis. If the community or respondents have a high sense of dynamite fishing as illegal, it might be good to rephrase the question as to “In the past six months, have you ever seen anyone fishing using dynamite” inside or outside the marine sanctuary?

Survey Questionnaire #10 Original Survey Question/Poorly Written Question Do you think that rats can spread the disease typhoid fever to humans and cause damage to nets? YES NO Not Sure Rewritten Survey Question I am going to read you to you a series of statements about rats. Please tell me if you think they area true, false or you are unsure: Rat spread typhoid fever to humans. Rats cause damage to fishing nets True False Not Sure The poorly written question is a biased question because it is considered a “double barreled” question, meaning it asks two questions at the same time – typhoid fever and damage to nets. The revised version of the question is unbiased, specific and clear. It also provides an opportunity for the target audience respondent to answer only one question at a time.

Survey Questionnaire #11 Original Survey Question/Poorly Written Question Do you agree that overfishing is a problem and that something needs to be done about it? YES NO Not Sure Rewritten Survey Question I am going to read you to you a series of statements and I would like you to tell me whether or not you strongly agree, agree, disagree, strongly disagree or have no opinion: Overfishing is a problem The government should create laws to prevent overfishing The Banga community should support a ban on overfishing Strongly Agree SA A Disagree D Strongly Disagree SD No Opinion NO The poorly written question is a biased question that forces the target audience respondent to feel like they have to agree (do you agree) and also is doubled barreled because it asks whether something should be done about it. The revised version of the question is unbiased and provides all potential target audiences responses, (SA, A, D, SD, NO). It separates the questions about “knowing” whether there is a problem from what should be done about it.

YOUR QUESTIONS?

READING ASSIGNMENTS FOR TOMORROW Read and review the lesson content especially on Biased and Unbiased questions (pages 326-336) 2. Read Lesson Content pages 337-348. Write in a piece of paper 2-3 questions (per participant) that you would want to be clarified or know more. READING ASSIGNMENTS FOR TOMORROW

DESIGNING YOUR QUESTIONNAIRE REQUIREMENTS: Identified Preliminary Target Audience Preliminary SMART Objectives DESIGNING YOUR QUESTIONNAIRE

How Many QUESTIONNAIRES? ONLY ONE QUESTIONNAIRE (Pre and Post KAP Survey) for all target audiences Some questions though may not be applicable to some audience Create a system where some questions are answered by certain target audience while others are not. How Many QUESTIONNAIRES?

How Many QUESTIONS should I ask? Factors that can affect your survey are: the number of respondents length of questions who will administer Data entry Data analysis and writing Cost Include questions that will answer only the six goals of survey How Many QUESTIONS should I ask?

Where do the survey questions come from? KEY CHARACTERISTICS Date Enumerator Name Form # Geographic Area/ Sampling location Respondent Information: Age, gender, education level, income sources, religion? Knowledge Question Attitude Question Interpersonal Communication Question Behavior Change Question Stages of Behavior Change Barrier Removal (As Appropriate) Exposure to Materials It is only after you know who you are going to interview and the preliminary objectives for your survey that you should begin to write survey questions.

SMART Examples REVISIT SMART OBJECTIVES SLIDE FROM PREVIOUS ABOUT DRAFTING SURVEY QUESTIONS FROM THE SMART OBJECTIVES… ADD CUSTOMIZED SLIDES HERE DEVELOPED FROM DRAFT OBJECTIVES

Behavior Change and Questionnaire Surveys Statement from stage of behavior change question in survey Stage of behavior change it should correspond to I have never considered using rat traps in my fishing boat. Validation stage I am considering using rat traps in my fishing boat but haven’t decided yet. Pre-contemplation stage I intend to use rat traps in my fishing boat in the near future, but I haven’t talked to anyone about it yet. Maintenance stage I have talked to someone about my intention to use rat traps in my fishing boat. Action Stage I used rat traps in my fishing boat at least once. Contemplation stage I always use rat traps in my fishing boat. Preparation stage the stages of change after the survey, but it is simpler for the respondent.

Sample Behavior Change Questions Clearly define the “desired behavior change” of your campaign Make one question each along the Stages of Behavior (Behavior Change continuum) Example:..\..\EXAMPLE behavior change questions.docx

Exposure Questions Make at least 8-10 Examples of potential materials that you will be producing in your campaign and ask the participants if they have see See example ..\..\EXAMPLE media exposure questions.docx

TOP TIPS Complete each interview in about 30 minutes or less The categories should also be “exhaustive;” that is, there is no other possible response that a respondent wants to give but cannot. Be sure that your questions directly relate to what it is you are studying, especially for those questions intended to measure SMART objectives.

Top Tips….. 4. Be very aware of the fact that people in your target communities may not have the same vocabulary as you. 5. If you want to force people to choose from a short list of possible responses, use the closed-ended format. 6. If there are some questions that you feel might be sensitive you should put these at the end of the survey so that respondents don’t get upset at the beginning.

Top Tips 7. Have one person do the first translation from the language the survey was drafted in into the second language. Then, have a different person translate the survey back into the original language (back translation). 8. “Administer the survey to a small number of respondents (five is usually enough) similar to your target population to make sure that categories are clear and complete”.

Key Elements of Good Survey Questions? Unbiased – not leading or pushing respondents towards a specific answer Tied to Objectives Time bound Clear Not Judgemental Refer to the lesson plan for examples and talk through each of the questions.

Writing good survey questions Other important criteria for good survey questions: Include a relevant time frame Use filter questions as needed Don’t overlap (e.g. age ranges) Use appropriate language for the respondents Refer to the lesson plan for examples and talk through each of the questions. Include a relevant time frame – “In the last six months, have you seen anyone fishing inside the MPA?” Use filter questions as needed - Don’t overlap (e.g. age ranges) Use appropriate language for the respondents

Writing good survey questions FILTER QUESTIONS In the past 6 months, have you talked to anyone about benefits to the community of a well enforced sanctuary? YES NO If you have, please tell me all of the people with whom you have talked to about this. [ ] Have not talked to anyone [ ] Talked to spouse/partner [ ] Talked to parents, or in-laws [ ] Talked to your children aged 16 or older [ ] Talked to your children aged 15 or younger [ ] Talked to friend or neighbour [ ] Talked to village elder or local authority [ ] Talked to National government environmental officials [ ] Talked to Non-government environment officials [ ] Talked to Local government environmental officials [ ] Talked to a Local XXXX Fisher or Gleaner [ ] Other ________________

Writing good survey questions RESPONSE QUESTIONS TO CATEGORIES SHOULD NOT OVERLAP PROVIDE ACCURATE TRANSLATION CLOSE VERSUS OPEN-ENDED QUESTIONS

ORDER OF QUESTIONS Use transitions between modules – use to switch from one topic to another Provide Accurate translation Review and Pretest your Questionnaire

Criteria for a Good Survey Questionnaire Uses the Rare template as a starting point, and asks the socioeconomic and demographic questions exactly as are in template. There is a clear link to the campaign’s objectives so that the data can help in campaign design, test assumptions and be used to measuring its impact across all stages of the Theory of Change. Questions cover most important factors such as: Socioeconomic and Demographic profiles Stages of behavior change question SMART objectives measurement Exposure to Campaign Activities & Messages

Criteria …. Questions are clear, unbiased and appropriate level of language and vocabulary for the target audiences. Interpersonal communication, behavior change, and exposure to Pride activity questions are time-bound, if appropriate, so you can measure change, normally this means they state a time of less than 1 year. Questionnaire did not touch upon socially sensitive issues which introduces bias. Questionnaire takes not more than 30 minutes to administer to a respondent. Questionnaire was pretested.

INDIVIDUAL/ GROUP WORKTIME Drafting your Survey Questions Your Draft Questions must follow the template provided to you earlier Add: Behavior Change Questions Media Exposure Questions

Conducting the Survey

Guide to Individual/Group Work (60 MINUTES) – 10:00-11:00 Review the Serena Island Research Plan (tables 5&6) and pages 312-313 in your packet Using your preliminary Results Chains and preliminary SMART Objectives begin to develop your site's survey questions. Take the time to develop survey questions Incorporate the newly developed survey questions into the Quantitative Data Collection section of your site's Research Plan. (Section 5.4 & 5.5) (30 MINUTES) 11:00-11:30 Share with another CF within your group Consult/ Share with your PPM on your Survey Questions

Other Considerations Filter Questions Response Categories for Questions Should not Overlap Closed versus Open-ended Questions Order of questions Provide accurate translation Review and pretest your questionnaire

Selecting Your Sample Representative sample Sample size Identify your random sampling procedure

ASSIGNMENT ASSIGNMENT #7, Draft Quantitative Survey (ComX 5337) Points: 30 points DATE ASSIGNED: 21 September 2012 DATE DUE: 24 September, Monday, 9:00PM What to Submit: Draft Survey Questions for K, A, IC, BC, Stage of Behavior Questions and Media Exposure

Grading Criteria Points Tied to SMART objectives Although there are a lot of questions that you might be interested in asking your respondents, the goal of your questionnaire survey is to understand your impact and exposure in relation to your objectives. Ensuring that the survey question is clearly tied to your objectives will make understanding your impact transparent. 10 Unbiased The questions are not leading or pushing the respondent towards a specific answer. 5 Follows logical order The survey should be organized so that simple and non-sensitive questions come first with any sensitive questions coming last.  Also, survey should test knowledge of topics before describing what those topics are.  Questions should flow in a logical order. Clear If the questions are complex or unclear, your responses will be as well! Simple questions that are not overly complicated by jargon will make responses to your survey precise and accurate. Not judgmental or emotional Overly emotional survey questions will make respondents feel uncomfortable and may answer in a way that doesn’t accurately reflect how they feel. Total   30 Penalty for late submission (-1.5)  

Revisiting Your Research Plan Template SECTION 5.0 QUANTITATIVE DATA COLLECTION 5.1 KAP Survey 5.2 Sampling Design 5.3 Survey Planning Table 5.4 Draft Survey Questions 5.5 Draft Stage-of-Behavior Questions

By the end of this lesson you will be able to: Define the 6 goals of the questionnaire survey. Differentiate between a biased and an unbiased survey question. Develop survey questions from preliminary objectives.

Design a survey that: has appropriate sample sizes and strategy measures the impact of your Pride campaign on SMART objectives for each target audience measures exposure to your Pride campaign activities