Hints and Tips on how to write a survey Kate Waller Information Manager

Only ask questions you want to know the answer to If you are asking for people’s opinions you have to be prepared to listen to the answers Better not to ask the question if you are not planning to do something with the results Make a distinction between what is essential to know, what is useful to know and what would be unnecessary. Focus on the ESSENTIAL

Don’t ignore your instincts The chances are if something doesn’t make sense to you or the person sat next to you then it won’t make sense to others Do you understand the questions? Would you be able to answer them? Do you think they all fit with the overall purpose of the survey? Do any of the questions feel intrusive? Is the survey too long?

Keep the survey as short as possible The quality of responses often suffers as respondents speed through lengthier questionnaires. QUALITY v QUANTITY Ideally between 5-8 minutes but can be as much as 20 minutes Longer questionnaires have higher drop out rates. Don’t over estimate the attention span of your audience

Funnel your questions Start with the easier, generic questions –Gain respondents’ commitment –Don’t want them to feel threatened –Save demographic questions until the end Organise the questions into themes –Don’t jump around –Start with broad themes and become more specific Remember what comes before can influence what comes after –How often do you use our mediation service? –Are you aware that we run a mediation service?

Ask one question at a time Are you rating phone rates or customer service? Are you asking whether television should be more innovative or whether innovation will increase ratings? Ambiguous questions lead to less reliable results

Don’t use complex or technical language See your survey as a way to educate rather than alienate –Identify commonly used abbreviations to be certain that everyone understands (don’t assume) –Use introductory text before a question that sets the scene and promotes understanding Pitch your survey at the lowest level of understanding –If you are surveying service users and professionals pitch it at service users –Or design two separate surveys Surveys are not a demonstration of your own knowledge

Only ask what people will know Do your think our service users value the service we deliver? –You should be asking the service users themselves! –Might reveal some interesting opinions but will not measure service user satisfaction Thinking back over the past 5 years, how many times have you used our service –People's memories are increasingly unreliable as you ask them to recall events farther and farther back in time Looking for reliable information not guess work

Avoid leading questions and forcing a particular response How much time does Funding Central save you each week? up to 30 minutes about 1 hour 1-2 hours 2-3 hours 3-5 hours 5-10 hours hours Other (please specify) Assumes that Funding Central saves time “Do you agree with the majority of people that the health service is failing?” Questions should be written in a way that does not influence responses Makes it more difficult for people to disagree

Focus on closed-ended questions Closed ended questions are easier to answer and easier to analyse –Too many open ended question can look onerous and put people off But closed ended questions can be frustrating if they do not cover all likely responses –Include ‘don’t know, ‘not applicable’ and ‘other’ option Use open-ended questions sparingly –Always good to include one at the end of the survey to pick up any other thoughts

Vary your closed ended questions Make respondents read each question by varying the way the questions are asked –If presented with a series or rating scales that are all the same respondents may simply whizz through and attribute the same rating to each question Use a combination of closed-ended questions Yes/No Ranking Rating Multiple choice

Examples Yes/NoRanking Did you drive here today?  Yes  No Please rank the following in order of importance, where 1 is most important to you and 4 is least important  Ease of parking  Cleanliness  Friendliness of staff  Cost Can include ‘Don’t know’ Don’t ask people to rank too many criteria

Examples Rating Multiple choice Which of the following items do you have with you today (tick all that apply)  Mobile phone  Lap top  Wallet/purse  Coat  Notebook Indicate how many can be selected Use a balanced rating scale

Use balanced rating scales Make sure that the scale is balanced with an equal number of positive and negative categories. Poor Fair Good Very Good Excellent Very poor Poor Fair Good Very good No opinion or uncertain is not a middle response on a rating scale A middle category in a scale between agree and disagree would be neither agree nor disagree. Options such as no opinion, not sure, undecided, don’t know, or, not applicable are placed off the scale

Be precise Need to make sure that everyone is interpreting the question and answers in the same way. The more precise you can be the more reliable the results How often have you used our service? –Rarely –Occasionally –Often –Frequently How often have you used our service in the past 12 months? –1-5 times –6-10 times –11-15 –More than 15 times

Be careful if asking follow-up “why” questions Don’t make one response less appealing than another It is sometimes tempting to ask ‘If no, why is this’ Some respondents will avoid answering honestly so they don’t have to give any more information Did you find today’s workshop useful  Yes  No (If ‘No’ please suggest how it could improved)

Think about lists of multiple choice responses Don't make the list of choices too long –If the list of answer categories is long and unfamiliar, it is difficult for respondents to evaluate all of them –You can always add ‘other’ category But, at the same time, try and make sure your list is comprehensive –Too many ‘others’ makes the analysis more difficult Respondents will not give their full attention to long lists. Do your give full attention to everything on a menu?

Use skip logic where appropriate Don’t make respondents answer questions that are not relevant –Paper survey and online surveys can incorporate skip logic –i.e. if you answer ‘no’ at question 8 skip to question 12 –But try and not make it too complicated

Make your survey sound interesting Some people discard an electronic message based entirely on its subject or sender –Consider titles that will arouse the interest of the recipients Once a recipient opens your survey, you may still need to motivate him or her to complete it. A good cover memo or introduction should be short and includes: Purpose of the survey Why it is important to hear from the correspondent What may be done with the results and what possible impacts may occur with the results. Person to contact for questions about the survey. Due date for response

Pilot the survey It is better to identify a problem during the pretest than after you have published the survey After they have completed the survey, brainstorm with them to see if they had problems answering any questions It would help if they explained what the question meant to them and whether it was valid to the questionnaire or not Before sending a survey to a target audience, send it out as a test to a small number of people