1. Dept. of Clinical Epidemiology & Biostatisitcs
School of Nursing’s Research Workshop and Seminar Series Survey Methods, Questionnaire Design & Implementation
Paul Krueger, Ph.D.
Associate Professor
Dept. of Clinical Epidemiology & Biostatisitcs
January 19, 2009

2. Surveys
Many study designs use surveys to collect information (exposures, confounders, outcomes, opinions, preferences, KABP)
Systematic method of collecting data
Structured and standardized questionnaire
Sample of the population

3. Advantages of Surveys
Most methods allow collection a lot of information, from many people, over a wide geographic area, in a relatively short time period
Everyone is asked the same questions in the same way (minimize interviewer bias)
If sampled appropriately, results can be representative of a larger population (within a certain degree of error)

4. Types of Surveys Mail (e.g. survey of all ER and Family MDs about CAP)
focus of this presentation
Telephone (e.g. air pollution - random sample)
Becoming more difficult (e.g. cell phones, answering machines, call display, call blocking, unlisted, do not call lists)
Fax (e.g. physician survey)
Face-to-face (e.g. LTCF resident QOL/satisfaction)
Internet ( and web based)
Combined methods (e.g. CAPIS, OARS…)

5. Dillman Approach to Mail Surveys
Total Design Method (1978)
What aspects of the survey process affect quality or quantity of response
A detailed and systematic way to design and implement surveys
Social exchange theory (see below)
High quality mail surveys
Acceptable response rates

6. Dillman Approach Cont’d
The Tailored Design Method (2000)
Many changes since 1978
Technology – computers, scanners…
More complete understanding of social exchange principles in survey research
Research on survey methods
Need to tailor the method vs same procedure
Note: Internet, Mail, and Mixed-Mode Surveys: The Tailored Design Method 3rd edition 2009

7. Social Exchange Theory
Theory of human behaviour
“The theory asserts that actions of individuals are motivated by the return these actions are expected to bring, and in fact usually do bring from others” (Dillman, 2000)
People will do things that are rewarding

8. Social Exchange Theory
Three elements for predicting action
Rewards
What one expects to gain from an activity
Costs
What one gives up/spends to obtain rewards
Trust
Expectation that in the long run, rewards of doing something will outweigh the costs of doing it

9. Application of Social Exchange Theory
In terms of questionnaire design & implementation:
Increase rewards for responding
Reduce perceived costs
Establish trust so the rewards outweigh costs of responding

10. Ways of Providing Rewards
Show positive regard
Show respect (e.g. give reason for survey, provide number to call, personally address letters)
Say thank you
e.g. “we appreciate your help”, “thanks in advance”
Ask for help, advice, assistance
Provides a sense of reward
Support group values
appeal to widely shared values – the study’s social usefulness

11. Ways of Providing Rewards
Give tangible rewards
Include a financial incentive (token) vs promise
Make the questionnaire interesting
Improve layout and design
Give social validation
Let them know that others have responded
Opportunities to respond are scarce
Respond quickly so information can be used

12. Ways of Reducing Social Costs
Avoid subordinating language
Ask for a favour e.g. to solve a problem vs it is necessary for you to complete this survey because….
Avoid embarrassment
E.g. avoid complex questions and instructions at the start of the questionniare
Avoid inconvenience
E.g. include return postage paid envelope – real stamp

13. Ways of Reducing Social Costs
Make questionnaire appear short and easy
Lesson the perceived cost of responding
Minimize requests to obtain personal information
Offer explanations why info is important & ensure confidentiality

14. Ways of Establishing Trust
Provide a token of appreciation in advance
Small amount – gesture of trust – vs payment for your time
Including actual stamps on return envelope
Sponsorship by legitimate authority (e.g. university or gov’t sponsored vs marketing research)
Make the task appear important
Personalized cover letters on letterhead paper
Professional looking questionnaire
Something useful will be done with results

15. Dillman’s Five Needed Elements for High Response Rates
Respondent friendly questionnaire
Up to 5 (varied) contacts
Inclusion of stamped return envelopes
Personalized correspondence
Token financial incentive

16. Respondent Friendly Questionnaire
Easy to understand questions
Well ordered
Easy to follow layout
Overall appearance

17. Writing Questions
Develop questions that every potential respondent will:
Interpret the same way (and the same way as the investigators)
Be able to respond accurately
Be willing to answer

18. Dillman’s examples of things to do when writing questions
Use simple words
Do not be vague
Keep it short
Avoid being too specific
Do not talk down to respondents
Avoid bias
Avoid objectionable questions
Etc.
However, many of the above can be contradictory!

19. Things to do…. Use equal numbers of pos and neg categories
Distinguish between undecided and neutral
Create mutually exclusive response categories
Avoid double-barrelled questions
Provide appropriate time referents
Etc. ……
See Dillman, Choi (and others) for many other things that should and should not be done when writing questions

20. Questionnaire Construction
Respondent friendly questionnaire design
Will improve response rates (and response rates among those least likely to respond)
Reduces measurement error (e.g. prevents items or answer categories from being missed)
Issues to consider:
Questionnaire format
Question order
Appearance of individual questions
Front and back covers

21. Questionnaire Format Booklet format is recommended
Handled more or less automatically without error
People are familiar with them
See Dillman for design issues and rationale

22. Ordering Questions A questionnaire is like a conversation
Choose the first question carefully
Should apply to everyone
Should be easy
Should be interesting
Should be connected to the study purpose
Needs to be a logical flow
Group questions with similar content
Sections with “transition statements”
More salient to least salient
Objectionable questions near end…….

23. Appearance of Questions
Respondents should see and understand every word of every question
Font, font size, bold, italics can ease the task of understanding and responding
Place instructions where needed
If combining questions with same answer categories – do it carefully
Number questions consecutively
More space between questions than subcomponents
List answer categories vertically (one column)
Etc……..

24. Front Cover Opportunity to motivate and increase response rates
Title (short and simple)
Neutral graphic
Name and address of study sponsor
Extension of cover letter (well known and legitimate source)

25. St. Joseph’s Lifecare Centre
2007 FAMILY MEMBER SATISFACTION WITH CARE SURVEY
The purpose of this survey is to collect information that will be used to help improve care and services provided for your loved one at the St. Joseph’s Lifecare Centre. Your participation is voluntary and all information will remain confidential. Answers will be pooled together so that no individual response can be identified. With this in mind, we are interested in your honest opinion, whether it be negative or positive. If you wish to comment on any question, please feel free to use the space in the margins. Your comments will be read and taken into account.
Thank you in advance for your participation.
Working together to enhance the quality of life by providing love, care, and dignity   
99 Wayne Gretzky Parkway, Suite 105
Brantford, Ontario
N3S 6T6
Attention: St. Joseph’s Health System Research Network

26. Back Cover Keep it simple
Invitation to make additional comments (chance for people to vent)
Thank you
Lots of “white space”
Avoid questions on back cover
Last few questions tend to be the most objectionable (e.g. income)

27. Pretesting Review by knowledgeable colleagues
Content validity (included all necessary questions; exclude unneeded questions; appropriate response categories….)
Diverse expertise (content + methodology)
Potential survey respondents
All words understood
All questions interpreted the same
All inclusive list of response categories
Mailout package create a positive attitude etc….

28. Pilot Testing
Revise questionnaire and then pilot test all proposed procedures
Examine response rates, item nonresponse, variable distributions
Make revisions based on findings
Then, have someone “fresh” that can review the questionnaire one last time
Questionnaire design is much more complex and takes much longer than most people expect

29. Mail Survey Implementation
Questionnaire design is only 1 part of a well designed survey
It is not the main determinant of high response rates
Survey implementation procedures collectively have a greater influence on response rates

30. Dillman Recommends 5 Contacts
Multiple, varied contacts are essential
Prenotice (advance notification) letter
Questionnaire mailout (package)
Thank you postcard (thank you / reminder letter)
Replacement questionnaire (package)
Final (special) contact

31. Prenotice Letter (First Contact)
Notice to expect a request to participate in an important study (all letters - low education level)
Brief, personalized, positively worded
Should build anticipation
Courtesy (people like to be told ahead of time)
Mention incentive
Mailed (first class) a few days to week before questionnaire

32. Prenotice Letter Letterhead paper (helps to legitimize survey)
Actual Date / Respondent name and address / Dear…
What will happen (you should receive a questionnaire)
What it is about (brief description + sponsor)
Usefulness of survey (how it will help)
Thank you
Real signature + title and affiliations
P.S. Incentive notice (if one is to be offered)

34. Questionnaire Mailout (Second Contact)
Sent a few days to 1 week after prenotice
Includes:
Cover letter
Questionnaire (with an ID#)
Token of appreciation ($, TH coupons, pen etc.)
One of the best ways to increase response
Should accompany questionnaire vs afterwards
Return postage paid envelope (with stamps)

35. Cover Letter 1 page on letterhead paper
Exact mailing date / Name & address / Dear…
Specifically written to the person vs a form letter
What this letter is about (your request)
Why were you selected (and how)
Usefulness of survey (how this will help)
Confidentiality
ID# on questionnaire (don’t hide it)
Voluntary (if refuse ask for reason)

36. Cover Letter Cont’d Enclosures (stamped return envelope + token)
Who to contact with questions (accessibility)
Thank you for helping with this important study
Real signature (contrasting ink – pressed)
Postscript (thanks again - token)

38. Assembling and Mailing
Attention to detail
Questionnaire ID# matches the name on letter and mailing label
Everything is included
Dillman “stuffing procedure” to ensure that none of the contents are left in the envelope
Larger outgoing envelope (avoid folding)
1st class mail
Select mailout time (avoid major holidays) and day

39. Thank you / Reminder (Third Contact)
Mail 1 week after questionnaire
Thank you / reminder post card (or letter)
Objective is to jog memories
Very short
Thanks those who completed and returned
Reminds others to “please do so today”
Statement of how important each recipient is to the success of the study
If you didn’t receive it, call and one will be sent
Signature

41. First Replacement Questionnaire (Fourth Contact)
Send 2 weeks after thank you / reminder
Only sent to non responders (need to track)
Cover letter - more persuasive/insisting tone
Questionnaire not yet received
Others have responded/importance to study (representative)
Not eligible? (return questionnaire)
Confidential / voluntary (still return it unanswered)
Note of appreciation
Personally signed / P.S. (any questions)
Replacement questionnaire (with ID#)
Return postage paid envelope - but no token

43. Special Contact (Fifth Contact)
4 weeks later to non respondents [PK 2 wks]
Impact is due to a different “special” method which shows respondent & study are important
Telephone call (encourage/send new one/complete)
Registered mail (requires signature)
Special delivery
Courier
More expensive follow-up
“New” cover letter, questionnaire (ID#), postage paid envelope

44. Survey Error (Dillman)
Sampling error
Precision is limited by the number surveyed
Coverage error
Occurs when list does not include everyone
Measurement error
Results from poor wording and construction
Nonresponse error
When significant numbers don’t respond

45. Example Readings for Other Ideas
Increasing Response Rates to Postal Questionnaires (Edwards et al. 2002)
Systematic review of RCTs
292 RCTs reviewed
75 strategies for influencing response
A Catalog of Biases in Questionnaires (Choi and Pak, 2005)
Identified 48 common types of bias
Categorized into 3 main sources of bias
Question design
Questionnaire design
Questionnaire administration

46. Problems/Issues Undelivered questionnaires
People move/incomplete address or errors
Attempt to correct problems and resend
New mail out dates – additional work
Handling respondent questions
Should attempt to answer all questions
Should anticipate questions (e.g. ID#, no proxies…)
Thank them for asking
Evaluating early returns
Identify problems (e.g. printing, item response, certain questions causing problems….)

47. Final Note About Mail Survey Response Rates
Mail surveys often criticized for low response
Example Response Rates (PK using “Dillman approach”)
Surveys of physicians
Physician Survey (elder abuse) 65%
Physician Survey (pneumonia) 77%
LTC Medical Director Survey 61%
Surveys of other health professionals
LTC Director of Care Survey 76%
Canadian Optometrist Survey 90%
Other “general population” health surveys
Contact Lens Use Survey 93%
LTC Family Member Satisfaction Surveys (80-86%)