Survey as a Measurement Tool Yes, I am feeling fall today.
What does this say about surveys and research?
Why will someone believe your survey results? Validity Does this survey measure what it is supposed to measure? Reliability Would you get consistent results when the characteristics have not changed? Considerations: goals, permission, wording, demographic information requests, …
Essential Elements of Good Survey Clear objective Why are you creating this survey? What is the ultimate goal of the research of which this survey is a measurement tool? How will you use the data? What decisions do you want to impact with this survey? What are the variables that could impact answers? Control with demographic questions Define survey population
Survey Questions and Answers Keep the survey short and simple Make questions clear and simple (no “and”) and unbiased Closed ended/structured questions vs. open ended questions Consistent scale, and include a “prefer not to answer” Use a logical ordering Make sure answers are relevant to questions
Inviting Survey Respondents Do a pilot study 10/12 and ask for feedback Consider how random you can make it Be clear in your request to complete the survey Clearly state intentions for research Include instructions Do not ask for private information
Analyzing the Results Simply report Statistical Analysis Clearly describe sample population Graphical depictions Statistical Analysis Is there a statistically significant difference in answers between demographic variables? Plan in advance of giving the survey how you will present results – this helps to drive questions
Common Mistakes Objective and data use not clearly planned up front Not mutually exclusive answers (10-20, 20-30, etc.) Strong language (forced vs required) Could, should, might have up to a 20% difference in results Multiple parts in question (is it fair and reasonable?) Unbalanced scales and/or different orders (strongly agree, agree, partially agree, disagree) and reverse Not including important demographic info Not truly random sample
Measuring Validity and Reliability of a Survey Multitrait approach: use two different measures (survey and what?) Judgement by a panel of experts Embed validity questions in a survey Reliability Inter-rater Internal Consistency Equivalent Forms Test-retest