1. To Survey or Not To Survey? That is the Question
Meredith J. Dean, Ph.D.
Virginia Commonwealth
University

2. AGENDA Background to Surveys & Survey Error Options to Gathering Data
Good Survey Design
Good Question Design
Sharing Results
Completing the Assessment Loop
Conclusion

3. Surveys

4. Understanding Your Landscape

5. Definition of Surveys
“systematic method for gathering information from a sample of entities…” (Groves et al., 2009, [p. 2)
“system for collecting information” (Sue & Ritter, 2007, p. 1)
Not just a questionnaire! (de Vaus, 2014; Sue & Ritter, 2007
Total Survey Design (Sue & Ritter, 2007: See Page 2)
Interviews, observations, content analysis (de Vaus, 2014)

7. The Problem Over surveying Too much data Meaningless survey questions
AKA Survey Fatigue
Too much data
Meaningless survey questions
Wording by questionnaire designers
Interpretation by participants
Asking participants about routine behavior or actions they probably don’t track
Image:

8. Pros and Cons to Survey Research
Interdisciplinary Field
Amount of Data
Many Choices
Platform Options
Administration Modes
Question Types
Interdisciplinary Field: Different Approaches, but you have to know the program and its outcomes/purpose
Question types: Lots to choose from, but how do you choose?
Large Sample Sizes, Large amounts of data, but you have to process it and analyze it and report its results
Platform options: Several choices, but what is best? Do your research.
Administration modes.

9. To Survey or Not to Survey… ...why is this important?
Validity and reliability
Survey error
Response rates
Bias
Response and Nonresponse
RESPONSE VALIDITY:
Social Desirability Bias
Inaccurate Estimates in Responses
Attitudes and NonAttitudes
Reliability: same results if repeated
Measurement
Processing
Coverage
Sampling
Nonresponse
Adjustment
Response Bias: The bias portrayed by respondent’s answers
Nonresponse Bias: What bias exists in the results that would be different if nonrespondents HAD participated?
Course evaluations… only certain people respond.

10. Survey Errors Measurement Processing Coverage Sampling Nonresponse
Adjustment
RESPONSE VALIDITY:
Social Desirability Bias
Inaccurate Estimates in Responses
Attitudes and NonAttitudes
Reliability: same results if repeated
Measurement
Processing
Coverage
Sampling
Nonresponse
Adjustment
Response Bias: The bias portrayed by respondent’s answers
Nonresponse Bias: What bias exists in the results that would be different if nonrespondents HAD participated?
Figure adapted from Groves et al., 2009; Adams, 2010.

11. So now, what can I do? No-Survey Options
Use other peoples’ survey results
Use institutional or organizational data
Interviews and focus groups
Creating a Survey
Tailored Design
Aligning institutional Data to your survey
Aligning questions to outcomes
Multiple Readers and Reviewers… Pilot!

12. Tailored Design (Dillman)
Use the survey design that works best for your population
Can any methods be combined
links → Mailed surveys → Phone
Paper → Phone
ing college students doesn’t work anymore

13. Good Survey Design: When is a survey appropriate?
It addresses outcomes
You are lacking data
Institutional or organizational does not already exist (e.g. Other surveys)
If you are seeking satisfaction or perception from a large population
You have the capacity to share the information!
Other Surveys: How many surveys are already administered to the population?
Have surveys already been done on this?
What data does the organization or institution already have?

14. Good Survey Question Creation
Learning from real-world examples and scenarios

15. Example A Why did you come to School of Business Advising? (check one)
Graduation
Internships, study abroad, undergraduate research
Registration
Military
Post Degree Plans
Other __________________

16. Example B To what extent do you agree with the following statements?
My advisor was knowledgeable of university policies, degree requirements, and campus resources.
My advisor made me feel comfortable.

17. Example(s) C
How many times have you had dairy products in the past week?
What is your major?
Who is your advisor?

18. Other Examples: Spring 2015
23 course evaluations
20 post-advising appointments
Two campus safety surveys
Three technology surveys
Library, The Well, NSSE last year, Noel-Levitz this year, eServices polls

19. Not Just One or the Other: You can use both!
Institutional and Organizational Data
Informational System Data
Card Swipes
University ID
Survey Data
Previous surveys
Course/advising evaluations tied to student ID
GPA, major, CH, etc.

20. Combining Results + Existing Data
Organizational or institutional data warehouses - a wealth of information!
Can be used in conjunction with other…
Research methodologies (e.g. (quasi) experimental designs; focus groups, etc.)
Archived Data; Previous assessment/evaluation results; Pre-Post Tests; Other surveys
Data definitions might be problematic (Liebig, 2009).
How to define a transfer student.
How to define completion rates for transfers.
When does the semester date start? (Banner vs. other programs)

21. Tailor your study design to your purpose, outcomes, and context.
Problem Solution
Know survey calendar
Know your data holders
Talk to people who conduct research with the same org. or population
Connect questions to outcomes or objectives
Survey fatigue (Over surveying)
Existent Data
Is someone else already doing it? Has someone already done it?
Will the info be used?
Tailor your study design to your purpose, outcomes, and context.

22. Data Analysis and Results
Don’t Just Collect Data!!
Data Analysis
Reports
Written
Presentations

23. Survey Results

24. Survey Results

25. How to Complete the Assessment Loop
Share the Data!
Sharing Results at Meetings!

26. Before You Create Your Survey Monkey Account...
Understand the literature.
Understand your purpose.
Know your institution and/or context.
Create a plan for your study.
Create & Pilot
Administer & Collect Data
Analyze Data & Report Results
Will you be able to do something about results?

27. To Survey or Not to Survey...
Depends on…
Research methodology & approach
The context & outcomes
What data are available
Your capability to handle the data, their analyses, and the results.

28. Plan Ahead!
Be aware of your survey design and its implications from start to finish.
Goal: High Quality, Meaningful Data
Usable Results
Survey Design that minimizes the potential for error

29. Keep in touch about your progress!
Any questions?
Meredith J. Dean, Ph.D.
Thank you!
Keep in touch about your progress!