1. An Insider’s Guide to Survey Design Amy Feder Annemieke Rice

2. Integrated Assessment Solutions Today’s Learning Outcomes Participants will….  understand the concepts of local survey planning with results in mind.  gain knowledge of best practices regarding instrument implementation.  practice specific survey design and editing skills.  be able to apply knowledge and skills to a campus project.

3. Integrated Assessment Solutions Professional Equivalencies ACPA ASK Standards Ability to evaluate and create a rigorous survey with focus on effective question wording, effective survey format for the intended sample population, appropriate administration method, along with ability to achieve appropriate response rate Ability to determine the manner in which participants with disabilities will utilize any assessment instruments selected Ability to review an instrument for inclusive and accessible language likely to be viewed as informed and respectful by those for whom use of the instrument is intended NASPA’s Assessment Framework Assessment 301: Home Grown Surveys for Assessment

4. Integrated Assessment Solutions Agenda I.Why Survey Design is Important II.Survey Design Concepts:  Developing Content  Response Formats  Writing Questions  Structure & Length  Arrangement & Design III.Survey Administration

5. Integrated Assessment Solutions WHY SURVEY PLANNING AND DESIGN IS IMPORTANT Section I:

6. Integrated Assessment Solutions Overview “Questions and answers are part of everyday conversation; they are part of the fabric of our social life.” “remote conversation” “conversation with a purpose”

7. Integrated Assessment Solutions Steps in survey design Pilot test survey and revise Review and revise survey Determine physical characteristics of survey Determine item sequence Write and edit items Choose response formats Outline topic(s) and draft items

8. Integrated Assessment Solutions DEVELOPING CONTENT SECTION II:

9. Integrated Assessment Solutions Beginning with the end in mind Is a survey the best method? What data is already available? Who should be involved? What can we learn from past assessments? Who will review results? How will this data be used?

10. Integrated Assessment Solutions Develop objectives Examples: Age Major Preferred communication method Satisfaction with program Interest in future programs Impact on overall college experience

11. Integrated Assessment Solutions RESPONSE FORMATS SECTION II:

12. Integrated Assessment Solutions Response Formats Open ended responses Free response - text Numeric Yes/No with please explain Types of multiple choice responses Yes/No Single response Multiple response (e.g., Check all that apply, Select 3) Ranking Scales (see handout)

13. Integrated Assessment Solutions What to consider Scales that match Mutually exclusive Exhaustive Neutral/Not applicable/Non-response options Choose not to respondDon’t know Not applicableUnable to judge No opinionNeutral Neither ___ nor ___

14. Integrated Assessment Solutions WRITING QUESTIONS

15. Integrated Assessment Solutions Reliability & Validity Reliability – yielding the same results repeatedly Test/Re-test – consistency over time Inter-rater – consistency between people Validity – accurately measuring a concept Internal – confidence results due to independent variable External – results can be generalized Face validity – does this seem like a good measure? If a survey is valid, it is almost always reliable!

16. Integrated Assessment Solutions Language Consistency Simplicity Spelling Grammar Q&A instead of statements

17. Integrated Assessment Solutions Pitfalls to avoid Socially desirable responding – based on social norms Can never be eliminated Consider sensitive topics like race, drug and alcohol use, sexual activity, and other areas with clear social expectations Leading questions – suggesting there is a correct answer Double-barreled questions – asking more than one question Double negatives – including negative phrasing which makes responding difficult

18. Integrated Assessment Solutions STRUCTURE & LENGTH SECTION II:

19. Integrated Assessment Solutions Creating structure: Matrices

20. Integrated Assessment Solutions Creating structure: Display Rules

21. Integrated Assessment Solutions Creating structure: Paths

22. Integrated Assessment Solutions Evaluating Length Greater attrition at 22 questions or 13 minutes What to consider: Excluding “nice to know” Eliminate what you already know Outlining how results will be used Number of open-ended questions Number of required questions

23. Integrated Assessment Solutions ARRANGEMENT & DESIGN SECTION II:

24. Integrated Assessment Solutions Organization of content InterestTrust Logic Dropoff

25. Integrated Assessment Solutions Design Descriptive title Branding Progress barVisibility of scale Questions per pageSection headers ADA compliant

26. Integrated Assessment Solutions SURVEY ADMINISTRATION SECTION III:

27. Integrated Assessment Solutions Data Collection Methods ProsCons Web  No data entry  Accuracy is excellent  Technology benefits (e.g., display rules, required questions)  Immediate results  Anonymous  Audience is not usually captive  Possible misinterpretation (can’t ask ?s)  Technology issues  Response sample unrepresentative Mobile  No data entry  Accuracy is good  Technology benefits (e.g., display rules, required questions)  Captive audience  Administrator is available for ?s  Technology issues  Response sample unrepresentative  Limited formatting  Anonymity is questionable Paper  Captive audience  Administrator is available for ?s  No technology issues  No benefits of technology  Accuracy can be compromised  Data entry necessary  Anonymity is questionable

28. Integrated Assessment Solutions Piloting 1. Take it as if you were respondent 2. Seek reviews from colleagues with no prior knowledge 3. Administer to sample of actual population being studied Focus group Questions at end of survey Observing

29. Integrated Assessment Solutions Confidentiality & Anonymity Confidentiality: data kept contained to a specific set of reviewers Anonymity: participants not individually identified

30. Integrated Assessment Solutions Invitations Importance/Purpose Relevancy to respondent Request for help How and by whom results are used How long it will take to respond Deadline Incentives/Compensation Contact information

31. Integrated Assessment Solutions Response rate strategies Preannouncement Reminders Timing Incentives/Compensation Sampling Culture of sharing results

32. Integrated Assessment Solutions References & Resources Brace, I. (2004). Questionnaire design: How to plan, structure and write survey material for effective market research. London: Kogan Page Limited. Bradburn, N.M., Sudman, S. & Wansink, B. (2004). Asking questions: The definitive guide to questionnaire design – for market research, political polls, and social and health questionnaires. San Francisco: Jossey-Bass. Fink, A. (1995). The survey handbook. Thousand Oaks, CA: Sage Publications, Inc. Fowler, F.J. (1993). Survey research methods. Newbury Park, CA: Sage Publications, Inc. Fowler, F.J. (1995). Improving survey questions: Design and evaluation. Thousand Oaks, CA: Sage Publications, Inc. Porter, S. (2004). Overcoming Survey Research Problems. New Directions for Institutional Research, 121. Rea, L.M. (1992). Designing and conducting survey research: A comprehensive guide. San Francisco: Jossey-Bass. Stage, F.K. (Eds.). (1992). Diverse methods for research and assessment of college students. Alexandria, VA: ACPA. Suskie, L. (1997). Questionnaire survey research: What works (2nd ed.) Tallahassee, FL: AIR. Tourangeau, R., Rips, L.J., & Rasinski, K. (2000). The psychology of survey response. New York: Cambridge University Press.

33. Integrated Assessment Solutions Questions? Annemieke Rice arice@studentvoice.com 716.652.9400 www.studentvoice.com Amy Feder afeder@studentvoice.com