1. Survey Design & Item Construction Lindsay Couzens, M.S. UNLV’s 2009 Academic Assessment Workshop May 14 th & 15 th

2.  What is your overall research question?  What questions do you want your survey to answer?  Is there an existing survey that answers your questions?  Keep length of the survey in mind  Define target group  Identify Objectives 2 UNLV Office of Academic Assessment

3.  Identify Objectives  Based on purpose/research question  Example: Fowler, 1995 Example: Fowler, 1995  Develop definitions for objectives  All respondents should have a common understanding of the meaning of each question  Specify the kinds of answers needed to meet the objectives  Informs item construction 3 UNLV Office of Academic Assessment

4.  In person  Can clarify confusing statements  Immediate feedback  Limited to those who are present  Mail  Can reach large geographic areas  Depend on respondents’ motivation  Requires current address database  Email  Can reach large geographic areas instantly  Cost-efficient  Requires computer access  Online  Cost-efficient  Requires computer access  Interview  Can ask exploratory or probing questions  Can assist with unclear questions  Requires trained interviewers  Requires up-to-date telephone numbers 4 UNLV Office of Academic Assessment

5.  Timeline  Number of data collection points  Cutoff dates  Cost  Who will construct the survey?  How? ▪ Survey Monkey ▪ Elisten ▪ WebCampus tool 5 UNLV Office of Academic Assessment

6.  How will the survey be distributed?  Who will analyze the survey?  Do you need special hardware or software?  Who will maintain the survey?  Who is responsible for updating, revising, editing? UNLV Office of Academic Assessment 6

7.  Pilot testing  Focus groups  How will the surveys be distributed?  How will they be collected? 7 UNLV Office of Academic Assessment

8.  What is the research question?  Do the objectives for the survey flow from the research question?  Is each objective stated in concrete terms?  Does each objective measure one idea or concept? UNLV Office of Academic Assessment 8

9. Item Construction

10. Question Content  Each question should be mapped to one of your identified and defined objectives  Consider the content, scope and purpose of each item  Response formats  Open-ended or closed-ended?  Survey layout  Items should be grouped by topic AND by response type 10 UNLV Office of Academic Assessment

11. Response Formats  Things to Consider  Clarity of Directions  Levels of measurement ▪ Nominal ▪ Ordinal ▪ Interval  Exhaustiveness ▪ Response alternatives have sufficient range 11 UNLV Office of Academic Assessment

12. Response Formats  Things to Consider  Exclusiveness ▪ Response alternatives are mutually exclusive  Balancing Categories ▪ There should be an even number of alternatives on both sides of neutral UNLV Office of Academic Assessment 12

13. Response Formats  Numerical Rating Scales  Likert-type scales ▪ Rates attitudes or opinions  Horizontal rating scales ▪ Used to indicate where a respondent’s attitude or opinion falls between two opposite attitude positions 13 UNLV Office of Academic Assessment

14. Response Formats  Numerical Rating Scales  Semantic differential ▪ Opposite adjectives rather than attitude positions  Vertical rating ladder ▪ Ranking UNLV Office of Academic Assessment 14

15. Response Formats  Scores  Out of 10 ▪ Respondents rate each item  Ranking  Items are ranked relative to other items in the set 15 UNLV Office of Academic Assessment

16. Response Formats  Checklists  Respondent checks all that apply  Binary choice formats  Dichotomous questions ▪ Ex: What is your sex? (M/F); Do you smoke? (Y/N)  Paired comparisons ▪ Respondent chooses between two overlapping options UNLV Office of Academic Assessment 16

17. Response Formats  Multiple choice formats  Choice between multiple nominal categories ▪ Marital status  Choice between ordinal categories ▪ How often do you eat in the dining commons? 17 UNLV Office of Academic Assessment

18. Response Formats  Multiple choice formats  Choice between ordered attitude statements ▪ Which attitude statement most closely matches your own?  Numerical answers ▪ Salary; how many people live in the household UNLV Office of Academic Assessment 18

19. Question Wording Checklist deVaus, 2002  Is the language simple?  Can the question be shortened?  Is the question double-barreled?  Is the question leading?  Is the question negative? 19 UNLV Office of Academic Assessment

20. Question Wording Checklist deVaus, 2002  Is the respondent likely to have the necessary knowledge?  Will the words have the same meaning for everyone?  Is there a prestige bias?  Is the question ambiguous? 20 UNLV Office of Academic Assessment

21. Question Wording Checklist deVaus, 2002  Is the question too precise?  Is the frame of reference for the question sufficiently clear?  Does the question artificially create opinions?  Is personal or impersonal wording preferable? 21 UNLV Office of Academic Assessment

22. Question Wording Checklist deVaus, 2002  Is the question wording unnecessarily detailed or objectionable?  Does the question have dangling alternatives?  Does the question contain gratuitous qualifiers?  Is the question a “dead giveaway”? 22 UNLV Office of Academic Assessment