1. Statistics and Research Desgin
Survey Says?
Statistics and Research Desgin

2. What Are Surveys? Millions of surveys conducted each year
Data used to make important decisions
Over 2,000 survey research firms - companies that specialize in the construction, administration, and analysis of survey data.

3. The Survey Research Method
Constructing reliable and valid surveys takes time
Constructing good surveys is a science and an art
Science - follow a scientific process
Art - knowing how much rigor to put into the design and writing good questions take years of practice

4. Quickly Developed Course Evaluation Survey

5. Better Course Evaluation Survey

6. The Survey Research Method
Scientific Method of Survey Design
Good surveys share the following characteristics:
specific and measurable objectives
straightforward questions understood similarly by most people
pretested to ensure no unclear questions or incorrect skip patterns
administered to adequate population or sample of respondents so generalizations can be made
appropriate analysis to obtain the objectives
accurate reporting of results (both verbal and written)
reliable and valid

7. Table 10.2: A Scientific Approach to Survey Design
Step in the Scientific Method
Corresponding Step for Designing Surveys
Competencies of the Survey Researcher
Identify a problem and form a hypothesis
Presurvey issues
Know how to conduct a literature review
Know how to gather people who are knowledgeable about the survey topic
Know how to conduct focus groups
Design a study to explore the problem and test the hypothesis
Construct the survey
Know the different types of surveys
Know the different types of survey questions
Know how to write effective survey questions
Know how to assemble questions into a survey instrument
Know the various methods for pretesting surveys
Know how to interpret data to revise and finalize the survey
Conduct the study
Administer the survey
Know the methods for sampling respondents
Understand the logistics of administering various surveys
Analyze the research data
Analyze the survey data
Know how to code survey data
Know how to enter survey data into a database
Know the methods for analyzing survey data
Communicate the research findings
Communicate the findings of the survey
Know how to write a report
Know how to prepare presentation materials
Know how to present information to a group

8. The Survey Research Method
Preparing for the Survey (1st phase)
Identifying the objectives
Purpose and what survey will measure
Defining the objectives operationally
Operational definitions are specific behaviors that represent the purpose
Constructing a plan

9. The Survey Research Method
Constructing the Survey (2nd phase)
Types of surveys
Self-administered
Mail
Internet
Individually administered
Personal interviews
Face to face
Telephone
Structured record reviews

10. The Survey Research Method
Developing survey questions
Open-ended questions
Closed-ended questions
Yes/No questions
Fill-in-the-blank questions
Implied no choice
Single-item choices
Enfolded formats
Free choices
Multiple-choice format
Ranking questions
Rating questions
Guttman format
Likert and other intensity scale formats
Semantic differential format
Paired comparisons and constant referent comparisons

11. Different Types of Rating Scales

12. The Survey Research Method
Characteristics of good survey questions:
Purposeful and straightforward
Unambiguous
In correct syntax
Use appropriate rating scales and response options
Include appropriate categorical alternatives
Ask one and only one question
At comfortable reading level

13. The Survey Research Method
Preparing the survey instrument
To catch respondents’ attention and motivate respondents to complete survey so as to reduce respond errors, pay attention to…
Title and seal
Appeal and instructions
Headings and subheadings
Transitions
Response directions
Bold typeface
Justification of response spaces
Shading
White space
Printing
Font type

14. The Survey Research Method
Pretesting the survey
Purpose of pretesting is to
Identify sources of nonsampling measurement errors—errors associated with the design and administration of the survey
Examine how effective revisions of a question or an entire survey are
Indicate the effect of alternative versions of a question or survey
Assess the final version of a questionnaire for respondent understanding, time to complete, and ease of completion
Allow the survey analysts to obtain data from the survey and to make changes to the format that might make data entry or analysis more efficient

15. The Survey Research Method
Methods for pretesting include
One-on-one interviews
Respondent focus groups
Behavior coding respondent-interviewer interactions
Interviewer and respondent debriefings
Split-sample tests
Item nonresponse rate

16. The Survey Research Method
Administering the Survey (3rd phase)
Involves three steps
Selecting appropriate respondents
Choosing a sample size
Distributing the survey

17. The Survey Research Method
Probability sampling
Simple random sampling – every population member has an equal chance of being selected
Systematic sampling – every nth member will be a member of the sample
Stratified sampling – after dividing into strata, every population member has an equal chance of being selected
Cluster sampling – after dividing into clusters, members selected from cluster
Nonprobability sampling
Convenience sampling – uses an available group

18. The Survey Research Method
Coding, Entering, and Analyzing Survey Data (4th phase)
Process includes
Coding the survey questions
Entering data into spreadsheet program
Verifying data are entered correctly
Conducting statistical analysis
Interpreting the results

19. The Survey Research Method
Conducting statistical analysis and interpreting the results
Typically plan analysis at time of survey construction
Important statistics
response rate
reliability
sampling error
univariate analyses
Frequency counts, means, modes, medians
bivariate and multivariate analyses
E.g. correlation coefficients, cross-tabulations, chi-square, t-tests, analyses of variance

20. The Survey Research Method
Presenting the Findings (5th phase)
Reporting results to those who commissioned the survey’s development and sometimes to the public at large

21. Report Outline

22. Survey Reliability and Validity
A reliable survey yields consistent data
Speak of two types of error
Random error – unpredictable errors
Measurement error – errors associated with how a survey performs in a particular population
Reliable surveys - free from measurement error caused by poorly worded questions, ambiguous terms, inappropriate reading level, unclear directions, and incorrect skip patterns, etc.

23. Survey Reliability and Validity
Survey validity
A valid survey yields accurate information
Various methods for determining validity
Content
Criterion-related
Construct
Face

24. Questions? Thoughts?