Statistics and Research Desgin Survey Says? Statistics and Research Desgin

What Are Surveys? Millions of surveys conducted each year Data used to make important decisions Over 2,000 survey research firms - companies that specialize in the construction, administration, and analysis of survey data.

The Survey Research Method Constructing reliable and valid surveys takes time Constructing good surveys is a science and an art Science - follow a scientific process Art - knowing how much rigor to put into the design and writing good questions take years of practice

Quickly Developed Course Evaluation Survey

Better Course Evaluation Survey

The Survey Research Method Scientific Method of Survey Design Good surveys share the following characteristics: specific and measurable objectives straightforward questions understood similarly by most people pretested to ensure no unclear questions or incorrect skip patterns administered to adequate population or sample of respondents so generalizations can be made appropriate analysis to obtain the objectives accurate reporting of results (both verbal and written) reliable and valid

Table 10.2: A Scientific Approach to Survey Design Step in the Scientific Method Corresponding Step for Designing Surveys Competencies of the Survey Researcher Identify a problem and form a hypothesis Presurvey issues Know how to conduct a literature review Know how to gather people who are knowledgeable about the survey topic Know how to conduct focus groups Design a study to explore the problem and test the hypothesis Construct the survey Know the different types of surveys Know the different types of survey questions Know how to write effective survey questions Know how to assemble questions into a survey instrument Know the various methods for pretesting surveys Know how to interpret data to revise and finalize the survey Conduct the study Administer the survey Know the methods for sampling respondents Understand the logistics of administering various surveys Analyze the research data Analyze the survey data Know how to code survey data Know how to enter survey data into a database Know the methods for analyzing survey data Communicate the research findings Communicate the findings of the survey Know how to write a report Know how to prepare presentation materials Know how to present information to a group

The Survey Research Method Preparing for the Survey (1st phase) Identifying the objectives Purpose and what survey will measure Defining the objectives operationally Operational definitions are specific behaviors that represent the purpose Constructing a plan

The Survey Research Method Constructing the Survey (2nd phase) Types of surveys Self-administered Mail Internet Individually administered Personal interviews Face to face Telephone Structured record reviews

The Survey Research Method Developing survey questions Open-ended questions Closed-ended questions Yes/No questions Fill-in-the-blank questions Implied no choice Single-item choices Enfolded formats Free choices Multiple-choice format Ranking questions Rating questions Guttman format Likert and other intensity scale formats Semantic differential format Paired comparisons and constant referent comparisons

Different Types of Rating Scales

The Survey Research Method Characteristics of good survey questions: Purposeful and straightforward Unambiguous In correct syntax Use appropriate rating scales and response options Include appropriate categorical alternatives Ask one and only one question At comfortable reading level

The Survey Research Method Preparing the survey instrument To catch respondents’ attention and motivate respondents to complete survey so as to reduce respond errors, pay attention to… Title and seal Appeal and instructions Headings and subheadings Transitions Response directions Bold typeface Justification of response spaces Shading White space Printing Font type

The Survey Research Method Pretesting the survey Purpose of pretesting is to Identify sources of nonsampling measurement errors—errors associated with the design and administration of the survey Examine how effective revisions of a question or an entire survey are Indicate the effect of alternative versions of a question or survey Assess the final version of a questionnaire for respondent understanding, time to complete, and ease of completion Allow the survey analysts to obtain data from the survey and to make changes to the format that might make data entry or analysis more efficient

The Survey Research Method Methods for pretesting include One-on-one interviews Respondent focus groups Behavior coding respondent-interviewer interactions Interviewer and respondent debriefings Split-sample tests Item nonresponse rate

The Survey Research Method Administering the Survey (3rd phase) Involves three steps Selecting appropriate respondents Choosing a sample size Distributing the survey

The Survey Research Method Probability sampling Simple random sampling – every population member has an equal chance of being selected Systematic sampling – every nth member will be a member of the sample Stratified sampling – after dividing into strata, every population member has an equal chance of being selected Cluster sampling – after dividing into clusters, members selected from cluster Nonprobability sampling Convenience sampling – uses an available group

The Survey Research Method Coding, Entering, and Analyzing Survey Data (4th phase) Process includes Coding the survey questions Entering data into spreadsheet program Verifying data are entered correctly Conducting statistical analysis Interpreting the results

The Survey Research Method Conducting statistical analysis and interpreting the results Typically plan analysis at time of survey construction Important statistics response rate reliability sampling error univariate analyses Frequency counts, means, modes, medians bivariate and multivariate analyses E.g. correlation coefficients, cross-tabulations, chi-square, t-tests, analyses of variance

The Survey Research Method Presenting the Findings (5th phase) Reporting results to those who commissioned the survey’s development and sometimes to the public at large

Report Outline

Survey Reliability and Validity A reliable survey yields consistent data Speak of two types of error Random error – unpredictable errors Measurement error – errors associated with how a survey performs in a particular population Reliable surveys - free from measurement error caused by poorly worded questions, ambiguous terms, inappropriate reading level, unclear directions, and incorrect skip patterns, etc.

Survey Reliability and Validity Survey validity A valid survey yields accurate information Various methods for determining validity Content Criterion-related Construct Face

Questions? Thoughts?