Semester Review. The Tailored Design Method Uses multiple motivational features in compatible and mutually supportive ways to encourage high quantity.

Slides:



Advertisements
Similar presentations
1 © 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation Response Rate in Surveys Key resource: Dillman, D.A.,
Advertisements

EMR 6500: Survey Research Dr. Chris L. S. Coryn Kristin A. Hobson Spring 2013.
Self-Administered Surveys: Mail Survey Methods ChihChien Chen Lauren Teffeau Week 10.
6 Developing Content and Layout Section 6.1 Generate and organize content ideas Write and organize Web text Section 6.2 Identify page dimension guidelines.
STATISTICS FOR MANAGERS LECTURE 2: SURVEY DESIGN.
5.00 Understand Promotion Research  Distinguish between basic and applied research (i.e., generation of knowledge vs. solving a specific.
1 Health Surveys January 2008 Diane Martin, MA, PhD.
EMR 6500: Survey Research Dr. Chris L. S. Coryn Lyssa N. Wilson Spring 2015.
Designing the Questionnaire
Chapter 8 Designing the Survey Questionnaire Copyright © 2013 by The McGraw-Hill Companies, Inc. All rights reserved. McGraw-Hill/Irwin.
© 2003 Prentice-Hall, Inc.Chap 1-1 Business Statistics: A First Course (3 rd Edition) Chapter 1 Introduction and Data Collection.
1 Sources:  SusanTurner - Napier University  C. Robson, Real World Research, Blackwell, 1993  Steve Collesano: Director, Corporate Research and Development.
The Complete Design Data Collection Methods Part Three.
© 2005 The McGraw-Hill Companies, Inc., All Rights Reserved. Chapter 8 Using Survey Research.
Data Collection Methods
Documentation and survey quality. Introduction.
EMR 6500: Survey Research Dr. Chris L. S. Coryn Spring 2012.
Chapter 2 Web Site Design Principles Principles of Web Design, 4 th Edition.
Survey Methods: Communicating with Respondents
Chapter 13 Survey Designs
EMR 6500: Survey Research Dr. Chris L. S. Coryn Kristin A. Hobson Spring 2013.
FINAL REPORT: OUTLINE & OVERVIEW OF SURVEY ERRORS
Chapter 9 Descriptive Research. Overview of Descriptive Research Focused towards the present –Gathering information and describing the current situation.
1 © 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation ONLINE SURVEYS.
Classroom Assessment A Practical Guide for Educators by Craig A. Mertler Chapter 9 Subjective Test Items.
EMR 6500: Survey Research Dr. Chris L. S. Coryn Kristin A. Hobson Spring 2013.
EMR 6500: Survey Research Dr. Chris L. S. Coryn Spring 2012.
Determining the Order of Questions
9: Designing the Questionnaire. 9-2 Copyright © 2008 by the McGraw-Hill Companies, Inc. All rights reserved. Hair/Wolfinbarger/Ortinau/Bush, Essentials.
Sample Design.
Business and Management Research
Power Point Slides by Ronald J. Shope in collaboration with John W. Creswell Chapter 13 Survey Designs.
Survey Research and Other Ways of Asking Questions
Questionnaires and Interviews
Dr. Engr. Sami ur Rahman Assistant Professor Department of Computer Science University of Malakand Research Methods in Computer Science Lecture: Research.
By: Christopher Prewitt & Deirdre Huston.  When doing any project it is important to know as much information about the project and the views of everyone.
Making Sense of the Social World 4th Edition
CHAPTER 12 – SAMPLING DESIGNS AND SAMPLING PROCEDURES Zikmund & Babin Essentials of Marketing Research – 5 th Edition © 2013 Cengage Learning. All Rights.
Lesli Scott Ashley Bowers Sue Ellen Hansen Robin Tepper Jacob Survey Research Center, University of Michigan Third International Conference on Establishment.
Type author names here Social Research Methods Chapter 10: Self-completion questionnaires Alan Bryman Slides authored by Tom Owens.
THE TAILORED DESIGN METHOD Damon Burton University of Idaho.
CHAPTER TEN AUTHORING.
Dr D.K. Nyahunzvi Midlands State University 2012.
بِسْمِ اللهِ الرَّحْمنِ الرَّحِيمِِ King Fahd University of Petroleum and Minerals Questionnaire Design presented to: Dr. Abdulaziz A. Bubshait by: Mohammed.
Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved.
Self-completion Questionnaires
10 Questionnaire Design. Role of Questionnaire Survey research, by definition, relies on the use of a questionnaire. A questionnaire is a set of questions.
Chapter X Questionnaire and Form Design. Chapter Outline Chapter Outline 1) Overview 2) Questionnaire & Observation Forms i. Questionnaire Definition.
The effects of Peer Pressure, Living Standards and Gender on Underage Drinking Psychologist- Kanari zukoshi.
Survey Research at WCU Hal Herzog. The truth about the subject pool: Little incentive to give you good data. Usually young and inexperienced. Often have.
Formatting a Mail Questionnaire  Appearance and placement of questions on page(s)  A well-planned and constructed questionnaire will increase response.
Creating & Building the Web Site Week 8. Objectives Planning web site development Initiation of the project Analysis for web site development Designing.
Creating A Survey Using Office of Student Affairs Assessment The University of Georgia A-Team Training-Skills Session 1 October 30, 2007.
Oman College of Management and Technology Course – MM Topic 7 Production and Distribution of Multimedia Titles CS/MIS Department.
PREPARATION OF QUESTIONNAIRES PREPARATION OF QUESTIONNAIRES Chapter - 4 Dr. BALAMURUGAN MUTHURAMAN
PRINCIPLES OF UNIVERSAL DESIGN: Usable learning environments for all students.
Section 29.1 Marketing Research Chapter 29 conducting marketing research Section 29.2 The Marketing Survey.
Data Collecting Techniques Telephone interviews Traditional telephone interviews involve phoning a sample of respondents and asking them a series.
Tier III Preparing for First Meeting. Making the Decision  When making the decision to move to Tier III, all those involve with the implementation of.
Using Surveys to Design and Evaluate Watershed Education and Outreach Day 5 Methodologies for Implementing Mailed Surveys Alternatives to Mailed Surveys.
From Question to Action: Creating In-House Surveys as a part of Data Driven informed Decision Making David Consiglio EDUCAUSE Connect april 22, 2015.
Social Research Methods
Survey Design Web Surveys.
Business and Management Research
Introduction to Survey Design
Business and Management Research
Social Research Methods
SDLC Phases Systems Design.
Multi-Mode Data Collection
Presentation transcript:

Semester Review

The Tailored Design Method

Uses multiple motivational features in compatible and mutually supportive ways to encourage high quantity and quality of responses

The Tailored Design Method Premised on social exchange perspective on human behavior Assumes that the likelihood of responding is greater when the expected rewards outweigh the anticipated costs

The Tailored Design Method Gives attention to all aspects of contacting and communicating with respondents Encourages response by considering survey sponsorship, the nature of the population and variations within it, and content of questions

The Tailored Design Method Emphasizes reducing errors of coverage, sampling, nonresponse, and measurement

Coverage Error Occurs when all members of a population do not have a known, non-zero probability of selection Occurs when those who are excluded are different from those who are included

Sampling Error Results from surveying only some rather than all members of a population Represented by B, the bound on the error of estimation

Nonresponse Error Occurs when people selected do not respond and are different than those who do Nonresponse can occur at the level of items within a survey or at the level of the survey – MAR – MCAR

Measurement Error Occurs when responses are inaccurate or imprecise Primarily related to poor layout and poor design and wording of questions

Social Exchange and Surveys Addresses three central questions about design and implementation 1.How can the perceived rewards for responding be increased? 2.How can the perceived costs of responding be reduced? 3.How can trust be established so that people believe the rewards will outweigh the costs of responding?

Increasing Benefits Provide information about the survey Ask for help or advise Show positive regard Say thank you Support group values Give tangible rewards Make the questionnaire interesting Provide social validation Inform people that opportunities to respond are limited

Decreasing Costs Make it convenient to respond Avoid subordinating language Make the questionnaire short and easy to complete Minimize requests for personal or sensitive information Emphasize similarity to other requests or tasks to which a person has already responded

Establishing Trust Obtain sponsorship by legitimate authority Provide a token of appreciation in advance Make the task appear important Ensure confidentiality and security of information

Features that can be Tailored Survey mode – Singular or multiple Sample design – Type of sample – Number of units sampled Incentives – Type of incentive – Amount or cost of incentive – Before or after

Features that can be Tailored Contacts – Number of contacts – Timing of initial and subsequent contacts – Mode of each contact – Whether contacts will be personalized – Sponsorship information – Visual design of each contact – Text or words in each contact

Features that can be Tailored Additional materials – Whether to provide them at all – Type of materials (e.g., research report) – Visual design of materials – Text or wording of materials

Features that can be Tailored Questionnaire – Topics included – Length (duration, number of pages/screens, number of questions) – First page or screen – Visual design – Organization and order of questions – Navigation through questionnaire

Features that can be Tailored Individual questions – Topic (sensitive, of interest to the respondent) – Type (open-ended versus closed-ended) – Organization of information – Text or wording – Visual design

Coverage and Sampling

Central Terminology An element is an object on which a measurement is taken A population is a collection of elements to which an inference is made from a sample A sample is a collection of sampling units drawn from a frame or frames Sampling units are nonoverlapping collections of elements from the population that cover the entire population A frame is a list of sampling units

Central Terminology A completed sample is the units that respond Sampling error is the result of collecting data from only a subset, rather than all, units from a frame – Again, represented by B, the bound on the error of estimation

Coverage The degree to which the units in a sampling frame correspond to the population of interest Coverage is likely one of the most serious problems in most surveys

Coverage and Frame Problems

Reducing Coverage Error Central questions: – Does the list contain everyone in the survey population? – Does the list include people who are not in the study population? – How is the list maintained and updated? – Are the same sample units included on the list more than once? – Does the list contain other information that can be used to improve the survey?

An Overview of Crafting Good Questions

Issues to Consider 1.What survey mode(s) will be used to ask the questions? 2.Is the question being repeated from another survey, and/or will answers be compared to previously collected data? 3.Will respondents be willing and motivated to answer accurately? 4.What type of information is the question asking for?

Choosing Words and Forming Question 1.Make sure the question applies to the respondent 2.Make sure the question is technically accurate 3.Ask one question at a time 4.Use simple and familiar words 5.Use specific and concrete words to specify the concepts clearly 6.Use as few words as possible to pose the question 7.Use complete sentences with simple sentence structures 8.Make sure “yes” means yes and “no” means no 9.Be sure the question specifies the response task

Visual Presentation of Survey Questions 1.Use darker and/or larger print for the question and lighter and/or smaller print for answer choices and answer spaces 2.Use spacing to create subgrouping within a question 3.Visually standardize all answer spaces or response options 4.Use visual design properties to emphasize elements that are important to the respondent and to deemphasize those that are not 5.Make sure words and visual elements that make up the question send consistent messages 6.Integrate special instructions into the question where they will be used rather than including them as freestanding entities 7.Separate optional or occasionally needed instructions from the question stem by font or symbol variation 8.Organize each question in a way that minimizes the need to reread portions in order to comprehend the response task 9.Choose line spacing, font, and text size to ensure the legibility of the text

From Questions to a Questionnaire

General Premises The design of a questionnaire should consider how to motivate the recipient to respond It should also avoid measurement errors, ranging from order effects to item nonresponse

Guidelines for Ordering Questions

Ordering Questions General guidelines 1.Group related questions that cover similar topics, and begin with questions likely to be salient to nearly all respondents 2.Choose the first question carefully 3.Place sensitive or potentially objectionable questions near the end 4.Ask questions about events in the order that they occurred 5.Avoid unintended order effects

Guidelines for Creating a Common Visual Stimulus

Visual Stimulus General guidelines 1.Establish consistency in the visual presentation of questions (across pages and screens) and use alignment and vertical spacing to help respondents organize information on the page 2.Use color and contrast to help respondents recognize the components of the questions and the navigational path through the survey 3.Visually group related information in regions through the use of contrast and enclosure 4.Use visual elements and properties consistently across questions to emphasize or deemphasize certain types of information 5.Avoid visual clutter 6.Minimize the use of matrixes and their complexity

Guidelines for Mail Questionnaires

Mail Questionnaires General guidelines 1.Determine whether keypunching or optical imaging and scanning will be used, and assess the limitations imposed on designing and processing questionnaires 2.Construct paper questionnaires in booklet formats, and choose physical dimensions based upon printing and mailing considerations 3.Decide question layout and how questions will be arranged on each page 4.Use symbols, contrast, size, proximity, and pagination effectively when designing branching instructions to help respondents correctly execute them 5.Create interesting and informative front and back pages that will have wide appeal to respondents 6.Avoid placing questions side-by-side on a page so that respondents are asked to answer two questions at once

Guidelines for Web Questionnaires

Web Questionnaires General guidelines 1.Decide whether an electronic alternative is appropriate 2.Choose how the survey will be programmed and hosted, commensurate with needs, skills, and sponsorship 3.Take steps to ensure that questions display similarity across platforms, browsers, and user settings 4.Decide how many questions will be presented on each page and how questions will be presented 5.Develop a screen format that emphasizes the respondent rather than the sponsor

Web Questionnaires General guidelines 6.Use a consistent page layout across screens and visually emphasize questions information that respondents will need to complete the survey while deemphasizing information not essential to the task 7.Do not require responses unless absolutely necessary 8.Design survey-specific and item-specific error messages to help respondents troubleshoot 9.Evaluate carefully the use of interactive features, balancing improvements in measurement with the impact on respondent burden and the implications with mixed-mode surveys

Web Questionnaires General guidelines 10.Use audiovisual capabilities sparingly, and evaluate the differential effect they have on respondents 11.Allow respondents to stop the survey and finish completing it at another time 12.Whenever possible, collect paradata that provide feedback on how respondents interact with questionnaire 13.Test the survey using a variety of platforms, connection speeds, browsers, and user-controlled settings, and test the database to ensure that items are collected and coded accurately 14.Take screenshots of each page of the final questionnaire for testing and documentation

Guidelines for Pretesting Questionnaires

Pretesting Questionnaires General guidelines 1.Obtain feedback on the draft of the questionnaire from a number of people, each of whom has specialized of some aspect of questionnaire quality 2.Conduct cognitive interviews of the complete questionnaire in order to identify wording, question order, visual design, and navigational problems 3.When the stakes are high, consider experimental evaluations of questionnaire components 4.Conduct a small pilot study with a subsample of the population in order to evaluate interconnections among questions, the questionnaire, and implementation procedures

Implementation Procedures

Mail Survey Implementation

When high quality implementation procedures are used, response rates of 50%-70% are not uncommon – In my own work, I have had response rates as high as 90% Implementation must receive considerable thought and planning – It should not occur after questionnaire design, but simultaneously

Mail Survey Implementation General guidelines 1.To the extent possible, personalize all contacts to respondents (even when names are unavailable) 2.Send a token of appreciation with the survey request 3.Use multiple contacts, each with a different look and appeal 4.Carefully and strategically time all contacts 5.Select all mail-out dates with the characteristics of the population in mind

Mail Survey Implementation General guidelines 6.Place information in the mailing exactly where it needs to be used 7.Take steps to ensure that mailings will not be mistaken for junk mail or marketing materials 8.Evaluate the size and weight of mailing materials on mailing costs 9.Assemble the mailings in a way that maximizes the appealing aspects of each element when the package is opened

Mail Survey Implementation General guidelines 10.Ensure that all addresses in the sample comply with current postal regulations 11.Assign an individual ID number to each sample member 12.Establish procedures for dealing with undeliverable mail 13.Establish procedures for dealing with returned incentives 14.Establish procedures for dealing with respondent inquiries 15.Evaluate early returns for problems that can be addressed mid-stride

Prenotice Letter Sent to respondents a few days prior to the questionnaire It notes that a questionnaire is for an important survey that will arrive in a few days and that the person’s response will be greatly appreciated This letter can increase response rates by 3%-6% – No recent research comparing postcards to letters (I prefer the former)

Questionnaire Mailing Includes a detailed cover letter explaining why a response is important The questionnaire A prepaid postage envelope A token incentive if one is to be provided

Thank You Postcard Sent a few days to a week after the questionnaire The mailing expresses appreciation for responding and indicates that if the completed questionnaire has not yet been mailed it is hoped that it will be returned soon

Replacement Questionnaire Sent to respondents 2 to 4 weeks after the previous questionnaire mailing It indicates that the person’s completed questionnaire has not yet been received and urges the recipient to respond

Final Contact Sent using a different mode of delivery 2 to 4 weeks after the previous mailing The different mode of contact distinguishes each type of final contact from regular mail delivery Special contacts have been shown to increase response

Web Survey Implementation

Major concerns associated with web surveys – Mode of initial contact – Extent to which personalized elements can be used – Whether and how incentives are delivered – Timing of contacts

Web Survey Implementation General guidelines 1.To the extent possible, personalize all contacts to respondents 2.Send a token of appreciation with the survey request 3.Use multiple contacts and vary the message across them 4.Carefully and strategically time all contacts with the population in mind 5.Consider contacting respondents by another mode when possible

Web Survey Implementation General guidelines 6.Keep contacts short and to the point 7.Take steps to ensure that s are not flagged as spam 8.Carefully select the sender name and address the subject line text for communications 9.Provide clear instructions for how to access the survey 10.Make obvious connections between the opening screen and other implementation features

Web Survey Implementation General guidelines 11.Assign each sample member a unique ID number 12.Know and respect the capabilities and limits of the web server(s) 13.Establish a procedure for dealing with bounced s 14.Establish procedures for dealing with returned incentives 15.Establish procedures for dealing with respondent inquiries 16.Implement a system for monitoring progress and evaluating early completes

When More than one Survey Mode is Needed

Why Consider a Mixed-Mode Survey Design Lower costs Improve timeliness Reduce coverage error Deliver incentives

Four Types of Mixed-Mode Surveys

Type I Use one mode to contact respondents and to encourage response by a different mode

Type II Use a second mode to collect responses from the same respondents for specific questions within a questionnaire

Type III Use alternative modes for different respondents in the same survey period

Type IV Use a different mode to survey the same respondents in a later data collection period

Mixed-Mode Surveys and their Implications TypeMotivationLimitations I.Use one mode to contact respondents and to encourage response by a different mode Improve response rates Reduce coverage and nonresponse error Increased implementation costs II.Use a second mode to collect responses from the same respondents for specific questions within a questionnaire Reduce measurement error Reduce social desirability bias for sensitive question Increased design costs Increased nonresponse if respondent must respond by other mode at a later time III.Use alternative modes for different respondents in a the same survey period Improve response rates Reduce coverage and nonresponse error Reduce survey costs Increased design costs Measurement error from differences that may be confounded with differences among subgroups IV.Use a different mode to survey the same respondents in a latter data collection period Different modes become available to survey respondents Reduce survey costs Increased design costs Measurement error from mode differences that impact the ability to measure change over time

Why Different Survey Modes Sometimes Produce Different Answers to Survey Questions

Presence versus Absence of an Interviewer Locus of control Normative question order effects Social desirability Acquiescence

Aural versus Visual Communication Effects Primacy/recency effects