EMR 6500: Survey Research Dr. Chris L. S. Coryn Spring 2012

Agenda The tailored design method Coverage and sampling Case Study #1

The Tailored Design Method

Uses multiple motivational features in compatible and mutually supportive ways to encourage high quantity and quality of responses

The Tailored Design Method Premised on social exchange perspective on human behavior Assumes that the likelihood of responding is greater when the expected rewards outweigh the anticipated costs

The Tailored Design Method Gives attention to all aspects of contacting and communicating with respondents Encourages response by considering survey sponsorship, the nature of the population and variations within it, and content of questions

The Tailored Design Method Emphasizes reducing errors of coverage, sampling, nonresponse, and measurement

Coverage Error Occurs when all members of a population do not have a known, non-zero probability of selection Occurs when those who are excluded are different from those who are included

Sampling Error Results from surveying only some rather than all members of a population Represented by B, the bound on the error of estimation

Nonresponse Error Occurs when people selected do not respond and are different than those who do Nonresponse can occur at the level of items within a survey or at the level of the survey – MAR – MCAR

Measurement Error Occurs when responses are inaccurate or imprecise Primarily related to poor layout and poor design and wording of questions

Competing Perspectives Economic exchange view of survey response Psychological models of survey response Leverage-saliency theory of survey response Social exchange theory of survey response

Economic Exchange Use monetary rewards as the primary motivation for seeking responses Widely adopted, especially in panel surveys

Psychological Models Extrinsic and intrinsic considerations motivate respondents Guided by social psychological concepts such as scarcity of opportunity, consistency with previous behavior, desire to reciprocate, enjoyment of task, and social proof

Leverage-Saliency Theory Respondents are differentially motivated by different aspects of the survey (leverage) and by how much emphasis is placed on each aspect by the surveyor (salience) Overemphasis on a single appeal that is attractive to some is not to others

Social Exchange Theory Premised on actions being motivated by the return that actions are expected to bring from others Simply, rewards are greater than costs

Social Exchange and Surveys Addresses three central questions about design and implementation 1.How can the perceived rewards for responding be increased? 2.How can the perceived costs of responding be reduced? 3.How can trust be established so that people believe the rewards will outweigh the costs of responding?

Increasing Benefits Provide information about the survey Ask for help or advise Show positive regard Say thank you Support group values Give tangible rewards Make the questionnaire interesting Provide social validation Inform people that opportunities to respond are limited

Decreasing Costs Make it convenient to respond Avoid subordinating language Make the questionnaire short and easy to complete Minimize requests for personal or sensitive information Emphasize similarity to other requests or tasks to which a person has already responded

Establishing Trust Obtain sponsorship by legitimate authority Provide a token of appreciation in advance Make the task appear important Ensure confidentiality and security of information

Features that can be Tailored Survey mode – Singular or multiple Sample design – Type of sample – Number of units sampled Incentives – Type of incentive – Amount or cost of incentive – Before or after

Features that can be Tailored Contacts – Number of contacts – Timing of initial and subsequent contacts – Mode of each contact – Whether contacts will be personalized – Sponsorship information – Visual design of each contact – Text or words in each contact

Features that can be Tailored Additional materials – Whether to provide them at all – Type of materials (e.g., research report) – Visual design of materials – Text or wording of materials

Features that can be Tailored Questionnaire – Topics included – Length (duration, number of pages/screens, number of questions) – First page or screen – Visual design – Organization and order of questions – Navigation through questionnaire

Features that can be Tailored Individual questions – Topic (sensitive, of interest to the respondent) – Type (open-ended versus closed-ended) – Organization of information – Text or wording – Visual design

Coverage and Sampling

Central Terminology An element is an object on which a measurement is taken A population is a collection of elements to which an inference is made from a sample A sample is a collection of sampling units drawn from a frame or frames Sampling units are nonoverlapping collections of elements from the population that cover the entire population A frame is a list of sampling units

Central Terminology A completed sample is the units that respond Sampling error is the result of collecting data from only a subset, rather than all, units from a frame – Again, represented by B, the bound on the error of estimation

Coverage The degree to which the units in a sampling frame correspond to the population of interest Coverage is likely one of the most serious problems in most surveys

Coverage and Frame Problems

Telephone Coverage Predominant sampling frame in the 1980s and 1990s – Random digit dialing (RDD) Since the introduction of the cellular telephone valid coverage is no longer possible – Approximately 18% of all households no longer have a landline – Differences between cellular phone users and traditional landline users

Internet Coverage Significant coverage gaps in the general population – Approximately 67% of the population has internet access in their homes – Only 47% have high speed conections Widely used for specific, targeted populations (e.g., students, professionals) No equivalent to the RDD algorithm

Mail Coverage As with telephone, widely used until the 1990s Increasingly unlisted telephone numbers (and addresses) Changes in social norms (e.g., double- listing of spouses with different last names) Address-based sampling using U.S. Postal Service DSF (all delivery point addresses) – Can be geo-coded

Reducing Coverage Error Most surveyors are interested in specialized subpopulations rather than the general population In certain instances, valid sampling frames can be established

Reducing Coverage Error Central questions: – Does the list contain everyone in the survey population? – Does the list include people who are not in the study population? – How is the list maintained and updated? – Are the same sample units included on the list more than once? – Does the list contain other information that can be used to improve the survey?

Respondent Selection Should be carefully coupled to the focal question of the study – Most recent birthday method (if interest is in adult population) – Greatest responsibility (if interested in household behaviors)

Coverage Outcomes Careful coverage analysis – Multiplicity Duplicate units in the sampling frame – Overcoverage Units included in the sampling frame that are not in the target population Units that do not meet inclusion criteria – Undercoverage Units that are not in the sampling frame but that are part of the target population Units that meet inclusion criteria

Probability Sampling Only method that allows the statistical properties of estimators to be assess probabilistically Always the preferred method for sampling, even in small finite populations

Sample Size It is the size of the sample, not the proportion of a population sampled, that determines precision

Basic Rationales Relatively few responses can provide precise estimates In large populations there is virtually no difference in the number of sampled units needed for a given level of precision In small populations greater proportions need to be sampled for a given level of precision In large samples additional increases yield small reductions in error Sample sizes need to be larger if interest is in subpopulations

Case Study #1

Case Study Activity In small groups, address the following questions in relation to Case Study #1 relying only on the material that was discussed in today’s lecture and readings 1.Has the surveyor committed any serious error(s)? 2.If so, what type and why? If not, why?