Basic Survey Design Techniques

Basic Survey Design Techniques
NOTE TO INSTRUCTORS: This training was designed to be eligible for non-government CPE hours; however, your individual audit/program evaluation office will need to make this determination. This training was designed to be augmented with examples and information regarding how survey techniques are used in your individual audit/program evaluation office. The anticipated training duration is 2 ½ to 3 hours. The training should be taught by someone familiar with survey techniques who can add and adapt the course content to meet your specific needs. Participant skill levels can be varied.

Objectives To gain an overall understanding of the survey cycle.
To understand the basics of survey design and question development. To consider the advantages and disadvantages of certain design choices and their effects on the survey results. To provide practical examples of how survey techniques have been applied and used in audits/program evaluations. Opening Questions: Who has had exposure to survey design techniques before? Where? In what ways? Who has participated in a survey before as a respondent? What kind of survey was it? What was its focus? What are your behaviors when you respond to a survey? For example, do you react to something about the survey, such as the question phrasing, the person administering the survey, the organization sponsoring it, or how the results might be used? Why do you behave in this manner? Communicate to participants that even if they aren’t going to conduct their own survey, familiarity with survey design techniques should provide them a basis for evaluating agencies’ practices. Basic Survey Design Techniques

Group Discussion Question #1
What is a survey? Surveys are a structured way of collecting information from a subject (e.g., individuals, organizations). Have the group brainstorm on creating a list of characteristics. Some characteristics might include: large number of respondents standardized questions and responses (though not always!) use of a sample to gather representative information about a population Be sure to explain that a survey is different from structured interviews, though they share many similar characteristics. Basic Survey Design Techniques

Group Discussion Question #2
When should I use a survey? Have the group brainstorm on creating a list of circumstances. Some circumstances might include: when information is not available in existing reports, studies, or databases file reviews, site visits, and interviews are impractical self-reported information and opinions are credible and necessary for conclusions representative information is needed from a large group of people and a contact list (or close proxy) is available straight forward questions will provide the information needed Basic Survey Design Techniques

The Survey Cycle DEVELOPMENT ADMINISTRATION REPORTING DATA
Plan your survey with the whole cycle in mind! Sometimes you will never get beyond development (e.g., your funding is cut, your project needs change). Sometimes you will spend significant resources in data cleaning and analysis or mine the same source data for multiple reporting purposes (academics are great at doing this). For audit/program evaluation shops with tight time frames, the whole cycle must often be completed in a matter of weeks (e.g., during your fieldwork). REPORTING DATA This training focuses on the Development Phase. Basic Survey Design Techniques

Development During the development phase, you are focusing on questions and issues related to: Overall survey goal and objective Sample selection Methodology Layout Questions and response sets This phase is the focus of this training. But it is always important to know what follows and where you want to end up, even at the development stage. Basic Survey Design Techniques

Administration During the administration phase, you are focusing on questions and issues related to: Pre-testing Approach Survey administration Close and follow up Basic Survey Design Techniques

Data During the data phase, you are focusing on questions and issues related to: Data entry Validation Analysis This phase can take a long time if you do not adequately plan and keep an eye on your time constraints. You could spend the whole audit project simply cleaning and analyzing the data. Concluding on your analysis is critical and an important part of moving you to the reporting phase. Basic Survey Design Techniques

Reporting During the reporting phase, you are focusing on questions and issues related to: Reporting the results of your survey Many different reporting options are available. Pick the one that serves your needs and best fits with your intended audience. Basic Survey Design Techniques

Establishing Goals The first step in any survey is deciding what you want to learn. The goals of the project determine who you will survey and what you will ask them. The more specific your goals, the easier it will be to get usable answers. If your goals are unclear, the results will probably be unclear. Sample goals include learning more about: The potential market for a new product or service. Ratings of current products or services. Employee attitudes. Customer/patient satisfaction levels. Reader/viewer/listener opinions. Association group, organization opinions. Opinions about political candidates or issues. Corporate images. Basic Survey Design Techniques

Sample Selection Surveys typically involve gathering information from a sample of individuals to be able to say something about the population. Lots of sampling methods available. The decision of which method to use is often driven by the study objectives, the level of precision required, and the resources available. You must have a good understanding of how your chosen sampling method affects what you can and cannot say. It is incumbent on the researcher to clearly define the target population. There are no strict rules to follow, and the researcher must rely on logic and judgment. Although we commonly think of surveys involving a statistically representative sample, this need not be the case… Sometimes, the entire population will be sufficiently small such that you can survey an entire population (e.g., all school districts, all legislators). This type of research is called a census study because data is gathered on every member of the population. Usually, the population is too large to attempt surveying all of its members. A small, but carefully chosen sample can be used to represent the population. The sample reflects the characteristics of the population from which it is drawn. Sometimes the population is readily identifiable. For example, if you are conducting an employee morale survey or patient/customer satisfaction survey, the population is obvious. If you are trying to determine the likely success of a change in procedure, however, the population may be less obvious. Correctly determining the target population is critical. If you do not survey the correct individuals, you will not successfully meet your goals for the survey. Sampling methods are classified as either probability or nonprobability (refer to handout). In probability samples, each member of the population has a known non-zero probability of being selected: random sampling systematic sampling stratified sampling In nonprobability sampling, members are selected from the population in some nonrandom manner: convenience sampling judgment sampling quota sampling snowball sampling Basic Survey Design Techniques

SAMPLING METHODS Probability Samples: Each member of the population has a known non-zero probability of being selected. Probability methods include random sampling, systematic sampling, and stratified sampling. The advantage of probability sampling is that sampling error can be calculated. Sampling error is the degree to which a sample might differ from the population. When inferring to the population, results are reported plus or minus the sampling error. Random sampling is the purest form of probability sampling. Each member of the population has an equal and known chance of being selected. When there are very large populations, it is often difficult or impossible to identify every member of the population, so the pool of available subjects becomes biased. Systematic sampling is often used instead of random sampling. It is also called an Nth name selection technique. After the required sample size has been calculated, every Nth record is selected from a list of population members. As long as the list does not contain any hidden order, this sampling method is as good as the random sampling method. Its only advantage over the random sampling technique is simplicity. Systematic sampling is frequently used to select a specified number of records from a computer file. Stratified sampling is commonly used probability method that is superior to random sampling because it reduces sampling error. A stratum is a subset of the population that share at least one common characteristic (e.g., gender or geographic location). The researcher first identifies the relevant stratums and their actual representation in the population. Random sampling is then used to select a sufficient number of subjects from each stratum. “Sufficient” refers to a sample size large enough for us to be reasonably confident that the stratum represents the population. Stratified sampling is often used when one or more of the stratums in the population have a low incidence relative to the other stratums. Nonprobability Samples: Members are selected from the population in some nonrandom manner. These include convenience sampling, judgment sampling, quota sampling, and snowball sampling. In nonprobability sampling, it is not possible to calculate a sampling error. Thus, the degree to which the sample differs from the population remains unknown. Convenience sampling is used in exploratory research where the researcher is interested in getting an inexpensive approximation of the truth. As the name implies, the sample is selected because they are convenient. This nonprobability method is often used during preliminary research efforts to get a gross estimate of the results, without incurring the cost or time required to select a random sample. Judgment sampling is a common nonprobability method. The researcher selects the sample based on judgment. This is usually an extension of convenience sampling. For example, a researcher may decide to draw the entire sample from one “representative” city, even though the population includes all cities. When using this method, the researcher must be confident that the chosen sample is truly representative of the entire population. Quota sampling is the nonprobability equivalent of stratified sampling. Like stratified sampling, the researcher first identifies the stratums and their proportions as they are represented in the population. Then convenience or judgment sampling is used to select the required number of subjects from each stratum. This differs from stratified sampling, where the stratums are filled by random sampling. Snowball sampling is a special nonprobability method used when the desired sample characteristic is rare. It may be extremely difficult or cost prohibitive to locate respondents in these situations. Snowball sampling relies on referrals from initial subjects to generate additional subjects. While this technique can dramatically lower search costs, it comes at the expense of introducing bias because the technique itself reduces the likelihood that the sample will represent a good cross section from the population. NOTE TO INSTRUCTORS: The information contained on this slide is intended to be put into a separate handout for course participants. It is not part of the PowerPoint presentation, but it is associated with the previous slide entitled “Sample Selection”. Basic Survey Design Techniques

Survey Method Your survey method refers to your method of data collection: In-person interviews Telephone surveys Mail surveys surveys Online (web-based) surveys Each method has advantages and disadvantages. Basic Survey Design Techniques

In-Person Interviews PROS
Chance for subject to “experience” the survey. Some people may be more willing to tolerate a longer survey if done in person. CONS Significant resource investment required. Not as timely as other methods. Researcher-administered surveys are often also referred to as structured interviews. Individuals administering in-person surveys often have extensive training in the survey to be able to elicit responses and move through the survey questions without creating biased responses. The goal is to create a uniform application so that the survey data can be reliably aggregated and variability in the data is not simply the result of differences in the way the interviewer asked the questions. Basic Survey Design Techniques

Telephone Surveys PROS Faster administration.
Potential for near universal coverage (96% of homes have a telephone). Ties to CATI (computer-assisted telephone interview) software. CONS People are suspicious of telephone surveys (e.g., they suspect a sales call). Many new screening tools (e.g., caller ID, voice mail). When tied with CATI software, the results can be available minutes after completing the last interview. Basic Survey Design Techniques

Mail Surveys PROS Among the least expensive to administer.
Not considered as intrusive since respondents answer at their leisure. CONS Mail surveys take a long time (initial mailing, response time, follow up). Response rates can often be low or varied. Boost response rates with: pre-approach mailing (e.g., tell people they will soon receive a survey in the mail) approach letter with the survey postage-paid return envelope follow-up reminder with another copy of the survey other incentive (e.g., ask for assistance in improving government ops, make recommendations to agency for improvements, etc.) distribution of the questionnaire is helping to address some of the timeliness issues regarding survey administration. But surveys can have problems with generalizing to broad populations. Basic Survey Design Techniques

Online (Web-Based) Surveys
PROS Fast and economical. Data entry and validation is integrated. Filter and other structured question logic can be used. CONS Respondents must have computer access and/or . Generalization problems. Software limitations. Data confidentiality. There is a difference between a survey that is administered online (via a web interface) versus simply ing respondents an electronic copy of the questionnaire. More and more state auditor offices and program evaluation offices are using sites like SurveyMonkey to administer web-based surveys. However, be sure to think about using an outside entity should the survey deal with protected or confidential information. Look at the terms of use and consider whether contracting with a service provides additional safeguards to your office. Basic Survey Design Techniques

Survey Method In general, your choice of survey method will depend on several factors: Population being targeted Goals of the survey Speed of administration Need to ask sensitive questions Cost Internet usage of respondents Literacy levels of respondents Speed – and web-based surveys are the fastest, followed by telephone surveys and in-person interviews. Mail surveys are the slowest. Cost – In-person interviews are the most expensive, followed by telephone and mail surveys. and web-based surveys are the least expensive for large samples. Internet Usage – Web-based and surveys offer significant advantages, but their results may not be able to be generalized to the broader population at-large. Literacy Levels – Illiterate and less-educated people rarely respond to mail surveys. Sensitive Questions – People are more likely to answer sensitive questions when the survey is administered by a computer in one form or another. Basic Survey Design Techniques

Survey Layout – User Principles
Navigability – movement through the survey, from one page to another, one section to another, one question to another. User instructions. Section breaks. Question and content groupings. Usability – The ease with which respondents are able to use/interface with the survey. Heavily affected by the chosen survey method. Keep the survey short and simple if at all possible. There are two broad issues to keep in mind when considering the layout (i.e., the question and answer choice order) for your survey. First, is how the question and answer choice order can encourage people to complete your survey. Second, is how the order of questions or the order of answer choices could affect your survey results. Question order can affect your results in two ways. One is that mentioning something (an idea, an issue, a product) in one question can make people think of it while they answer a later question, when they might not have thought of it if it had not been previously mentioned. In some cases, you may be able to reduce this problem by randomizing the order of related questions. Separating related questions with unrelated questions can also reduce this problem. However, you have to consider how such a move will affect the survey’s usability from the respondent’s perspective. Another way that question order and affect results is “habituation.” People tend to start giving the same answer, without really considering it, after being asked a series of similar questions with similar answer choices. Basic Survey Design Techniques

Survey Layout Provide instructions. Tell the respondent how to complete the survey, mark the questions, etc. Establish a logical flow to the overall survey, as well as to each section. Questions that the respondent is most likely to perceive as most important should come first. When possible, place difficult or sensitive questions toward the end of your survey. Use transitions or headings to denote different groups of questions. Put demographic questions at the end of the survey. Basic Survey Design Techniques

Question and Response Sets
Driven by your survey goals and objectives. Questions and response sets are interrelated. Driven by how you anticipate analyzing and reporting on the data. Garbage in, garbage out. You cannot simply throw together a bunch of questions and expect your survey to be successful. Basic Survey Design Techniques

10 Qualities of a Good Question
Is relevant. Evokes the truth. Asks for a response on only one dimension. Can reasonably accommodate all possible responses. Has mutually exclusive options. Produces variability of responses. Does not presuppose a current state of affairs. Does not imply a desired response. Does not use emotionally loaded or vaguely defined words. Does not use unfamiliar words or abbreviations. Irrelevant questions confuse the respondent and add to the burden (and cost) of completing the survey. Also, irrelevant questions are not useful for data analysis. Ask yourself what you will do with the information from each question. If you cannot give a good answer, leave the question out. Avoid the temptation to add a few more questions just because you are doing the survey anyway. To help look for irrelevant questions (not to mention reduce the length of your survey), group questions into 3 categories: must know, useful to know, and nice to know. If a question cannot be placed into one of these categories, get rid of it. Furthermore, even consider discarding the “nice to know” group unless the other sections are very short. Questions must be non-threatening. When a respondent is concerned about the consequences of answering a question in a particular manner, there is a good possibility that the response will not be truthful. Anonymous or confidential questionnaires are more likely to produce honest responses. If your survey does contain sensitive items, be sure to clearly state your policy on confidentiality in the approach letter. The purpose of a survey is to find out information. A question that asks for a response on more than one dimension will not provide the information you are seeking. Sometimes it is necessary to split a question into two questions: “Do you plan to leave the school district and quit teaching in the next two years?” – Are we asking about whether the respondent is considering leaving the school district or teaching altogether, or both? Alternative Question #1 -- “Do you plan to quit working for this school district within the next two years?” Alternative Question #2 -- “Do you plan to quit teaching in the next two years?” Multiple choice items are the most popular type of survey questions because they are generally the easiest for a respondent to answer and the researcher to analyze. However, asking a question that does not reasonably accommodate all possible responses can confuse and frustrate the respondent. A good question leaves no ambiguity in the mind of the respondent. There should only be one appropriate choice for the respondent to make. “Where did you grow up?” (A-country; B-farm; C-city) – A person who grew up on a farm in the country would not know whether to select A or B. When a question produces no variability in responses, we are left with considerable uncertainty about why we asked the question and what we learned from the information. If a question does not produce variability in responses, it may not be possible to perform any analysis on the resulting data. Among the most subtle mistakes in questionnaire design are questions that make an unwarranted assumption. “Are you satisfied with the State’s online unemployment insurance system?” (Yes or No) – Issues can be avoided by adding an additional response category acknowledging that not all respondents may use the online system (e.g., some may apply over the phone or in person). “What percentage of your budget do you spend on monitoring boilers?” (Fill in the blank) – You are assuming that the respondent knows the information you seek, but understand that the respondent may not know the answer to this question without looking it up, and very few may take the time to do this. If you ask questions similar to this, know that the responses are rough estimates and there is a strong likelihood of error. Also, consider whether the information you seek is available in more definitive source. The wording of a question is extremely important. We are striving for objectivity in our surveys and, therefore, must be careful not to lead the respondent into giving the response we would like to receive. Leading questions can sometimes be easily spotted because they use negative phraseology: “Don’t you think the number of uninsured individuals in the state is too high?” “Isn’t there something more that the State could do to ensure better radio communications for emergency response personnel?” Quantifying adjectives (e.g., most, least, majority) are frequently used in questions, but it is important to understand that these words mean different things to different people. Remember who your audience is and write your questionnaire for them. Complex terms and abbreviations are okay only if you are absolutely certain that every single respondent will understand their meanings. If there is any doubt at all, provide a definition or consider an alternative word or words. “How does PPAV affect your willingness to seek Tier 1 funding?” – This question assumes the respondent knows what PPAV is, what Tier 1 funding is, and also how one affects the other in the decisionmaking process. This is a very complex question. Basic Survey Design Techniques

Example: Lengthy Question
Students must take standardized tests in reading, writing, and mathematics. Some teachers feel that students take too many tests. Others feel more should be required to test students in science and social studies. Do you think that your students take too many standardized tests? What options exist to make this question less lengthy? NOTE TO INSTRUCTORS: This slide plus the following four slides provide five examples of problem questions that need to be reworded. Depending on the number of participants, the suggested approach is to split into five groups and have each group address one of the questions. Then each group should report out its revisions to the larger group for discussion. Basic Survey Design Techniques

Example: Complex Question
Does it seem possible or impossible that the Cubs will win the World Series, or do you feel they will always find a way to lose?” What options exist to make this question less complex? Basic Survey Design Techniques

Example: Complex Question
Including yourself, any pets you own, but not including any relatives or friends that do not live in your household for at least 320 days a year, how many household occupants did you buy groceries, by groceries we mean any item of food bought at the grocery store, for in the last year? What options exist to make this question less complex? The problem with asking questions like this is that you are going to tax your respondents memory and some of them will not even get what you are trying to say. Individuals differ in how much their working memory can handle. Questions that pose a burden to some may not pose a burden to others. Avoid complex syntax in your survey questions. This means that its grammatical composition cannot be dense, structurally ambiguous, or not well formed syntactically. Memory overload can occur with extremely long sentences, using too many logical operators (such as or, and, if-else, then), and quantifiers. In order to make responses more precise it is better to ask a series of simpler questions. Basic Survey Design Techniques

Example: Leading Question
State employees love their jobs. How do you feel about your job with the State? What options exist to make this question more neutral? Basic Survey Design Techniques

Example: Loaded Question
Do you believe that we should redistribute wealth by allowing tax credits for the rich and wealthy to expire? What options exit to remove the loaded language in this question? Rewrite the question to remove the loaded language. In order to obtain accurate results a survey question should be neutral in order to welcome as many points of views as possible. A loaded question contains an incriminating assumption if the respondent accepts it to be true. The loaded word here is “redistribute” as it is often associated with socialism. The question is asked in such a way that a respondent may not feel comfortable giving an honest answer. Injecting emotional language in surveys can bias survey results. Alternative Question: “Do you believe that the government should allow certain tax credits available to individuals in higher tax brackets to expire?” Basic Survey Design Techniques

Response Sets Response sets are an integral part of the question design and must be carefully considered along with the wording of the question. Think about how you are going to code and analyze the data when putting together your response sets. You must remember that the way in which you put together a response set depends in large part on the question’s phrasing. Basic Survey Design Techniques

Response Sets Ordinal Questions – Responses have an underlying order. However, the numeric value assigned to the response may or may not have intrinsic meaning. Ranking/Rating Questions – Responses are measured based on defined intervals (e.g., scale of 1 to 5). Guttman Scales Likert Scales Symantic Differential Scales Nominal Questions – Responses have no underlying order. Numeric values are assigned simply as placeholders, but have no inherent meaning. Basic Survey Design Techniques

Response Sets Single Option Questions – The respondent can only select one response option. Multiple Option Questions – The respondent can select more than one response option (e.g., mark all that apply). Filter Questions – Used to screen and direct respondent’s progression through the survey based on a set of logical parameters. Open-Ended Questions – The response format is unstructured. The respondent can enter any type of information. Basic Survey Design Techniques

NOTE TO INSTRUCTORS: The information contained on this slide is intended to be put into a separate handout for course participants. It is not part of the PowerPoint presentation, but it is associated with the previous slides entitled “Response Sets”. Basic Survey Design Techniques

Response Set Principles
Consistency: Responses are orderly, predictable, describable by a few rules, and therefore easy for the respondent to learn and retain. Completeness: Response choices reasonably reflect the full range of possible answers to a question. Failure to follow these two principles generally has two outcomes: Respondents will get confused Error is introduced into your survey data CONSISTENCY Whenever there is a logical or natural order to responses, use it. Present agree-disagree responses in that order. Presenting responses in a disagree-agree order could seem odd to the respondent. For the same reason, positive to negative and excellent to poor scales should be presented in that order. When using numeric rating scales, higher numbers should mean a more positive or more agreeing answer. Yes is typically followed by No. Discuss example on handout: Questions 1 through 3 are consistent with one another. The rating scales have 5 points with the same anchors and values. The scale goes from Very Satisfied (5) to Very Unsatisfied (1). However, Question 4 is inconsistent. It still has 5 points with the same values, but the anchors are reversed. This scale goes from Very Unsatisfied (5) to Very Satisfied (1). Some researchers suggest that you vary the order of the response sets (especially when using a battery of agree-disagree type questions) to minimize respondents’ habitual response tendencies. However, a better approach would be to keep the response set the same and instead alter the lead-in statement: “My supervisor usually ignores my suggestions.” “My supervisor gives me positive feedback.” In the first statement a high level of agreement would generally be seen as positive, whereas a high level of agreement with the second statement would generally be seen a negative. When altering response sets and questions, recoding of the data to reverse the scales is required prior to starting the analysis. COMPLETENESS Remember, one of the 10 principles of a good survey question is to evoke the truth. Respondents who feel they are being forced into giving an answer they do not want to give either do not complete the survey, or will give responses that are “in error.” Usefulness of don’t know, not applicable, and no opinion type responses. These responses ultimately become missing data and are excluded from your analysis (unless your study is specifically targeting those with non opinions). However, these are often necessary responses to avoid frustrated respondents. The rule of thumb is to provide these types of categories whenever these are a logically possible response to a question. The completeness of response sets applies to more than just “don’t know” type responses. Sometimes leaving out a relevant choice can yield misleading results. Polls that use a simple yes/no response format as in Question #1A have found about 70-75% of the respondents choosing “Yes.” Polls that offer a choice between the death penalty and life in prison without the possibility of parole as in Question #1B show support for the death penalty at about 50-60%. Polls that offer a third option of inmates working in prison to pay restitution to their victims’ families have found support for the death penalty even closer to 30%. So, what is the true level of support for the death penalty? The lowest figure is probably the truest, since it represents the percentage of respondents that favor that option regardless of the alternatives offered. Question #2A does not provide a response set that covers all possible answers. The respondent, if they even own a car, may alternate between fuel types. Additionally, how would someone with a diesel car or a hybrid or electric vehicle respond to such a question? Question #2B provides a more complete response set and is more likely to yield accurate results. People like to think of themselves as normal or average, or to provide responses they think will reflect well on them. This is a constant problem in surveys, and one that generally can only be mitigated by a well thought out response set. However, be aware that the range of answer choices you provide when asking for a quantity or frequency can affect the results: Questions #3A and #3B each have the same lead in question and also offer six possible responses. However, if you use the response set in Question #3A fewer people will pick “4 hours or more” than will pick “4 hours” in Question #3B. This is because Question #3A makes the 4 hours appear to be an extreme response, whereas in Question #3B 4 hours seems typical. Basic Survey Design Techniques

NOTE TO INSTRUCTORS: The information contained on this slide is intended to be put into a separate handout for course participants. It is not part of the PowerPoint presentation, but it is associated with the previous slide entitled “Response Set Principles”. Basic Survey Design Techniques

Survey Pretest The last step in survey questionnaire design is to pretest it. Ideally you should test the survey with the same kinds of people you will include in the main study. If this is not possible, have a few people other than the question writer try the questionnaire. Pretests are invaluable for revealing problems with question wording, response sets, instructions, and respondent understanding. Basic Survey Design Techniques

Examples NOTE TO INSTRUCTORS: This training was designed to be augmented with examples and information that may be specific to your individual audit/program evaluation office. You need to identify and provide examples to course participants that illustrate how your office has used surveys when conducting audits or program evaluations. For each example, use participants to discuss (1) how the survey was used as part of the audit goals and objectives, (2) how the survey was administered, (3) how the survey results were used and reported, and (4) any barriers or other issues you had to deal with while developing, administering, and reporting on the survey. Depending on the number of examples and amount of discussion expected, it is suggested that you allocate at least 30 minutes to this activity. Basic Survey Design Techniques

Basic Survey Design Techniques

Similar presentations

Presentation on theme: "Basic Survey Design Techniques"— Presentation transcript:

Similar presentations

About project

Feedback

Log in

Auth with social network:

Basic Survey Design Techniques

Similar presentations

Presentation on theme: "Basic Survey Design Techniques"— Presentation transcript:

Similar presentations

About project

Feedback