Survey design and sampling Friday 18 th January 2008

Outline Surveys Thinking about what you’re researching: case, population, sample Non-probability samples Probability samples Weighting Sampling error

Survey Analysis Is most often used where individuals are the unit of analysis (this is not always the case though, for example the Workplace Employment Relations Survey surveyed workplaces) Individuals, known as Respondents, provide data by responding to questions. The instrument that is used to gather data is called a questionnaire. Questionnaires: –Collect standardised information. –Are used to elicit information to be used in analysis.

Three Types of Surveys: 1.Self-administered Questionnaires Including: 1.Mailed Survey (or ) 2.Web-based surveys 3.Group Survey (i.e. in a classroom) 2.Interview Surveys (face to face) 3.Telephone Surveys (including CAT interviewing)

MethodAdvantagesDisadvantagesTips to Remember Self- completion Cheap Cover wide area Anonymity protected Interviewer bias doesn’t interfere People can take their time Low response rate (and possible bias in this) Questions need to be simple No control over interpretation No control over who fills it in Slow Simplify questions Include covering letter Include stamped addressed response envelope Send a reminder Telephone Survey Can do it all from one place Can clarify answers People may be relatively happy to talk on the phone Relatively cheap Quick People may not have home phones/be ex-directory You may get wrong person or call at wrong time May be a bias in whose name is listed/who’s at home Easy for people to break off No context to interview Because you rely totally on verbal communication – questions must be short and words easy to pronounce Minimize number of response categories (so people can remember them) Face-to- face interveiw High response rate High control of the interview situation Ability to clarify responses Slow Expensive Interviewer presence may influence way questions are answered If more than one interviewer, may have different effects Important that interviewer be non- threatening Interviewer can clarify questions, but be wary of elaborations that effect the content Aim to ask questions in a clear standardized way If there is a long list of possible responses, show these to respondent to read while the question is read out

Response Rate You must keep track of the response rate. This is calculated as the proportion of people who are qualified to take part in the survey (part of the sample) who actually participate. i.e. if you receive 75 surveys back from a sample of 100 people your response rate is 75% Example: –You are studying women over 50 and are stopping women in the street asking them their age, and if they qualify you are asking to interview them. –If you stop 30 women and 20 are under 50 and 10 over 50, your starting number (those qualified to take part) is 10. –If 5 of these are willing to talk to you, you have a 50% response rate (5/10) –Note: it is irrelevant that you originally stopped 30 women, your response rate is NOT 17% (5/30) – you ignore those people who are not qualified in calculating response rate.

Time as a Dimension in Survey Research Cross Sectional Studies Observations of a sample, or cross-section of a population or phenomena made at one point in time – most surveys are cross-sectional  leading to a common criticism of survey research: that it is ahistorical (although this critique also relates to its focus on the asocial individual). Longitudinal Studies Permits observations of the same phenomena over an extended period.  enables analysis of change.

Types of Longitudinal Study 1.Trend Studies – examines change within a population over time (i.e. the census) 2.Cohort Studies – examines specific subpopulations or cohorts (although not necessarily the same individuals) as they change over time (i.e. interview people who were age 30 in 1970; 40 in 1980; 50 in 1990; and 60 in 2000) 3.Panel Study – examines the same set of people each time (i.e. interview same sample of voters every month during a campaign).

Strengths of Survey Research Useful in describing the characteristics of a large population. Make large samples feasible. Flexible - many questions can be asked on a given topic. Has a high degree of reliability (and replicability). Is a relatively transparent process.

Weaknesses of Survey Research Can seldom deal with the context of social life. Inflexible – cannot be altered once it’s begun (therefore poor for exploratory research). Subject to artificiality – the product of respondents’ consciousness that they are being studied. Weak on validity. Poor at answering questions where individual is not the unit of analysis. Usually inappropriate for historical research. Particularly weak at gathering at certain sorts of information: –Highly complex or ‘expert’ knowledge –People’s past attitudes or behaviour –Subconscious (especially macro-social) influences –Attitudes (or at least embodied attitudes) –Shameful or stigmatized behavior or attitudes (especially in face-to- face interview) – although survey research may be able to achieve this in some circumstances.

Thinking about what you’re researching: Case, Population, Sample Case: each empirical instance of what you’re researching So if you’re researching celebrities who have been in trouble with the law Michael Jackson would be a case, as would Winona Ryder, Pete Doherty, Kate Moss, Boy George, George Michael and OJ Simpson… If you were interested in Fast Food companies McDonalds would be a case, Burger King would be a case, as would Subway… If you were interested in users of a homeless shelter on a particular night, each person who came to the shelter on the specified night would be a case.

Thinking about what you’re researching: Case, Population, Sample Population – all the theoretically relevant cases (i.e. “Tottenham supporters”). Note: This may be different to the study population, which is all of the theoretically relevant cases which are actually available to be studied (i.e. “all Tottenham club members or season ticket holders”).

Sometimes you can study all possible cases (the total population in which you are interested) For example: –Post WW2 UK Prime Ministers –Homeless people using a particular shelter on Christmas Day 2007 –National football teams in the 2006 World Cup –Secondary schools in Coventry

Often you cannot research the whole population because it’s too big and doing so would be too costly, too time consuming, or impossible. For example, if your ‘population’ is: –Voters in the UK since WW2 –All the homeless people in the UK on Christmas Day 2007 –Club and National Football teams involved in cup competitions in 2006 –Secondary schools in the UK On these occasions you need to select some cases to study. Selecting some cases from the total population is called Sampling

How you sample depends (among other things) on some linked issues: What you are especially interested in (what you want to find out) The frequency with which what you are interested in occurs in the population The size/complexity of the population What research methods you are going to use. How many cases you want (or have the resources/time) to study

Sample and population Much statistical analysis is done on a sample. However we are generally interested in population parameters (i.e. whether women in the UK earn more or less than men, not whether the 3,452 women in our study earn more on average than the 2,782 men in our study). Therefore statistical analysis usually involves techniques for inferring from the sample to the population.

Probability and Non-Probability Sampling Probability Samples Have a mathematical relationship to the total population: we can work out mathematically the likelihood (probability) of what is found within the sample being the same as what would be found within the whole population (if we were able to analyze the whole population).  Probability sampling allows us to make inferences about the whole population. Non-Probability Samples  Do not formally allow us to make inferences about the whole population. However there are often logistical reasons for their use, and (despite being statistically dodgy) inferential statistics are frequently employed (and published!).

Types of Non-probability Sampling: 1. Reliance on available subjects: Literally choosing people because they are available (i.e. approaching the first five people you see outside of the library) Only justified if less risky sampling methods are not possible. Researchers must exercise caution in generalizing from their data when this method is used.

Types of Non-probability Sampling: 2. Purposive or judgmental sampling Selecting a sample based on knowledge of a population, its elements, and the purpose of the study. Selecting people who would be ‘good’ informants. Used when field researchers are interested in studying cases that don’t fit into regular patterns of attitudes and behaviors (i.e. deviance). Relies totally on the researcher’s prior ability to determine ‘suitable’ subjects.

Types of Non-probability Sampling: 3. Snowball sampling Researcher collects data on members of the target population she can locate, then asks them to help locate other members of that population. Appropriate when members of a population are difficult to locate. By definition respondents who are located by snowball sample will be connected to one another and so likely to be more similar to one another than other members of the population.

Types of Non-probability Sampling: 4. Quota sampling Begin with a matrix of the population (i.e. that it’s 50% female, 9% minority, with a particular age structure). Data is collected from people with the characteristics of a given cell. Each group is assigned a weight appropriate to their portion of the population. (so if you were going to sample 1,000 people you would want 500 of them to be female and 45 to be minority women). Data should provide a representation of the total population. However the data may not represent the population in terms of criteria that were not factored in to the initial matrix. You cannot measure response rates And the selection may be biased.

The Logic of Probability Sampling Representativeness: A sample is representative of the population from which it’s selected if it has the same aggregate characteristics (i.e. same percentage of women, of immigrants, of poor and rich…) EPSM (Equal Probability of Selection Method): Every member of the population has the same chance of being selected for the sample.

Random Selection: Each element has an equal chance of selection independent of any other event in the selection process. Tables of random numbers are often used (these come in print form or can be generated by computer). Sampling Frame: List of every element/case from which a probability sample is selected. Sampling frames may not include every element. It is the researcher’s job to asses the extent of omissions and to correct them if possible.

A Population of 100

Types of Probability Sampling: 1.Simple Random Sample Feasible only with the simplest sampling frame. Enumerate sampling frame, and randomly select people. Despite being the ‘pure’ type of random sampling this actually rarely occurs.

A Simple Random Sample

Types of Probability Sampling: 2. Systematic Random Sample Random start and then every kth element selected (i.e. if you wanted to select 1,000 people out of 10,000 you’d select every 10 th person: i.e. the 3 rd, 13 th, 23 rd …). Arrangement of elements in the list can result in a biased sample (i.e. example of picking corner apartments only).

Types of Probability Sampling: 3. Stratified Sampling Rather than selecting sample from population at large, researcher draws from homogenous subsets of the population (i.e. random sampling from a set of undergraduates, and from a set of postgraduates). Ensures that key sub-populations are included in the sample. Results in a greater degree of representativeness by decreasing the probable sampling error.

A Stratified, Systematic Sample with a Random Start

Types of Probability Sampling: 4. Multistage Cluster Sampling Used when it's not possible or practical to create a list of all the elements that compose the target population. Involves repetition of two basic steps: creating lists of clusters and sampling. Highly efficient but less accurate.

Example of Cluster Sampling Sampling Coventry residents 1.Write a list of all neighbourhoods in Coventry 2.Randomly select (sample) 5 neighbourhoods 3.Write a list of all streets in each selected neighbourhood 4.Randomly select (sample) 2 streets in each neighbourhood 5.Write a list of all addresses on each selected street 6.Randomly select (sample) every house/flat. 7.Write a list of all residents in each selected house/flat 8.Randomly select (sample) one person to interview.

Types of Probability Sampling: 5. Probability Proportionate to Size (PPS) Sample Sophisticated form of cluster sampling. Used in many large scale survey sampling projects. Like cluster-sampling, but here clusters are selected with a probability proportionate to their size (i.e. a city 10 times larger than another is 10 times more likely to be selected in the first stage of clustering).

Note The sampling strategy used in real projects often combines elements of cluster sampling and elements of stratification. See example of Peter Townsend’s survey of poverty (p. 120 Buckingham and Saunders)

Group Exercise Imagine that you are going to conduct a ‘smoking survey’, and want to get as accurate as possible a sample of Warwick students. What sampling strategy would you choose and why? What biases might this strategy produce?

Weighting Used when you have “over-sampled” a particular group. This is called “disproportionate sampling” It assigns some cases more weight than others on the basis of the different probabilities each case had of selection The simplest form of weighting is to give each case a weight that’s the inverse of the case’s probability of selection

Weighting Example I have a population of 10,000 university students that is 10% ethnic minority. I want to sample 100 people and compare white and minority respondents. If I sample randomly I will probably get only about 10 minority respondents. This won’t give me much to go on to make a comparison. So I stratify my sample and sample 50/1000 minority students. A rate of selection of.05 And 50/9,000 white students. A rate of selection of.0056 We now have 50 white and 50 minority respondents – this is useful because it will capture internal heterogeneity within each sub- population. However, it now looks like the population is 50% minority, which is wrong. To re-weight the responses to make them represent the ‘real’ population I can multiply each minority respondent by the inverse of their chance of selection (1000/50 = 20) and each white respondent by the inverse of their chance of selection (9000/50 = 180).

A Parameter is the summary description of a given variable in a population (i.e. percent of women in the US population) When researchers generalize from a sample they’re using sample observations to estimate population parameters Sampling Error is the degree of error to be expected from a given sample design in making these estimations Sampling Error

The most carefully selected sample will never provide a perfect representation of the population from which it was selected. There will always be some sampling error The expected error in a sample is expressed in terms of confidence levels (i.e. that you’re 95% confident of being right about the proportion of the population that is Catholic, based on how many people in your sample were Catholic)

A population of ten people with $0 - $9

The Sampling Distribution of Samples of 1

The Sampling Distribution of Samples of 2

The sampling Distribution of Samples of 3,4,5, and 6

Sample Size (reducing sampling error) Sample Size Depends on: Heterogeneity of the population – the more heterogeneous, the bigger the sample Number of sub-groups – the more sub groups, the bigger the sample Size of the phenomenon you’re trying to detect – the closer to 50% (of the time) that it occurs, the bigger the sample How accurately you want your sample statistics to reflect the population – the more accurate, the bigger the sample

Other considerations when you’re thinking about Sample Size Response Rate – when you think that a lot of people will not respond, you need to start off with a larger sample Analysis – some forms of statistical analysis require a large number of cases. If you plan on doing these you will need to ensure you’ve got enough cases Generally (given a choice): Bigger is Better!