How to Design a Sample and Improve Response Rates Alex StannardScottish Government Kevin RalstonUniversity of Stirling

Plan Types of sample Sampling frames Improving response rates Group exercise

Census or sample? Edinburgh City Council wants to know peoples thoughts on its leisure facilities. A decision is made to send a questionnaire to every household in Edinburgh City. A 5% response rate is expected. This is a census of all households in Edinburgh City

What is Sampling? Identify population Select members of population to sample Study selected members (the sample) Draw inferences about population from sample

Types of samples Non-Probability Samples Not all members of the population have a chance to be included in the sample. Selection method not random. Probability Samples Every member of the population has a known, nonzero chance of being included in the sample. The selection method is random.

Probability Sampling Example – 1000 households randomly selected across Argyll and Bute. More expensive and slower Non response a problem – but resources can be targeted and extent of non response bias can be estimated. Enables precision of final statistics to be assessed. Sample selection method is objective, specified and replicable.

Simple random sampling Sampling method completely random based on random numbers. Is easy to understand, but can be expensive Example – every household in Aberdeen City assigned a number. 1,000 random numbers chosen between 1 and 204,683. These numbers identify which households are in sample. Tables of random numbers Excel function ‘=rand()’

Systematic Sampling Uses a ‘random’ start on the sampling frame and then selects every i’th unit/person. Easy to understand, quick and easy to implement. Can lead to some stratification depending on how the list is ordered. Can be expensive Need to be careful on how list is ordered to avoid bias.

Systematic Sampling Example Total units on sampling frame = 60 Want sample of 10 Interval size is 60/10 = 6 Select random start between 1 and 6 Select every sixth unit

Stratified Sampling Units/people are aggregated into subgroups called strata. A certain number of units are sampled from each stratum. Guards against unusual samples Stratification information has to be available United Kingdom (60m) England (51m)Northern Ireland (2m)Scotland (5m)Wales (3m)

Stratified Sampling Proportionate Stratification Chance of inclusion in sample is same for all units/people regardless of strata Population United Kingdom England Northern Ireland Scotland Wales Sample Size Sampling Fraction 0.01%

Stratified Sampling Disproportionate Stratification Chance of a unit/person being included in the sample depends on the strata they are in. Often used to target small sub groups to help analysis Population United Kingdom England Northern Ireland Scotland Wales Sample Size Sampling Fraction 0.01% 0.05% 0.02% 0.03%

Sampling Frames List of all units/people that could be included in sample Sample is only as good as the sampling frame oEligible units/people not on frame cannot be selected – leads to coverage error oUnits/people on frame more than once changes probability of being selected oIneligible units/people on frame can lead to final sample being smaller than intended.

Intended and achieved populations Intended Population Coverage Bias Sampling Frame Sampling Variance Selected Sample Non Response Achieved Sample

Improving Response Rates

Incentives Advanced Letters & Reminders Respondent Burden Call Back

Incentives Incentives – monetary or gift, e.g. pen, gift token? Incentives generally improve response rates. For postal surveys: If the budget is an issue then follow up mailings are preferred, if time is an issue incentives are preferred (Larson and Chow 2003). –Robertson et al. (2005) found a Lottery scratch card incentive increased postal survey response by 9.6%. –Prize draws may be ineffective - the incentive is abstract and distant from the participant (McCarty 2006).

Advanced Letters and Reminders Cost Effective – they work Sponsor – who is the survey for? People are more likely to provide information to the government (or local authority) than to a private company. Salience – emphasise the importance a topic may have to a respondent. …but... Overemphasising importance or sensitivity may put some people off (Groves 2006). Strike a balance.

Respondent Burden Burden –The right questions and only those that need to be asked. McCarty et al. (2006) found that a 10 minute increase in survey length results in a 7% decrease in response rates for telephone surveys.

Call Back Repeated call back improves response particularly used in face to face interviews. –Minimum standard: rules about timing and number of call attempts before classification as non-contact. Training techniques to avoid refusal. –Extended measures: passing on to a senior interviewer; phone call and appointment, letter and appointment; letter from study director. (Lynn and Clarke 2002) In postal surveys we can re-issue the questionnaire

General Recommendations Include incentives (if the budget is an issue follow ups are preferred) Issue advanced letters Keep respondent burden as low as possible Have a clear call back procedure Make it salient

Group Exercise