Understanding sample survey data

Underlying Concept A sample statistic is our best estimate of a population parameter If we took 100 different samples from the same population to measure, for example, the mean height of men, we would get a 100 different estimates of mean height. The mean of these means would be very close to the real population mean.

Population of men (Population Size = N)

Sample of men from population (Sample size = n) We take a sample from the population and measure the heights of all our sample members. The mean height from this sample is cm.

We take a another sample from the same population and measure the heights of all sample members. The mean height from this sample is cm. Sample of men from population (Sample size = n)

Sample of sample means We take a another 100 samples from the same population and measure the heights of all sample members. Sample 1 mean was cm Sample 2 mean was cm Sample 3 mean was cm Sample 4 mean was cm Sample 5 mean was cm Etc

We don’t ever take hundreds of samples. We just take 1. The concept of the mean of sample means is central to all survey statistics. The central limit theorem says that if we took a sufficiently large number of samples, the mean of the sample means would be normally distributed around the true population mean. This is true even if the thing we are measuring is not normally distributed. The central limit theorem can be proved mathematically. It is the basis of how we calculate our required sample size, how we check our surveys are giving us sufficiently robust estimates, how we set targets, how we assess whether there has been any change between time………………. Sample of sample means

Confidence Intervals Central Limit Theorem allows us to calculate values around our estimates that tell us how close to the population parameter our sample statistic is likely to be

Confidence Intervals For example, Our survey suggests that 18% of people are poor. The confidence interval is 2 percentage points either side. So.. If we took 100 samples, 95 of them would have a proportion somewhere between 16 and 20% Or… we can be 95% confident that the true proportion of poor people in the population is somewhere between 16 and 20%

Measuring change over time As a rule of thumb, if two confidence intervals do not overlap we can be confident that there has been a change in the population This requires that broadly similar sample methodology was used, and exactly the same survey questions If different methodologies are used or the question changes, it becomes very difficult to say whether change in the population has occurred

Showing confidence intervals graphically

Setting a target 1.Calculate the confidence interval around the baseline estimate 2.Estimate what the confidence interval will be around the target figure 3.Make sure they don’t overlap

For example There is a local initiative to increase community cohesion by encouraging neighbours to get to know each other A 2009 survey reveals 30% (+ or – 3 percentage points) of people regularly speak to their neighbours By 2012 we want that to have increased What is the lowest target we can reliably measure? a)33% b)34% c)37%

Answer = 37% In 2009 we can be 95% sure that between 27 and 33% of people regularly speak to their neighbours In 2012 we would need our estimate to be 37% in order for us to be 95% sure that more people are speaking to their neighbours than in 2009

Word of caution Don’t mistake “statistically significant” for “meaningful” A change of 0.01% can be statistically significant if the survey is large and precise enough, but most people wouldn’t call that meaningful A meaningful change in the population could be missed if the survey isn’t designed to be precise enough: Make sure the survey is designed with the purpose of monitoring change in mind

Further Information Statistics Without Tears by Derek Rowntree (Penguin, 1981) Scottish Government Statistics Glossary Forthcoming Analytical Guidance Library on the ScotStat Network website