Measuring change in sample survey data

Underlying Concept A sample statistic is our best estimate of a population parameter If we took 100 different samples from the same population to measure, for example, the mean height of men, we would get a 100 different estimates of mean height. The mean of these means would be very close to the real population mean.

Population of men (Population Size = N)

Sample of men from population (Sample size = n) We take a sample from the population and measure the heights of all our sample members. The mean height from this sample is cm.

We take a another sample from the same population and measure the heights of all sample members. The mean height from this sample is cm. Sample of men from population (Sample size = n)

Sample of sample means We take a another 100 samples from the same population and measure the heights of all sample members. Sample 1 mean was cm Sample 2 mean was cm Sample 3 mean was cm Sample 4 mean was cm Sample 5 mean was cm Etc

We don’t ever take hundreds of samples. We just take 1. The concept of the mean of sample means is central to all survey statistics. The central limit theorem says that if we took a sufficiently large number of samples, the mean of the sample means would be normally distributed. This is true even if the thing we are measuring is not normally distributed. The central limit theorem can be proved mathematically. It is the basis of how we calculate our required sample size and how we calculate confidence intervals around our estimates………………. Sample of sample means

Variance, standard deviation and standard error Variance = the sum of squared differences from the mean divided by n-1 Variance = 30 / 4 = 7.5 Standard deviation = the square root of the variance SD = √ variance = √7.5 = 2.74 Standard error = the square root of the variance divided by the sample size SE = √ (variance / n) = √ (7.5 / 5) = 1.22 Sample values (n=5) Difference from mean Squared difference from the mean = x -1 = Mean = 171 Sum = 0Sum of squares = 30

Standard Error The standard error is our best estimate of the standard deviation of the sample means. In other words if we took 100 samples from the same population and got 100 estimates of men’s mean height, the standard deviation of that mean is the standard error.

Confidence Intervals Because the means of sample means are normally distributed, we can use the characteristics of the normal distribution to look at our mean and standard error. We know that in a normal distribution 68.3% of values fall within one standard deviation of the mean and 95% fall within 1.96 standard deviations of the mean. So 1.96 times the standard error gives us the 95% confidence limits. Our standard error is x 1.22 = 2.4 Our sample mean is – 2.4 = = So.. If we took 100 samples, 95 of them would have a mean somewhere between and Or… we can be 95% confident that the true mean (the population mean) lies between and

It works the same for proportions The 95% confidence interval around a proportion is 1.96 times the standard error of the estimate. The standard error of a proportion is √ ( (p (100-p)) / n ) Where p is the percentage and n is the sample size. So if we estimate that 75% of people prefer dogs from a sample of 45, p=75 and n= 45. = √ (( 75 x (100-75)) / 45 ) = √ ( (75 x 25) / 45 ) = √ ( 1875 / 45 ) = √ 42.7 = – 6.5 = 68.5 and = 81.5 So.. if we took 100 samples, 95 of them would have a percentage somewhere between 68.5 and Or… we can be 95% confident that the true percentage of people who prefer dogs (the population percentage) lies between 68.5 and 81.5.

Design effects If the sample is not a simple random sample then an adjustment will need to be made to the standard error Proportionate stratification will decrease the standard error Disproportionate stratification will increase the standard error Clustering will increase the standard error See PEAS website for information about design effects

Finite Population Correction If the sample size is a large proportion of the population size (>5%) then applying the finite population correction will reduce the standard error

Showing confidence intervals graphically

Measuring change over time As a rule of thumb, if two confidence intervals do not overlap we can be confident that there has been a change in the population This requires that broadly similar sample methodology was used, and exactly the same survey questions If different methodologies are used or the question changes, it becomes very difficult to say whether change in the population has occurred

Setting a target 1.Calculate the confidence interval around the baseline estimate 2.Estimate what the confidence interval will be around the target figure 3.Make sure they don’t overlap

Word of caution Don’t mistake “statistically significant” for “meaningful” A change of 0.01% can be statistically significant if the survey is large and precise enough, but most people wouldn’t call that meaningful A meaningful change in the population could be missed if the survey isn’t designed to be precise enough: Make sure the survey is designed with the purpose of monitoring change in mind And don’t forget all the non-statistical issues!!!