1. Improving estimates of confidence intervals around smoking quit rates
Marie Horton, Paul Fryers, Clare Griffiths
Public Health England

2. Overview The Local Tobacco Control Profiles
Indicators calculation for smokers’ quit attempts
The original confidence interval method
Why change?
The updated confidence interval method
The impact of the change
Improving estimates of confidence intervals around smoking quit rates

3. Local Tobacco Control Profiles
Improving estimates of confidence intervals around smoking quit rates

4. Local Tobacco Control Profiles
provides a snapshot of the extent of tobacco use, tobacco related harm, and measures being taken to reduce this harm at a local level.
designed to help local government and health services to assess the effect of tobacco use on their local populations. 
inform commissioning and planning decisions to tackle tobacco use and improve the health of local communities.
allows you to compare your local authority against other local authorities in the region and benchmark your local authority against the England or regional average.
data split into six domains
smoking prevalence in adults smoking related ill health
smoking prevalence in young people impact of smoking
smoking related mortality smoking quitters
Improving estimates of confidence intervals around smoking quit rates

5. Smoking Quitters Domain
Improving estimates of confidence intervals around smoking quit rates

6. Indicator calculation
Latest data:
Indicator calculation
Data from NHS Digital Stop Smoking Services Annual Report
100,000
Smoking population
Three measures:
No. setting a quit date
No. of successful quitters at 4 weeks
No. of successful quitters at 4 weeks (CO validated)
Smoking prevalence (Annual Population Survey)
ONS mid-year population estimates
Improving estimates of confidence intervals around smoking quit rates

7. Complexities of these indicators
Data are collected on the basis of the LA providing the stop smoking service. Clients are not restricted to attending stop smoking services in the LA with which they are registered or resident.
For those LAs whose residents go elsewhere for stop smoking services this will be an underestimate of the true rate, for those LAs who see many people for stop smoking services from outside the LA, this rate will be an overestimate of the true rate.
Smoking prevalence as measured by the APS is self-reported and as such, prone to response bias.
The prevalence data is not age standardised and its methodology is still classified as experimental.
Numerator is for age 16+ while denominator is for age 18+
Improving estimates of confidence intervals around smoking quit rates

8. Confidence interval methods
Previously….
Byar’s method:
gives very accurate approximate confidence intervals for counts based on the assumption of a Poisson distribution
is sufficiently accurate for counts as low as 5 (below 5, an exact method should be used, based on Poisson tables or the Chi-squared distribution)
is the preferred method for calculating confidence intervals for counts, crude rates, indirectly standardised rates and indirectly standardised ratios
However….
It cannot take into account the uncertainty surrounding the denominator which is based on the smoking prevalence which has associated confidence intervals
Improving estimates of confidence intervals around smoking quit rates

9. New method (1) Before the simulation:
Using Excel and VBA we used the Wilson Score confidence interval method to calculate the standard errors for the logit-transformed smoking prevalence estimates from the Annual Population Survey (APS).
Before the simulation:
The prevalence is logit-transformed logit (prev) = ln (prev / (1 – prev))
The standard deviation (SD) of the logit-transformed prevalence is calculated. This is done by first calculating the Wilson Score confidence intervals for the prevalence, using n as the denominator and (prev × n / 100) as the numerator
These confidence limits are then logit-transformed, and the difference between the logit-transformed upper and lower limits is divided by (2 × z1–α/2) to give an estimate of the SD of the logit-transformed prevalence
Improving estimates of confidence intervals around smoking quit rates

10. New method (2) The simulation: for each iteration
4. A random number between 0 and 1 is generated using the Mersenne Twister algorithm, and used to provide a random value from a normal distribution with mean = logit (prev) and SD = the SD of logit (prev)
5. This random sampled value is transformed back to a prevalence using the inverse of the logit function, and multiplied by pop to give a randomised estimated number of smokers in the LA (the denominator of the randomised indicator)
6. A second random number is generated and used to provide a random value from a normal distribution with mean = num (eg the number of smokers who have set a quite date in the LA) and SD = sqrt (num) – this is the randomised numerator value
Improving estimates of confidence intervals around smoking quit rates

11. New method (3) After the simulation:
7. The randomised indicator value is calculated by dividing the randomised numerator by the randomised denominator
8. The randomised indicator value is added to an array of randomised indicator values
After the simulation:
9. From the array of randomised indicator values, the (1–α/2)th centile smallest and largest values are identified and returned as the lower and upper confidence limits respectively (where α is the confidence level specified for the confidence intervals, eg 0.95)
Improving estimates of confidence intervals around smoking quit rates

12. Effects of the change Confidence intervals widened 2015/16
Improving estimates of confidence intervals around smoking quit rates

13. How did we communicate this?
Confidence intervals were removed temporarily when the problem was identified
To make users aware that we were taking action
To prepare them for the change
To stop them basing decisions on inaccurate estimates
Rates were displayed by quintile rather than RAG rating during the transition period
The calculation was carried out for the entire time series
We clearly displayed on the tool further details of the changes
We explained in a webinar
Improving estimates of confidence intervals around smoking quit rates

14. Conclusion It is important to …..
take into account the complexities within indicator calculation when deciding on a confidence interval method in order to get the most accurate estimates possible
Survey data
Confidence intervals surrounding numerator/denominator
Data collection/sampling
ensure that users are aware of confidence interval methods and why they have been chosen
ensure that users are aware of upcoming changes and the reasons for them
communicate with users about how confidence intervals should be used to interpret the data
Improving estimates of confidence intervals around smoking quit rates

15. https://fingertips.phe.org.uk/profile/tobacco-control
Any questions?
Improving estimates of confidence intervals around smoking quit rates