Improving estimates of confidence intervals around smoking quit rates Marie Horton, Paul Fryers, Clare Griffiths Public Health England

Overview The Local Tobacco Control Profiles Indicators calculation for smokers’ quit attempts The original confidence interval method Why change? The updated confidence interval method The impact of the change Improving estimates of confidence intervals around smoking quit rates

Local Tobacco Control Profiles Improving estimates of confidence intervals around smoking quit rates

Local Tobacco Control Profiles provides a snapshot of the extent of tobacco use, tobacco related harm, and measures being taken to reduce this harm at a local level. designed to help local government and health services to assess the effect of tobacco use on their local populations.  inform commissioning and planning decisions to tackle tobacco use and improve the health of local communities. allows you to compare your local authority against other local authorities in the region and benchmark your local authority against the England or regional average. data split into six domains smoking prevalence in adults smoking related ill health smoking prevalence in young people impact of smoking smoking related mortality smoking quitters Improving estimates of confidence intervals around smoking quit rates

Smoking Quitters Domain https://digital.nhs.uk/data-and-information/publications/statistical/statistics-on-nhs-stop-smoking-services-in-england Improving estimates of confidence intervals around smoking quit rates

Indicator calculation Latest data: https://digital.nhs.uk/data-and-information/publications/statistical/statistics-on-nhs-stop-smoking-services-in-england/april-2017-to-march-2018 Indicator calculation Data from NHS Digital Stop Smoking Services Annual Report 100,000 Smoking population Three measures: No. setting a quit date No. of successful quitters at 4 weeks No. of successful quitters at 4 weeks (CO validated) Smoking prevalence (Annual Population Survey) ONS mid-year population estimates Improving estimates of confidence intervals around smoking quit rates

Complexities of these indicators Data are collected on the basis of the LA providing the stop smoking service. Clients are not restricted to attending stop smoking services in the LA with which they are registered or resident. For those LAs whose residents go elsewhere for stop smoking services this will be an underestimate of the true rate, for those LAs who see many people for stop smoking services from outside the LA, this rate will be an overestimate of the true rate. Smoking prevalence as measured by the APS is self-reported and as such, prone to response bias. The prevalence data is not age standardised and its methodology is still classified as experimental. Numerator is for age 16+ while denominator is for age 18+ Improving estimates of confidence intervals around smoking quit rates

Confidence interval methods Previously…. Byar’s method: gives very accurate approximate confidence intervals for counts based on the assumption of a Poisson distribution is sufficiently accurate for counts as low as 5 (below 5, an exact method should be used, based on Poisson tables or the Chi-squared distribution) is the preferred method for calculating confidence intervals for counts, crude rates, indirectly standardised rates and indirectly standardised ratios However…. It cannot take into account the uncertainty surrounding the denominator which is based on the smoking prevalence which has associated confidence intervals Improving estimates of confidence intervals around smoking quit rates

New method (1) Before the simulation: Using Excel and VBA we used the Wilson Score confidence interval method to calculate the standard errors for the logit-transformed smoking prevalence estimates from the Annual Population Survey (APS). Before the simulation: The prevalence is logit-transformed logit (prev) = ln (prev / (1 – prev)) The standard deviation (SD) of the logit-transformed prevalence is calculated. This is done by first calculating the Wilson Score confidence intervals for the prevalence, using n as the denominator and (prev × n / 100) as the numerator These confidence limits are then logit-transformed, and the difference between the logit-transformed upper and lower limits is divided by (2 × z1–α/2) to give an estimate of the SD of the logit-transformed prevalence Improving estimates of confidence intervals around smoking quit rates

New method (2) The simulation: for each iteration 4. A random number between 0 and 1 is generated using the Mersenne Twister algorithm, and used to provide a random value from a normal distribution with mean = logit (prev) and SD = the SD of logit (prev) 5. This random sampled value is transformed back to a prevalence using the inverse of the logit function, and multiplied by pop to give a randomised estimated number of smokers in the LA (the denominator of the randomised indicator) 6. A second random number is generated and used to provide a random value from a normal distribution with mean = num (eg the number of smokers who have set a quite date in the LA) and SD = sqrt (num) – this is the randomised numerator value Improving estimates of confidence intervals around smoking quit rates

New method (3) After the simulation: 7. The randomised indicator value is calculated by dividing the randomised numerator by the randomised denominator 8. The randomised indicator value is added to an array of randomised indicator values After the simulation: 9. From the array of randomised indicator values, the (1–α/2)th centile smallest and largest values are identified and returned as the lower and upper confidence limits respectively (where α is the confidence level specified for the confidence intervals, eg 0.95) Improving estimates of confidence intervals around smoking quit rates

Effects of the change Confidence intervals widened 2015/16 Improving estimates of confidence intervals around smoking quit rates

How did we communicate this? Confidence intervals were removed temporarily when the problem was identified To make users aware that we were taking action To prepare them for the change To stop them basing decisions on inaccurate estimates Rates were displayed by quintile rather than RAG rating during the transition period The calculation was carried out for the entire time series We clearly displayed on the tool further details of the changes We explained in a webinar Improving estimates of confidence intervals around smoking quit rates

Conclusion It is important to ….. take into account the complexities within indicator calculation when deciding on a confidence interval method in order to get the most accurate estimates possible Survey data Confidence intervals surrounding numerator/denominator Data collection/sampling ensure that users are aware of confidence interval methods and why they have been chosen ensure that users are aware of upcoming changes and the reasons for them communicate with users about how confidence intervals should be used to interpret the data   Improving estimates of confidence intervals around smoking quit rates

https://fingertips.phe.org.uk/profile/tobacco-control Any questions? Marie.Horton@phe.gov.uk https://fingertips.phe.org.uk/profile/tobacco-control Improving estimates of confidence intervals around smoking quit rates