1. Methods for assessing fidelity and quality of delivery of smoking cessation behavioural support Fabiana Lorencatto, Robert West, Carla Bruguera, & Susan Michie Centre for Outcomes Research and Effectiveness, University College London Health Behaviour Research Centre, University College London 3 National Centre for Smoking Cessation and Training

2. Why is there substantial variability in quit outcomes across services? Is it the content of support delivered? Many services have structured behavioural support treatment manuals –Specify recommended content and format of sessions –Help guide and standardise practitioner behaviour Practitioners from same NHS SSSs, operating under same manual have widely varying success rates (Brose et al. 2012) Actually following the treatment manual?

3. Two aspects of delivery need to be examined: 1.What manual specified content is actually delivered? - Fidelity of delivery - extent to which interventions are delivered as intended (i.e. recommended practice  actual practice) 2. How is this content actually delivered? - Quality of delivery -Appropriate + comprehensive delivery of intervention content *** Does this affect outcomes?

4. Aims Two linked studies Study 1: 1.Develop a reliable method for assessing fidelity of delivery of behavioural support in practice. Study 2: 1.Develop a reliable method for assessing quality of delivery of key technique, setting a quit date (i.e. goal setting) 2.Examine the association between quality of goal setting and likelihood of clients making a quit attempt.

5. Study 1 assessing fidelity of delivery ‘WHAT is delivered’

6. METHODS: Sample and materials -National telephone behavioural support service (i.e. quit line) -Six practitioners -All trained to NCSCT standard -All aware of treatment manual -Service treatment manual (i.e. recommended practice) -Outlines recommended content/format of pre-quit, quit day, and post quit sessions -Transcripts of 64 audio-recorded, consecutively delivered behavioural support sessions (i.e. actual practice). -Pre-quit sessions (n=27) -Quit-day sessions (n=16) -Post-quit sessions (n=21).

7. Procedure Two coders identified + categorised BCTs in treatment manuals and session transcripts using taxonomy of 43 smoking cessation BCTs. Inter-rater reliability assessed using % agreement Examined what % of manual-specified BCTs were present in transcripts of practice sessions (recommended vs. actual practice)

8. RESULTS Reliability: average percentage agreement was high (87.1%; range: 78.4% - 95.6%) Fidelity: Note: <50% = ‘low fidelity;’ 51-79%= ‘moderate;’ 80-100% =‘high’ SESSION TYPENUMBER OF BCTS IN MANUAL AVERAGE NUMBER (%) MANUAL BCTs DELIVERED SD/RANGE PRE-QUIT (n=27) 2210 (46%)SD 16.9 Range: 14-82% QUIT-DAY (n=16) 259 (35%)SD 14.8 Range: 8-60% POST-QUIT (n=21) 2812 (42%)SD 16.3 Range: 8-82% ACROSS SESSIONS 25 (average)10.3 (41.2%)-

9. Study 2 ‘HOW’ assessing quality of delivery of ‘setting a quit date’ (i.e. goal setting) association between quality of goal setting and quit attempts.

10. METHODS Sample & Materials: -Same quit line service and practitioners -New set of transcripts: 85 pre-quit behavioural support sessions -85 clients -Pre-quit sessions as this when quit date likely to be set -Outcomes: whether client made a quit attempt as planned by following session (yes/no)

11. PROCEDURE 1) Development of quality rating scale: -Quality= appropriate + comprehensive delivery of technique -Two researchers independently examined guidance documents to identify components of high quality ‘setting a quit date’ -NCSCT training standard, standard treatment programme, training module, quit line service treatment manual -Extracted components featured across all guidance documents  description of ‘gold standard’ goal setting -Basis for content of rating scale

12. 0 Absence of goal setting +1Prompts goal setting (i.e. encourage client set quit date) +1Agrees clear quit date (dd/mm/yy) +1 Agreed quit date within appropriate time frame (1-2 weeks) +1Quit date allows appropriate time to obtain medication +1 Emphasise goal is to not smoke a single cigarette/puff +1Provide relevant normative information and examples -1Inappropriate goal setting (i.e. unclear date, incorrect time frame) -1Undermines client commitment to quit date (i.e. imply flexibility) Points allocated for appropriate delivery Points deducted for inappropriate delivery Score range: -2 to 6 Higher score implies more comprehensive + appropriate delivery

13. Scored sections of transcripts in which ‘setting a quit date’ occurred using scale Inter-rater reliability assessed using Cohen’s Kappa (weighted) Examined association between quality scores and whether client made a quit attempt as planned –Multilevel logistic regression both using total quality score as predictor And individual scale component as predictor

14. RESULTS Inter-rater reliability: Weighted kappa = 0.68  ‘substantial agreement’ (Landis & Koch,1977) Average quality score: - overall: 1.6 (SD 1.2)  low quality (scale range: -2 to 6) - clients made attempt (n=18; 21.2%): 2.2 (SD.70) - clients did not make attempt (n=67; 78.8%): 1.4 (SD 1.27)

15. Association with quit attempts: Higher quality goal setting in pre-quit sessions significantly increased likelihood of clients making quit attempt as planned [p<.001 OR 2.60 (95% CI: 1.54 to 4.40)] Only 1 scale component independently associated significantly with quit attempts: -Setting a clear quit date (dd/mm/yy): p<.001, OR 36.9, 95% CI: 4.52 to 302.11) -Opposite of this: Setting inappropriate quit date (i.e. unclear date): p<.001, OR.03, 95% CI:.24 to 2.88)

16. DISCUSSION It is possible to reliably assess fidelity and quality of delivery of smoking cessation behavioural support in practice Preliminary analysis of national telephone quit line service: –Less than half of what is specified in treatment manuals is not being routinely delivered in practice –Quality of ‘setting a quit date’ is low, but higher quality significantly increases likelihood of initiating quit attempts –Particularly if a clear date is set. –Deconstruction of technique into individual components: identify specific aspect driving outcome (i.e. ‘active ingredient’)

17. Limitations: –Limited generalisability- single service examined; other modalities –Service manual extremely comprehensive (22-29 BCTs per session!)  feasible? Relevant? Implications & Future work: –Identification of training needs + aspects of service delivery to be targeted for improvement –Expand quality scale to other behaviour change techniques –Need to examine how practitioners actually use manuals in practice –And whether manuals are fit for purpose Conclusions: Stop smoking services should establish routine procedures for monitoring fidelity and quality of behavioural support delivered.

18. Thank you for listening. Fabi.lorencatto@ncsct.co.uk The team: Fabiana Lorencatto, Robert West, Carla Bruguera, & Susan Michie Acknowledgements: Collaborating services Funding: National Centre for Smoking Cessation and Training; Department of Health