1. Economic evaluations alongside Stepped Wedge Trials: Systematic review and proposal of analysis
Gian Luca Di Tanna
Vladislav Berdunov
Richard Hooper
Pragmatic Clinical Trials Unit, Queen Mary University of London
4th International Clinical Trials Methodology Conference, Liverpool, 09/05/2017

2. The Stepped Wedge Design
SWT: each cluster receive the intervention at different time points
Clusters (e.g. general practices, hospitals, study sites) randomised according to sequence of crossover
All the clusters start in control and exposed to intervention in stepwise fashion
t t t t t4
Group of Clusters 1 2 3 4
Intervention
Control
Rentsch/Record Linkage Tanzania

3. Economic evaluation alongside SWT
Intra-cluster correlation: Individuals within a cluster may be homogeneous
Correlation between costs and outcomes at individual level
Time effect: Timing of crossover may impact on both costs and outcomes – changes in practice, service quality and cost over time

4. Step 1: literature review
Search strategy of Medline/PubMed, DARE, NHS-EED, HTA and the Cochrane Library
Articles indexed up to July 2016
Abstracts were independently screened by two investigators using the following inclusion criteria:
use of the SWT design
economic evaluation as part of the research question(s) in the study.

5. Study flow Screening Eligibility Inclusion
75 Studies identified in Medline
19 Reviews identified in the Cochrane library
6 studies identified in DARE, NHS-EED and HTA
Identification
100 records identified
Screening
58 records were not SWTs
42 full-texts identified (31 primary studies, 11 systematic reviews/HTA)
Eligibility
24 full-texts excluded:
2 reviews did not include SWTs
6 SWTs included in the reviews were already included
16 were not SWT designs (3 of which have been called SWT by the authors but were not SWT)
18 articles included in the scoping review
Inclusion
15 Protocols
2 Trial results
1 methods paper

6. Characteristics of the studies included
Year
2011
2012
2013
2014
2015
2016/
No. of studies 2 3 1 4 7
Country
No. of Studies
The Netherlands 9
Australia 4
UK, Canada, Norway, Germany, Peru 1
Method
No. of Studies
Generic/bias-corrected boostrap 9
Mixed models with clustering 4
Extrapolations/generic PSA 2
Not reported 3
Economic Evaluation
No. of Studies
CUA 8
CEA 6
CEA and CUA 1
CBA

7. Step 2: simulation study
Built-in correlation between costs and outcomes across individuals in a cluster and time effect on costs and outcomes
Range of circumstances: number of participants per cluster, number of clusters per group, ICCs and time effect
Costs and outcomes generated at cluster level and then at individual level using Monte Carlo simulation
Comparing three alternatives:
Multi-level model (MLM)
Seemingly unrelated regression (SUR)
Two-stage boostrap

8. Multilevel model (MLM)
Let citk and eitk represent the costs and the outcomes for the ith individual (with i=1,…,m same number of individuals in each cluster)
time t (t=0,1,2, …T after randomization)
cluster k (k=1,…,K)
Xtk=1 in the intervention group and Xtk=0 in the control group.

9. Seemingly unrelated regression (SUR)
Similar in to MLM in terms of design
No random cluster effect
Robust standard errors to control for clustering
Individual error term follows bivariate normal distribution

10. Two-stage bootstrap (TSB)
Non-parametric bootstrapping commonly used in economic evaluations with non-normally distributed endpoints
Extended to two stages to account for clustered data and time effects using bivariate resampling
Resampling clusters then individuals within the cluster
Shrinkage correction to avoid overestimation of variance

11. Assessing performance
Each model assessed net monetary benefit assuming £30,000 per QALY
Performance was assessed using a selection of metrics:
Mean bias (SE) and root mean squared error (rMSE)
Mean CI width and 95% CI coverage
A number of scenarios
No. of clusters per group (5-25)
Cluster size (5-30 participants)
No. of steps (2-4)
ICC for costs and outcomes ( )
Cost time effect (£0-500)
Outcome time effect ( QALYs)

12. Base case scenario
Fixed parameters: Incremental cost £500 (SD 1000), incr. outcome 0.05 QALYs (SD 0.1) Equal cluster size Cost time effect £100, outcome time effect 0.01 QALYs
Base case variable values:
10 clusters x 20 participants
2 steps
ICC for costs & outcomes 0.01
Cost time effect £100, outcome time effect 0.01 QALYs
Table 1: Performance of SUR, MLM and TSB in base case scenario
SUR
MLM
TSB
Mean bias (SE)
10.00 (14.8)
10.31 (14.65)
(17.78)
rMSE
314.31
302.45
192.89
CI coverage
93.2
93.4 65
Mean CI width
1214.7
1185.3
754.29

13. CI coverage and rMSE

14. CI coverage and rMSE

15. Mean bias (SE) SUR MLM TSB Base case 10.00 (14.80) 10.31 (14.65)
Table 4: Mean bias (SE) under alternative scenarios
SUR
MLM
TSB
Base case
10.00 (14.80)
10.31 (14.65)
(17.78)
Low no. of clusters (5)
22.73 (20.43)
17.49 (20.00)
(22.29)
High no. of clusters (25)
8.29 (9.35)
5.69 (9.29)
(10.63)
Low cluster size (5)
12.43 (27.51)
8.29 (28.34)
(36.64)
High cluster size (30)
4.85 (12.08)
4.02 (12.00)
(15.00)
High no. of steps (4)
(4.87)
9.63 (6.95)
(10.19)
High ICC for costs and outcomes (0.3)
8.92 (36.85)
23.76 (33.84)
(25.23)
High cost/outcome time effect (500/0.025)
5.1 (14.63)
(17.78)
Table 5: Performance of alternative methods under different temporal trends
Base case
Cost/outcome effect 0
Cost/outcome effect 250/0.025
Cost/outcome effect 500/0.05
SUR
Bias (SE)
10.0 (14.8)
19.8 (14.8)
5.1 (14.6)
rMSE
314.3
311.3
316.1
MLM
10.3 (14.7)
9.8 (14.6)
17.6 (14.7)
302.5
301.5
TSB
289.3 (17.8)
22.7 (17.8)
665.3 (16.7)
(17.8)
192.9
191.0
204.5
235.4

16. Discussion
MLM and SUR performed well under a variety of conditions, even with low cluster size, low no. of clusters, large time effects – CI coverage >90%, low bias
TSB had a large bias but lower rMSE – appeared to be less accurate but more precise
With small no. of clusters/small cluster size, bootstrapping may be an appropriate option, given MLM and SUR produce a high rMSE
SUR & MLM outperformed bootstrapping in the presence of time effects. If no time effect can be assumed, TSB is superior (low bias, lower rMSE, higher coverage)

17. Questions? Vlad Berdunov v.berdunov@qmul.ac.uk
Gian Luca Di Tanna