1. Selection and bias - two hostile brothers
P.Bauer
Medical University of Vienna
P.BAUER, F.KÖNIG, W.BRANNATH and M.POSCH, Stat Med, 2009

2. The problem Comparison of k treatments with a single control
Independent normal distributions, equal known variance σ2, means μ1,…,μk and μ0, respectively
The same sample size n is planned for each of the k treatments and the control
After a fraction of rn, 0 ≤ r ≤ 1, observations in each of the treatment and the control group an interim analysis is planned (first stage)
In the interim analysis “good” treatments (and the control) are selected and investigated at the second stage
Quantify mean bias and mean square error (MSE) of the conventional ML estimates of the mean treatment to control differences for different selection rules
16/11/2018

3. The problem of estimation
Consider the conventional fixed sample size design with k treatments and a control (no interim analysis, r=1)
It is correct that the final mean treatment control differences are unbiased estimates of the unknown effects
However, it would be hiding ones head in the sand to ignore that the magnitude of the effects plays an important role in decisions and actions following such a trial
E.g., the plausible strategy to go on with the most effective (and sufficiently safe) dose generally will tend to produce positively biased estimates of the true effect size of this dose when planning the next steps of drug development
16/11/2018

4. The problem of estimation (cont.)
Bias and selection seem to be tied together in all areas where decisions have to be taken
A straightforward example is the two pivotal trial paradigm in drug registration: Getting approval for a new treatment only if the effect estimates have crossed certain positive thresholds in two independent clinical trials will per se be linked to a positive bias of the effect estimate communicated at registration
On the other hand also getting approval only if the treatment has been shown to be sufficiently safe in the two trials will generally produce a rather optimistic estimate of safety aspects of the new treatment
16/11/2018

5. Notation
16/11/2018

6. Selecting the s best treatments (s prefixed) Selection bias of the ordered final effect estimates
DAHIYA, JASA, 1974; POSCH et al., Stat Med, 2005
This holds because is an unbiased estimate of μ0
16/11/2018

7. Selecting the s best treatments (s prefixed) Mean square error
The selection mean square error
can be defined accordingly, however, the variability arising from the mean of the control group has to be accounted for
16/11/2018

8. Only selecting the single best treatment s=1
Conjecture
The selection bias is maximal if all the treatment means are equal (μ1=μ2= … =μk)
Proof for k=2 :
PUTTER & RUBINSTEIN, Technical Report TR 165, Statistics Department, University of Wisconsin, 1968.
STALLARD, TODD & WHITEHEAD, JSPI, 2008.
For k=3: Numerical solution in
BAUER et al., Stat Med, 2009
16/11/2018

9. Only selecting the single best treatment s=1
Under the „worst case scenario“ of equal treatment means closed formula for mean bias and MSE can be derived:
16/11/2018

10. Maximum mean selection bias and √MSE (both in units of σ√(2/n)) as a function of k and r
16/11/2018

11. To take home
When randomly selecting a treatment (r=0) there is no bias
The (maximum) bias increases with increasing number of treatments k, for k tending to infinity it tends to infinity too
It sharply increases with r and is largest for r=1 („post trial selection“)
However, for differing treatment means when selecting early the probability of wrong selections will increase due to the large standard errors of the interim effect estimates!
If there is one treatment considerably better than all the others the bias will decrease because for an increasing margin it will be selected almost always. Then no bias would occur and we would get the conventional MSE
16/11/2018

12. To take home (cont.)
The corresponding √MSE does not increase with k to the same extent as the bias
In fact it is identical for k=2 and k=1 which holds true when selecting between two rival treatments with symmetrically distributed outcomes from a single location parameter family and interim selection is only depending on the difference of the interim effect estimates
POSCH et al., Stat Med, 2005
In units of the conventional standard error at the end √MSE increases linearly with the “selection time” r
16/11/2018

13. Selecting the two best treatments s=2
16/11/2018

14. To take home for s=2
In case of equal effects when selecting two treatments very early (r→0) the effect estimate of the treatment best at the end will be positively biased and accordingly that of the inferior treatment will be negatively biased
For r→1 the results for the best treatment will tend to those found if only a single treatment is selected (s=1)
The worst case scenario for a positive bias of the treatment second best at the end is that one treatment is much better than all the others. For an increasing margin bias and MSE of the second best then tend to the bias and MSE when selecting a single among k-1 treatments
k=3: The estimate of the second best treatment at the end is negatively biased for r<1 and unbiased for r=1
16/11/2018

15. The effect of reshuffling the planned sample size to the selected treatments
As an extreme example we consider the prefixed scenario that after selecting a single treatment best at interim the rest of the „planned“ (1-r)kn sample size for the second stage is equally distributed between the selected treatment and the control
Essentially this a design with fixed sample sizes per treatment at both stages. The only data driven decision during the trial is to select which treatment is compared to the control at the second stage
16/11/2018

16. Reshuffling sample size
16/11/2018

17. Reshuffling sample size – to take home
Sample size reshuffling will reduce bias and MSE of the selected best treatment
If selection is early there is a lot to reshuffle so that the enlarged second stage sample size will cause substantial regression to the mean of the final effect estimate
(This relates to the result in conventional two armed trials that enlarging the sample size in an interim analysis only when a large effect has been observed does not inflate the type I error rate
POSCH, BAUER & BRANNATH, Stat Med, 2003;
CHEN, DeMETS & LAN, Stat Med, 2004 )
16/11/2018

18. A complex selection rule
Stop for futility if all interim effect estimates are negative
Select one or two treatments with positive effect estimates:
Select a single best if
If with probability 1/2 either go on with
the best or with the two best
If with probability 1/3 either select the single
best, the best two or only the second best (safety concerns?)
The bias is calculated conditionally on no early stopping
16/11/2018

19. A complex selection rule (cont.)
The sample size n is chosen such that an (unadjusted) treatment control comparison by a one-sided fixed sample size test at the level has a power of 90% for a true mean treatment-control difference of σ
Two parameter constellations are investigated:
1. Equal treatment means
2. Linearly increasing treatment means
16/11/2018

20. A complex selection rule – the bias
16/11/2018

21. Complex selection – complex bias
Equal treatment means:
For very early selection the bias is negligible because due to the large variability we nearly always go on with a single treatment where the unbiased estimate from the second stage sample dominates the overall estimate
As compared to the simple selection rule with s=2 the bias of the estimate of the best treatment increases more sharply with r and for r→1 it is larger because of the stopping for futility option
If two treatments are selected the second best has to be close to the best leading to a larger bias of the final estimate of the second best than in simple selection (s=2)
16/11/2018

22. Complex selection– complex bias (cont.)
Linearly increasing means
If the effects are different in general the bias is reduced
k=2: The bias of the treatment best at end is non-monotonic in r. For r→1 the bias of the finally second best treatment slightly exceeds the bias of the finally best treatment because we go on with two treatments only if the second best effect is rather close to the best (and because the stopping for futility option is more effective for the treatment with the true lower effect).
There are regions of r where the bias changes the sign as compared to the scenario with equal effects
16/11/2018

23. Reporting bias of all observed effect estimates
Each observed effect estimate is reported separately regardless if the treatment has been selected or not
In this case we report the effect estimate in the total sample if the treatment has been selected and the interim effect estimate if it has not been selected
The reporting MSE can be defined accordingly!
16/11/2018

24. Reporting bias, MSE s=1
For s=1 and equal treatment means explicit formula can be derived for the reporting bias and MSE (for the latter is has been a little harder)
16/11/2018

25. Reporting bias s=1
16/11/2018

26. Reporting bias (s=1) - to take home
For equal treatment means the reporting bias is generally negative: On the one hand if the interim effect is large we tend to dilute the treatment effect by the independent second sample. On the other hand if the interim effect is small we tend to stay with the small effect as it is
It is most accentuated and equal for k=2 and k=3
As k increases the probability to be selected decreases. For any j we more often will use the hardly biased first stage estimate, the reporting bias coming closer to zero
For r→1 (no selection) the reporting bias tends to zero
For r→0 a treatment is selected with a highly variable effect estimate whose distribution is shifted to the left (the reporting bias diverges to minus infinity)
16/11/2018

27. Admission bias caused by the two pivotal trials paradigm
Two identical independent trials comparing a new treatment to a control
Each of the preplanned one sided z-tests for the primary outcome variable at the level has a power of 90% at an effect size of Δ/σ=1
Estimates are only reported (or relevant for the public in case of registration of a new drug) if both one sided z-tests have been rejected!
This will result in a “bias at admission”
See earlier work on bias in meta-analyses:
HEDGES, J.Educat.Stat., 1984; HEDGES & OLKIN, 1985; BEGG & BERLIN, J.R.S.S.A, 1988
16/11/2018

28. Admission bias (one or two pivotal trials) as a function of the true effect size δ/σ
Here the probability for registration
is small (0.025x0.025= )!
16/11/2018

29. Admission bias – to take home
The mean bias is largest for δ=0. However, there rejection only occurs with a probability of for a single trial and with a probability of for two independent trials
If the true effect is close to the effect size used for planning the sample size the bias is small and for larger effect sizes approaches 0 quite fast
The mean bias is the same for rejection in a single planned study or for the simultaneous rejection in two planned independent trials
The MSE is lower in the two as compared to the single study scenario. If in the single study scenario the true effect size is slightly below the targeted one the MSE is slightly below the mean square error of the unconditional mean from a fixed sample size trial (truncated distribution!)
16/11/2018

30. Selection and bias - concluding remarks
Without fixed selection rules the bias can not calculated
There are different ways to quantify the bias
There are many design features influencing bias
The better (later) the selection – the worse the bias
The earlier the selection the higher the risk of a wrong selection
(Reasonably) reshuffling sample size decreases the bias
With increasing treatment differences the bias decreases
Do we end with series of two armed fixed sample size trials? But then how to deal with such series?
The bad news: the problem of admission bias becomes serious if we deal with underpowered studies
The good news: if the studies are well powered for the true effect size the admission bias becomes negligible
16/11/2018

31. Note that the phenomena of bias has to be
considered as an intrinsic feature of human life
when selecting, e.g., jobs, friends and partners
based on a comparison of past observations
afflicted by random variation
Thank you for your patience!
16/11/2018