Selection and bias - two hostile brothers P.Bauer Medical University of Vienna P.BAUER, F.KÖNIG, W.BRANNATH and M.POSCH, Stat Med, 2009

The problem Comparison of k treatments with a single control Independent normal distributions, equal known variance σ2, means μ1,…,μk and μ0, respectively The same sample size n is planned for each of the k treatments and the control After a fraction of rn, 0 ≤ r ≤ 1, observations in each of the treatment and the control group an interim analysis is planned (first stage) In the interim analysis “good” treatments (and the control) are selected and investigated at the second stage Quantify mean bias and mean square error (MSE) of the conventional ML estimates of the mean treatment to control differences for different selection rules 16/11/2018

The problem of estimation Consider the conventional fixed sample size design with k treatments and a control (no interim analysis, r=1) It is correct that the final mean treatment control differences are unbiased estimates of the unknown effects However, it would be hiding ones head in the sand to ignore that the magnitude of the effects plays an important role in decisions and actions following such a trial E.g., the plausible strategy to go on with the most effective (and sufficiently safe) dose generally will tend to produce positively biased estimates of the true effect size of this dose when planning the next steps of drug development 16/11/2018

The problem of estimation (cont.) Bias and selection seem to be tied together in all areas where decisions have to be taken A straightforward example is the two pivotal trial paradigm in drug registration: Getting approval for a new treatment only if the effect estimates have crossed certain positive thresholds in two independent clinical trials will per se be linked to a positive bias of the effect estimate communicated at registration On the other hand also getting approval only if the treatment has been shown to be sufficiently safe in the two trials will generally produce a rather optimistic estimate of safety aspects of the new treatment 16/11/2018

Notation 16/11/2018

Selecting the s best treatments (s prefixed) Selection bias of the ordered final effect estimates DAHIYA, JASA, 1974; POSCH et al., Stat Med, 2005 This holds because is an unbiased estimate of μ0 16/11/2018

Selecting the s best treatments (s prefixed) Mean square error The selection mean square error can be defined accordingly, however, the variability arising from the mean of the control group has to be accounted for 16/11/2018

Only selecting the single best treatment s=1 Conjecture The selection bias is maximal if all the treatment means are equal (μ1=μ2= … =μk) Proof for k=2 : PUTTER & RUBINSTEIN, Technical Report TR 165, Statistics Department, University of Wisconsin, 1968. STALLARD, TODD & WHITEHEAD, JSPI, 2008. For k=3: Numerical solution in BAUER et al., Stat Med, 2009 16/11/2018

Only selecting the single best treatment s=1 Under the „worst case scenario“ of equal treatment means closed formula for mean bias and MSE can be derived: 16/11/2018

Maximum mean selection bias and √MSE (both in units of σ√(2/n)) as a function of k and r 16/11/2018

To take home When randomly selecting a treatment (r=0) there is no bias The (maximum) bias increases with increasing number of treatments k, for k tending to infinity it tends to infinity too It sharply increases with r and is largest for r=1 („post trial selection“) However, for differing treatment means when selecting early the probability of wrong selections will increase due to the large standard errors of the interim effect estimates! If there is one treatment considerably better than all the others the bias will decrease because for an increasing margin it will be selected almost always. Then no bias would occur and we would get the conventional MSE 16/11/2018

To take home (cont.) The corresponding √MSE does not increase with k to the same extent as the bias In fact it is identical for k=2 and k=1 which holds true when selecting between two rival treatments with symmetrically distributed outcomes from a single location parameter family and interim selection is only depending on the difference of the interim effect estimates POSCH et al., Stat Med, 2005 In units of the conventional standard error at the end √MSE increases linearly with the “selection time” r 16/11/2018

Selecting the two best treatments s=2 16/11/2018

To take home for s=2 In case of equal effects when selecting two treatments very early (r→0) the effect estimate of the treatment best at the end will be positively biased and accordingly that of the inferior treatment will be negatively biased For r→1 the results for the best treatment will tend to those found if only a single treatment is selected (s=1) The worst case scenario for a positive bias of the treatment second best at the end is that one treatment is much better than all the others. For an increasing margin bias and MSE of the second best then tend to the bias and MSE when selecting a single among k-1 treatments k=3: The estimate of the second best treatment at the end is negatively biased for r<1 and unbiased for r=1 16/11/2018

The effect of reshuffling the planned sample size to the selected treatments As an extreme example we consider the prefixed scenario that after selecting a single treatment best at interim the rest of the „planned“ (1-r)kn sample size for the second stage is equally distributed between the selected treatment and the control Essentially this a design with fixed sample sizes per treatment at both stages. The only data driven decision during the trial is to select which treatment is compared to the control at the second stage 16/11/2018

Reshuffling sample size 16/11/2018

Reshuffling sample size – to take home Sample size reshuffling will reduce bias and MSE of the selected best treatment If selection is early there is a lot to reshuffle so that the enlarged second stage sample size will cause substantial regression to the mean of the final effect estimate (This relates to the result in conventional two armed trials that enlarging the sample size in an interim analysis only when a large effect has been observed does not inflate the type I error rate POSCH, BAUER & BRANNATH, Stat Med, 2003; CHEN, DeMETS & LAN, Stat Med, 2004 ) 16/11/2018

A complex selection rule Stop for futility if all interim effect estimates are negative Select one or two treatments with positive effect estimates: Select a single best if If with probability 1/2 either go on with the best or with the two best If with probability 1/3 either select the single best, the best two or only the second best (safety concerns?) The bias is calculated conditionally on no early stopping 16/11/2018

A complex selection rule (cont.) The sample size n is chosen such that an (unadjusted) treatment control comparison by a one-sided fixed sample size test at the level 0.025 has a power of 90% for a true mean treatment-control difference of σ Two parameter constellations are investigated: 1. Equal treatment means 2. Linearly increasing treatment means 16/11/2018

A complex selection rule – the bias 16/11/2018

Complex selection – complex bias Equal treatment means: For very early selection the bias is negligible because due to the large variability we nearly always go on with a single treatment where the unbiased estimate from the second stage sample dominates the overall estimate As compared to the simple selection rule with s=2 the bias of the estimate of the best treatment increases more sharply with r and for r→1 it is larger because of the stopping for futility option If two treatments are selected the second best has to be close to the best leading to a larger bias of the final estimate of the second best than in simple selection (s=2) 16/11/2018

Complex selection– complex bias (cont.) Linearly increasing means If the effects are different in general the bias is reduced k=2: The bias of the treatment best at end is non-monotonic in r. For r→1 the bias of the finally second best treatment slightly exceeds the bias of the finally best treatment because we go on with two treatments only if the second best effect is rather close to the best (and because the stopping for futility option is more effective for the treatment with the true lower effect). There are regions of r where the bias changes the sign as compared to the scenario with equal effects 16/11/2018

Reporting bias of all observed effect estimates Each observed effect estimate is reported separately regardless if the treatment has been selected or not In this case we report the effect estimate in the total sample if the treatment has been selected and the interim effect estimate if it has not been selected The reporting MSE can be defined accordingly! 16/11/2018

Reporting bias, MSE s=1 For s=1 and equal treatment means explicit formula can be derived for the reporting bias and MSE (for the latter is has been a little harder) 16/11/2018

Reporting bias s=1 16/11/2018

Reporting bias (s=1) - to take home For equal treatment means the reporting bias is generally negative: On the one hand if the interim effect is large we tend to dilute the treatment effect by the independent second sample. On the other hand if the interim effect is small we tend to stay with the small effect as it is It is most accentuated and equal for k=2 and k=3 As k increases the probability to be selected decreases. For any j we more often will use the hardly biased first stage estimate, the reporting bias coming closer to zero For r→1 (no selection) the reporting bias tends to zero For r→0 a treatment is selected with a highly variable effect estimate whose distribution is shifted to the left (the reporting bias diverges to minus infinity) 16/11/2018

Admission bias caused by the two pivotal trials paradigm Two identical independent trials comparing a new treatment to a control Each of the preplanned one sided z-tests for the primary outcome variable at the level 0.025 has a power of 90% at an effect size of Δ/σ=1 Estimates are only reported (or relevant for the public in case of registration of a new drug) if both one sided z-tests have been rejected! This will result in a “bias at admission” See earlier work on bias in meta-analyses: HEDGES, J.Educat.Stat., 1984; HEDGES & OLKIN, 1985; BEGG & BERLIN, J.R.S.S.A, 1988 16/11/2018

Admission bias (one or two pivotal trials) as a function of the true effect size δ/σ Here the probability for registration is small (0.025x0.025=0.000625)! 16/11/2018

Admission bias – to take home The mean bias is largest for δ=0. However, there rejection only occurs with a probability of 0.025 for a single trial and with a probability of 0.000625 for two independent trials If the true effect is close to the effect size used for planning the sample size the bias is small and for larger effect sizes approaches 0 quite fast The mean bias is the same for rejection in a single planned study or for the simultaneous rejection in two planned independent trials The MSE is lower in the two as compared to the single study scenario. If in the single study scenario the true effect size is slightly below the targeted one the MSE is slightly below the mean square error of the unconditional mean from a fixed sample size trial (truncated distribution!) 16/11/2018

Selection and bias - concluding remarks Without fixed selection rules the bias can not calculated There are different ways to quantify the bias There are many design features influencing bias The better (later) the selection – the worse the bias The earlier the selection the higher the risk of a wrong selection (Reasonably) reshuffling sample size decreases the bias With increasing treatment differences the bias decreases Do we end with series of two armed fixed sample size trials? But then how to deal with such series? The bad news: the problem of admission bias becomes serious if we deal with underpowered studies The good news: if the studies are well powered for the true effect size the admission bias becomes negligible 16/11/2018

Note that the phenomena of bias has to be considered as an intrinsic feature of human life when selecting, e.g., jobs, friends and partners based on a comparison of past observations afflicted by random variation Thank you for your patience! 16/11/2018