How to optimize research in ICU - September 24th 2008 How I interpret statistics? Jean-François TIMSIT MD PhD Medical ICU Outcome of cancers and critical illnesses University hospital A Michallon INSERM U 823 Grenoble FRANCE

Biases in RCT and non RCTs Selection bias : sampling frame sufficiently different from the target population, or because the sampling procedure cannot be expected to deliver a sample that is a mirror image of the sampling frame Information biais : risk factor or end-point improperly catched,(no blood culture, no bacteremia…) Confusion bias: variable (event) that both contribute to the end-point (death) and to the disease

Carefully look to your data+++ 90% of the amount of energy necessary to draw conclusions… –Distribution –Outliers –Reproducibility –Missing values –Correlation between variables  data reduction

Stroke 1999;30: à 77.7 Data structure

N Engl J Med 2002;549:556 Analyze the data structure Lancet 2001;357:9-14

External validity

Demonstrate that the patients you enroled are the ones of interest?? Mortality of the control group

Prowess 1690 pts/ 11 countries/ 164 sites!!!! A very few % of the severe sepsis admitted The overal treatment are not standardized… External validity..? –More pragmatic studies enroling all the patients with severe sepsis…. –But…there was a learning curve!!

Outcome= Judgment criteria Precise Reproducible Reflect of what we wanted to measure...This choice is a very important issue

Estimated rate of nosocomial pneumonia? The real rate of NP is 20% The rate of misclassification vary according to the accuracy of the diagnosis True VAP True non VAP total Diagnosed VAP acx Diagnosed Non-VAP bdy Totala+bc+dTotal Se=a/(a+b) Sp=d/(c+d) a+c=x b+d=y

True rate vs estimated rate of an event No VAP VAP T- T Rate of VAP: 26% No VAP VAP T-800 T Se=p[T+]/[D+]= 1 Sp=p[T-]/[D-]= 1 Rate of VAP: 20% No VAP VAP T-722 T = 0.9 X 20 = 0.9 X 80 Se=p[T+]/[D+]= 90% Sp=p[T-]/[D-]= 90%

Estimated effect of a new treatment PlaceboTreatment No CRI CRI Sp=p[T+]/[D+]= 100% Se=p[T-]/[D-]= 100% True rate of CRI: 5% RR=2 « True » 0R=0.49, p= What’s happen if the diagnostic test is not perfectly accurate?

Estimated effect of a new treatment PlaceboTreatment No CRI?? CRI?? 1000 Sp=p[T+]/[D+]= 90% Se=p[T-]/[D-]= 100% True rate of CRI: 5% RR=2 =True CRI * Se + True no CRI*(1-Sp) =50* *0.1=145!!!! 145 « True » 0R=2.05, p=

Estimated effect of a new treatment PlaceboTreatment No CRI CRI Sp=p[T+]/[D+]= 90% Se=p[T-]/[D-]= 100% True rate of CRI: 5% RR=2 « True » 0R=0.49, p= Estimated 0R=0.82, P value= 0.051

Estimated effect of a new treatment PlaceboTreatment No CRI CRI Sp=p[T+]/[D+]= 100% Se=p[T-]/[D-]= 70% True rate of CRI: 5% RR=2 Estimated 0R=0.505, P value= 0,0006 =True CRI * Se + True no CRI*(1-Sp) =50* *0=35 « True » 0R=2.05, p=

Measurement errors If the prevalence of the event is low, you need a very specific test to avoid measurement error of the treatment effect If the prevalence is high, you need a very sensitive one….

What is the optimal clinical end-point? Underlying illnesses Acute disease time Day 14Day 28Day 901y

What is the best??? Day 14  more related to the disease itself…low noise (death due to other cause) Day 28  compromise? Day 90  competing events?, probably more important at the patient’s point of view 1 year   competing events, more important for patient and at the societal point of view All of the end-points  YES!!BUT Multiple comparisons (  NNT,  power) « Survival analyses? »

 (Type I error (%)) 1-  (Power (%)) Number of tests

Genetic profiles > 1000 signals for bacterias > signals for humans Decrease of power and increase in the type I error Signal 1 Signal 2.. Pat 1 Pat 2 Pat 3 Pat 4 Pat 5 Pat 6 Pat 7 Pat 8 Pat 9 Pat 10 Pat 11 Pat 12 …….. Signal 1 Signal 2 Signal 3 Signal 4 signal 5 Signal 6 Signal 7… Pat 1 Pat 2 Pat 3 Pat 4 Pat 5  Mondial consortium, external validation

Time pitfalls Time to measurement of exposure Competing events

NIV failure has not been measured at the beginning of the follow up (time dependent event) JAMA 2000 NIV success NIV failure Invasive ventilation 1,0 0,2 0,4 0,6 0, ,0 0,2 0,4 0,6 0, Cumulative proportion Of patients without penumonia days

Competing events= Informative censor time-to-death (survival analysis) all the survival models consider that censor is non informative « an individual i that is censored at a time t is exposed to same risk of death at the time t+1 than the other patients exposed to the risk » However, this strong assumption is frequently violated, particularly in ICU studies where the time to ward discharge and the time to death are totally dependant ICU discharge acts as a competing event

Randomization…what for? Well done multivariate analysis is able to adjust on known confonders Random allocation is the only way to equilibrate groups on confounding factors..known AND UNKNOWN +++ Treatment A Treatment B DC 5%DC 40% SAPSII 32SAPS II 40 Genetic Fact X 90% Genetic Fact X 10%

RCT: Dogmatic strategy Main principles 1 Do you reach the goal about the statistical power of the study? 2 Do you analysed all included patients? 3 Do you limit the analyses to one primary end-point?  In RCT, if all the goals are reached, only one statistical test and no comparisons between population is sufficient

But… In practice not really applicable –Intermediate analysis should lead to early and more ethical studies (LnMMA, HCG) –It should be more appropriate to analyze data about patients that were effectively treated or with a confirmation of the disease there have been hypothesized at inclusion Ex: Severe sepsis definition needs the occurrence of an infection proven or suspected… Gram negative septicemia need to be immediately treated before the results of the BC –At least 2 judgment criteria: efficacy and side effects… But inflation of type I and II errors (acceptable if a priori designed)

In practice Exclusion is possible if exclusion criteria has been obtained before randomization (even the results are not available) at random if planned in the original protocol Exclusion criteria should not depend of the attending physician expertise One primary end-point and previously designed secondary end-points As final groups are not fully decided at random, group comparability is needed.

A CONFOUNDER… A confounder is associated with the risk factor and causally related to the outcome Carrying matches Lung cancer Smoking

In ICU Many intercurrent events Many interactions between events DNR orders++

Crit Care Med patients included, 1415 (39.2%) experienced one or more AEs 821 (22.7%) had two or more AEs Mean number of AEs per patient was 2.8 (range, 1–26). Six AEs were associated with death: primary or catheter-related BSI OR 2.9;95% CI, 1.6 –5.32 BSI from other sources OR, 5.7; 95% CI 2.66 –12.05 nonbacteremic pneumoniaOR, 1.7; 95% CI 1.17–2.44 deep and organ/space SSI without BSI OR, 3.0; 95% CI, 1.3– 6.8 pneumothorax OR, 3.1; 95% CI, 1.5– 6.3 gastrointestinal bleeding OR, 2.6; 95% CI, 1.4–4.9

Adjustement using a magic « multivariate model » x y z Truth universe in your sample

Adjustement using a magic « multivariate model » x y z

x y z

x y z

x y z

x y z Model using interactions and polynomes…

Validation using external samples x y z Other representative sample of the truth universe

Messages As many possible models as individuals (even more!!) Parcimony decreases model discrimination but improves external validity  the statistical analyses should be precisely designed a priori  Primary and secondary analyses should be precisely planned

Rules for multivariate models Select the model according to the end point Check for its hypotheses The explanatory variables should be –Precisely defined –Not related one to another –Sufficiently frequent in both groups (problem with perfect or quasi perfect discrimination) Ex: Multiple logistic regression in CCM ( ) (Poster 0524 – P Lambrecht and D Benoit – Ghent, Belgium) –Median 6 shortcomings by multiple logistic regression –(significantly decreased when a statistician is a co-author)

How I interpret the result? Discussion with a statistician if you are not familiar with statistics What is the title of the paper you want to do? Subgroup analyses lead to a important increase in the type I error and also in a decrease of the power of your study -exploratory analyses that should be confirmed

Interpret the results with some wisdom…