1. Veterinary clinical studies Key issues for statistical analysis Didier Concordet d.concordet@envt.fr ECVPT Workshop July 2009 Ecole Nationale Vétérinaire de Toulouse Can be downloaded at http://www.biostat.envt.fr/spip/spip.php?article34

2. Vocabulary Bias (Statistical & Operational) Blind Review Content Validity Double-Dummy Dropout Equivalence Trial Frequentist Methods Full Analysis Set Generalisability, Generalisation Global Assessment Variable Independent Data Monitoring Committee (IDMC) (Data and Safety Monitoring Board, Monitoring Committee, Data Monitoring Committee) Intention-To-Treat Principle Interaction (Qualitative & Quantitative) Inter-Rater Reliability Intra-Rater Reliability Interim Analysis Meta-Analysis Multicentre Trial Non-Inferiority Trial Preferred and Included Terms Per Protocol Set (Valid Cases, Efficacy Sample, Evaluable Subjects Sample) Safety & Tolerability Statistical Analysis Plan Superiority Trial Surrogate Variable Treatment Effect Treatment Emergent Trial Statistician From ICH Topic E 9

3. Aim of clinical trials To assess the efficacy of a drug in a (target) population Population : the set of individuals that can receive the drug Practically Population Design/Sampling Sample Inference

4. ISSUES When designing the trial When collecting data When analysing data When interpreting results

5. ISSUES When designing the trial When collecting data When analysing data When interpreting results Sampling the target population Different kinds of clinical trials How to detect bias

6. ISSUES When designing the trial When collecting data When analysing data When interpreting results Sampling the target population Different kinds of clinical trials How to detect bias

7. Sampling the target population There exist sources of variation that make the judgment criterion vary Example with two breeds Judgment criterion

8. Sampling the target population The two same breeds with different proportions Judgment criterion

9. Sampling the target population The sample should be representative of the target population Target population breed 3 breed 5 breed 1 breed 2 breed 4 breed 6.<1 year 1<=. <2 years 2<=. <3 years Male Female Sample The sample has the same structure as the population

10. Two main ways to sample the population Randomization: leave chance make the job the percentage of the animals in each subgroup should be close to the population's one. Stratification: help the chance to do the job Build a sample of animals that has exactly the same percentage of individuals in each subgroup as the population. This requires to know the repartition of subgroups in the population.

11. 11 Target population definition An experiment in 2 years old beagles showed that the temperature of dogs treated with the antipyretic drug A decreased by 2 °C. What assumptions do we need for this result to hold for all 3 years old beagles beagles dogs man

12. ISSUES When designing the trial When collecting data When analysing data When interpreting results Sampling the target population Different kinds of clinical trials How to detect bias

13. Different kinds of clinical trials Non inferiority Superiority Equivalence

14. Different kinds of clinical trials Efficacy Non inferiority Reference New treatment Reference –  (penalty)  : non inferiority margin Efficacy Reference +  Reference New treatment Superiority  : superiority margin low high

15. Different kinds of clinical trials Efficacy Equivalence trial Reference Reference –  New treatment Reference + 

16. Non inferiority trial Efficacy Reference Reference –  (penalty) New treatment The new treatment can have a smaller efficacy than the reference treatment

17. Non inferiority trial Efficacy Reference Reference –  (penalty) New treatment the reference treatment is not efficacious animals included in the trial are not sick the judgment criterion is not relevant (e.g. does not vary) delta is too large

18. Is there a problem ? Reference treatmentNew treatment Decrease of rectal temperature of at least 1.5°C Decrease of rectal temperature of at least 1.2°C Cure rate = 75 %Cure rate = 65 % Is there a problem organizing a non inferiority trial able to demonstrate

19. A clinical trial should avoid bias Bias : the difference between the compared drugs at the end of the trial due to other things than the drugs Confusion bias Selection bias Follow-up bias Attrition bias

20. Confusion bias Arises when one do not taking into account a confusion factor. To avoid such bias, the trial should be comparative and should have a contemporary control group used as a reference group. QuestionsWarning Is there a control group ? Is the treatment effect determined with respect to this control group ? Despite a control group the treatment effect is measured with a "before-after" comparison.

21. Selection bias Arises when the two groups to be compared are different (with respect to the endpoint before the beginning of the trial. To avoid it one uses a randomisation : a random allocation of animals into treatment groups QuestionsWarning Is there a randomisation procedure ? Are the two groups balanced ? There is a historical control group (no randomisation) The investigators were able to select the animals for a group

22. Follow-up bias Arises when the follow-up is not the same for the two drugs to be compared. Destroy initial comparability. To avoid it : double blind Questions Warning Is the trial double blind ? Is the rate of concomitant medications the same for the two groups ? Are the protocol deviations similar ? Are the drop-out number similar ? The treatments were discernable The investigators were able to select the animals for a group The judgment criterion was subjective (eg : the animal feels better )

23. Attrition bias Arises when some randomised animals are excluded. To avoid it Analysis of the Intention to Treat dataset QuestionsWarning Is the number of analysed animals equal to the number of randomized animals ? Was an imputation method used for missing data ? Intention to treat analysis Per Protocol analysis (only the animals alive and non excluded were analysed) High rate of concomitant treatments ? High rate of protocol deviations ? High rate of drop-out ?

24. Example Efficacy of an antipyretic drug. Inclusion of 30 dogs with at least 39.5°C of temperature. Temperature (°C) Before treatmentAfter treatment 41 38

25. ISSUES When designing the trial When collecting data When analysing data When interpreting results Missing data

26. 26 Missing data should be adequately reported Three kinds of missingness mechanism data Missing Completely At Random (MCAR) The missingness is independent of data data Missing At Random (MAR) The missingness depends on observed data data Missing Not At Random (MNAR) The missingness depends on the non observed data Ignorable missing data: Data imputation does allow to treat such missing data Leads to the ITT dataset Non-Ignorable missing data: The missingness mechanism has be clearly described

27. 27 Missing Completely At Random MCAR Missingness and outcome are independent the owner of the animal missed a visit to the vet the investigator forgot to write the results the owner moves house Unlikely to occur in a clinical trial

28. 28 Missing At Random MAR Missingness depends on data that have been observed but not on the unobserved (missing) data dropout related to baseline characteristics the animal health has markedly improved or deteriorated since inclusion Assumes that the future trajectories of animals who dropout are similar to those who share the same measurements whether or not they dropout. Frequent in clinical trials.

29. 29 Missing Not At Random MNAR Missingness depends on data that have been unobserved (missing data) sudden decline or improve in health that has not been observed in the previous visits Assumes that the future trajectories of animals who dropout are different to those who share the same measurements Occurs in clinical trials.

30. 30 Can you classify these missing data ? The battery of the thermometer is discharged. I cannot measure the temperature. At the last visit, the dog was well. I called the owner by phone, he did not want to come because he said that the dog was cured. The owner did not come back. I don't know why.

31. ISSUES When designing the trial When collecting data When analysing data When interpreting results Statistical tests Multiple comparisons Data drying off What dataset to analyse ?

32. ISSUES When designing the trial When collecting data When analysing data When interpreting results Statistical tests Multiple comparisons Data washing What dataset to analyse ?

33. Statistical analysis Objective : To draw conclusions on the target population from observation of a sample PopulationSample Inference

34. Things to know about statistical tests Observed difference Test There is a difference in the population. This conclusion is drawn with less than 5% risk. Non significant difference. We would take too much risk by claiming a difference in the population. P≥5% P<5% Sample Target population

35. Repetition of tests also called multiple comparisons Test 1 Risk to wrongly conclude to a difference = 5% Test 2 Risk to wrongly conclude to a difference = 5% Test 4 Risk to wrongly conclude to a difference = 5% Test 3 Risk to wrongly conclude to a difference = 5% Globally, the risk to wrongly conclude to a difference for 4 comparisons is 18%. nglobal risk 10.05 20.10 30.13 50.23 100.40 Risk inflation

36. 36 Multiple comparisons 12345 Mean700880730790930 SD4850554460 One wants to compare the ADG obtained with 5 different diets in pig 13425 Ten T-tests A risk of 5% for each comparison : the global risk can be very large here 40%

37. 37 Choosing the question to get an answer Occurs frequently in the analysis of clinical trials results The question becomes random : it changes with the sample of animals. The question is chosen with its answer in hands… Think about a flip coin game where you win 1€ when tail or head occurs. You choose the decision rule once you know the result of the flip ! Such an approach increases the number of false discoveries.

38. Dog (eff. NSAID)P difference with placebo 1Age<100.92 2Age>=100.95 3Male0.81 4Female0.78 5Format small0.63 5 Format medium0.91 6 Format Large0.74 7Food dry0.01 8Food wet0.63 Data drying off: Analysis in subgroups Target population?!

39. Data drying off: a posteriori choice of the judgment criterion Main criterion Death all causes Secondary criteria Death cardiovascular Sudden death Infarct Vascular cerebral accident Surgery Death all causes Death cardiovascular origin Sudden death Infarct Vascular cerebral accident Surgery No definition of a main criterion Risk to wrongly conclude to efficacy of the new treatment : 30% 7 statistical test Risk to wrongly conclude to efficacy = 5% a priori definition of a main criterion A single statistical test From Cucherat 2005

40. What dataset to analyse ? Intention To Treat dataset is based on the initial treatment intent, not on the treatment eventually administered regardless the drop-out. Per Protocol dataset contains animals who have not dropped out for any reason regardless of initial randomization.

41. 41 Example : complete data

42. 42 Example : complete data

43. 43 Drop-out (MAR) Drop-out

44. 44 Intention To Treat dataset with Last Observation Carried Forward (LOCF)

45. 45 Per Protocol dataset Only the animals that did not dropped-out were used

46. ISSUES When designing the trial When collecting data When analysing data When interpreting results Standard error and standard deviation P-Values

47. 47 Standard error / standard deviation The clairance of the drug was equal to 68 ± 5 mL/mn Two possible meanings depending on the meaning of 5 If 5 is the standard error of the mean (se) there is 95 % chance that the population mean clearance belongs to [68 - 2  5 ; 68 + 2  5 ] If 5 is the standard deviation (SD) 95 % of animals have their clearance within [68 - 2  5 ; 68 + 2  5 ]

48. 48 P values The difference between the effect of the drugs A and B is not significant (P = 0.56) therefore drug A can be substituted by drug B. NO The only conclusion that can be drawn from such a P value is that you didn't see any difference between the effect of the drugs A and B. That does not mean that such a difference does not exist. Absence of evidence is not evidence of absence

49. 49 P values The drug A has a higher efficacy than the drug B (P = 0.001) The drug C has a higher efficacy than the drug B (P = 0.04) Since 0.001<0.04 the drug A has a higher than the drug B. NO The only conclusion that can be drawn from such a P value is that you are sure than A>B and less sure than C>B. This does not presume anything about the amplitude of the differences. Significant does not mean important

50. 50 How to avoid these problems ? Consult your preferred statistician for help in the design of complicated experiments Use basic descriptive statistics first (graphics, summary statistics,…) Use common sense Consider to learn more statistics