CBER Selecting the Appropriate Statistical Distribution for a Primary Analysis P. Lachenbruch

CBER A Study of Xeroderma Pigmentosa (XP)  A characteristic of XP is the formation of Actinic Keratoses (AK s )  Multiple lesions appear haphazardly on a patient’s back  The rate of appearance may not be the same for different patients

CBER Background  Analysis: Rank Sum test.  Late in study the Statistical Analysis Plan (SAP) was amended to use Poisson regression  Unclear if stepwise selection of covariates was planned a priori

CBER Study Results  Poisson regression analysis showed highly significant treatment difference (p=0.009) adjusting for baseline AK, age, and age x treatment interaction (stepwise selection)  All these effects were highly significant.  Substantial outlier problem

CBER Assumptions Each patient has the same incidence rate,  per area unit. Chance of more than one AK in small area unit is negligible. Non-overlapping lesions are independent, that is, lesions occurring in one area of the body are not affected by those occurring in another area.

CBER Outliers  Outliers are observations that are jarringly different from the remainder of the data May be multiple outliers If frequency is large, this may be evidence that we have a mixture distribution.  Can substantially affect analysis

CBER Analyses Two-Sample Wilcoxon rank-sum (Mann-Whitney) test trt | obs rank sum expected | | Combined| unadjusted variance adjustment for ties adjusted variance Ho: ak12tot(trt==0) = ak12tot(trt==1) z = Prob > |z| =

CBER Distribution of AK Data at Baseline (Stem and Leaf) (Yarosh et al, Lancet) Lead | Trailing digits 0* | // 4* | 27 // 10* | 0  oops! Lead | Trailing digits 0* | // 4* | 27 // 10* | 0  oops!

CBER Distribution of 12 Month AK Total Data (Stem and Leaf). stem ak12tot,w(10) Lead| Trailing digits 0* | * | * | 3* | 7 // 7* | 1 8* | 9 // 19*| 3  same patient - in placebo group. stem ak12tot,w(10) Lead| Trailing digits 0* | * | * | 3* | 7 // 7* | 1 8* | 9 // 19*| 3  same patient - in placebo group

CBER Results of Poisson Analyses Poisson regression Number of obs = 29 LR chi2(3) = Prob > chi2 = Log likelihood = Pseudo R2 = ak12tot | Coef. Std. Err. z P>|z| [95% Conf. Interval] age | trt | akb | _cons |  G-O-F in control group,  2 = with 8 d.f.  G-O-F in treatment group,  2 =682.5 with 19 d.f.

CBER Permutation Test  Procedure: Scramble treatment codes and redo analysis. Repeat many (5,000?) times.  Count number of times the coefficient for treatment exceeds the observed value.

CBER Command and Output. permute trt "permpois trt ak12tot age akb" rtrt=rtrt rage=rage rakb=rakb,reps(5000) d command: permpois trt ak12tot age akb statistics: rtrt = rtrt rage = rage rakb = rakb permute var: trt Monte Carlo permutation statistics Number of obs = 30 Replications = T | T(obs) c n p=c/n SE(p) rtrt | rage | rakb | Note: c = #{|T| >= |T(obs)|} I deleted the confidence intervals for the proportions

CBER Permutation Tests (2)  Poisson with 5000 Replications  Treatment: p = 0.57  Age: p = 0.62  AK Baseline: p = 0.28  All significant results disappear

CBER Results of Poisson Analysis  Sponsor found that all terms were highly significant (including the treatment x age interaction).  We reproduced this analysis.  We also did a Poisson goodness-of-fit test that strongly rejected the assumption of a Poisson distribution.  What does a highly significant result mean when the model is wrong?

CBER Conclusions  The data are poorly fit by both Poisson and Negative Binomial distributions Permutation tests suggest no treatment effect unless treatment by age interaction is included  Justification of interaction term by stepwise procedure is exploratory  Outliers are a problem and can affect the conclusions.

CBER Conclusions (2)  The results of the study are based on exploratory data analysis.  The analysis is based on wrong assumptions of the data.  Our analyses based on distribution free tests do not agree with the sponsor’s results.  The results based on appropriate assumptions do not support approval of the product.

CBER Suggestions  Conduct a phase II study to determine appropriate covariates.  Need to use appropriate inclusion / exclusion criteria.  Stratification.  a priori specification of full analysis

CBER Reference Yarosh D. et al., "Effect of topically applied T4 endonuclease V in liposomes on skin cancer in xeroderma pigmentosum: a randomised study" Lancet 357: , 2001.

CBER The End

CBER Grid on “Back”

CBER The Data | sex trt akb ak12tot| | | | F | | M | | F | | F | | F | | | | M | | F | | M | | M | | M | | | | F | | F | | F | | F | | F | | | | sex trt akb ak12tot| | F | | F | | M | | F | | M | | | | F | | F | | F | | F | | F | | | | M | | F | | F | | F | | M... |

CBER Descriptive Statistics (1) Baseline AK N Mean SD Control Treatment Months Total AK Control Treatment

CBER Descriptive Statistics (2) Baseline AK Median Min Max Control Treatment Months Total AK Control Treatment

CBER Negative Binomial Model  Need a model that allows for individual variability.  Negative binomial distribution assumes that each patient has Poisson, but incidence rate varies according to a gamma distribution.  Treatment: p = 0.64  Age: p = 0.45  AK Baseline: p =  Age x Treat: p <0.001 Main effect of treatment is not interpretable. Need to look at effects separately by age.

CBER Negative Binomial Results  This model shows only that the baseline AK and age x treatment effects are significant factors.  It also gives a test for whether the data are Poisson; the test rejects the Poisson Distribution: p<  A test based on chisquare test (obs - exp) suggests that these data are not negative binomial.