Associate Director, Biostatistics Qualifying a cognition endpoint for use in multiple sclerosis by combining data from many clinical trials Adam Jacobs Associate Director, Biostatistics Premier Research

The obligatory “big data” quote Big data is like teenage sex: everyone talks about it, nobody really knows how to do it, everyone thinks everyone else is doing it, so everyone claims they are doing it Dan Ariely, January 2013

Outcomes in multiple sclerosis Most widely used outcome measure in MS trials is the EDSS Focused mainly on physical symptoms, particularly walking But other outcomes are important in MS 4 main domains of MS-related disability: Ambulation Dexterity Visual acuity Cognition

Cognition measures 2 main cognition measures have been used in MS trials: Paced Auditory Serial Addition Test (PASAT) Symbol Digit Modalities Test (SDMT) Both mainly measure processing speed, but PASAT is also affected by working memory capacity Neither is currently accepted by regulators as a valid outcome measure for MS trials

FDA clinical outcome assessment qualification program “COA qualification is based on a review of the evidence to support the conclusion that the COA is a well-defined and reliable assessment of a specified concept of interest for use in adequate and well-controlled (A&WC) studies in a specified context of use. COA qualification represents a conclusion that within the stated context of use, results of assessment can be relied upon to measure a specific concept and have a specific interpretation and application in drug development and regulatory decision-making and labeling.”

Objectives of SDMT qualification Floor or ceiling effects Test-retest reliability Changes in scores over time Construct and convergent validity Practice effects Known group validity Sensitivity to change Minimum clinically important difference

MSOAC database A database of 16 MS studies from various sponsors Most (11 studies, 11,843 patients) in RRMS A variety of different treatments and study designs Data lovingly crafted into standardised SDTM datasets by the Critical Path Institute 14 studies (N=12,776) included in current analyses: 2 studies did not measure cognition

Results

Number of Patients (N = 12,776) Analysis Population Number of Patients (N = 12,776) Number of Studies (N = 14)   All studies set 12,776 (100.0%) 14 (100.0%) PASAT set 11,702 (91.6%) 13 (92.9%) SDMT and PASAT set 1,512 (11.8%) 1 (7.1%) SDMT set 2,606 (20.2%) 2 (14.3%)

Age at baseline All studies set SDMT Set N 12,727 2,586 Mean 39.5 38.6 Std. Deviation 9.92 9.43 Minimum 17 18 Median 40.0 39.0 Maximum 72 61

More demographics at baseline (all studies set)

Disease duration at baseline (years) All studies set SDMT Set N 6641 2546 Mean 6.5 5.9 Std. Deviation 7.26 5.41 Minimum Median 4.0 5.0 Maximum 48 40

Distribution of SDMT and PASAT pre-dose 2,583 11,630 Min 0.0 5th centile 23.0 25.0 1st quartile 37.0 42.0 Median 48.0 52.0 3rd quartile 57.0 95th centile 72.0 60.0 Max 110.0

Assessment of test-retest reliability Mixed model repeated measures analysis of SDMT or PASAT Measures taken only from period when EDSS was stable, and no more than 6 months after baseline Included about 30% of all available data Statistical model included terms for test number and interval since previous test to allow for practice effects

Measures of test-retest reliability SDMT PASAT N (observations) 8,567 24,327 N (patients 2,094 7,962 Intraclass correlation coefficient 0.85 0.86 Within-subject residual SD 6.2 4.5

Regression coefficients for practice effects Test number SDMT PASAT 2 -0.02 (-0.09 to 0.05) 0.10 (0.08 to 0.12) 3 0.05 (-0.02 to 0.13) 0.22 (0.20 to 0.24) 4 0.10 (0.03 to 0.18) 0.32 (0.30 to 0.34) 5 0.23 (0.15 to 0.31) 0.44 (0.41 to 0.47) 6 0.33 (0.25 to 0.40) 0.48 (0.45 to 0.51) 7 0.36 (0.28 to 0.44) 0.51 (-0.27 to 1.29) Regression coefficients are relative to first test and expressed as effect sizes with 95% CI

Spearman correlation coefficients: baseline values Measure SDMT PASAT CC 95% CI EDSS -0.34 -0.38 to -0.29 -0.21 -0.23 to -0.19 9-HPT -0.47 -0.51 to -0.43 -0.32 -0.34 to -0.31 T25FW -0.42 -0.46 to -0.38 -0.29 -0.30 to -0.27 LCVA 0.34 0.30 to 0.39 0.20 0.18 to 0.23 BDI -0.20 -0.24 to -0.15 -0.19 -0.23 to -0.15 0.54 0.50 to 0.57

Spearman correlation coefficients: change from baseline at endpoint Measure SDMT PASAT CC 95% CI EDSS -0.12 -0.15 to -0.08 -0.02 -0.04 to 0.00 9-HPT -0.22 -0.27 to -0.18 -0.04 -0.06 to -0.02 T25FW -0.21 -0.25 to -0.16 LCVA 0.19 0.14 to 0.23 0.01 -0.02 to 0.04 BDI -0.30 -0.34 to -0.26 -0.03 -0.07 to 0.02 0.20 0.15 to 0.25

Change in SDMT and PASAT scores with age

Sensitivity to change of SDMT Nature of change N Mean change 95% CI P value Last score before relapse to first score during relapse 185 -0.14 [ -1.20; 0.93] 0.8024 First score during relapse to first score after end of relapse 144 2.16 [ 0.63; 3.69] 0.0060 Baseline to first score after EDSS worsening 212 2.96 [ 1.43; 4.49] 0.0002 Baseline to first score after EDSS improvement 224 5.35 [ 3.57; 7.13] < 0.0001

Some challenges Big data means big datasets: slow to run! Create subsets of data for debugging programs Beware of P values! With 12776 patients, tiny, trivial effects may be statistically significant Standardisation of data from many different trials with different designs Data were not collected for the purpose for which we are using them SDMT and SDTM!

Any questions?