1. Associate Director, Biostatistics
Qualifying a cognition endpoint for use in multiple sclerosis by combining data from many clinical trials
Adam Jacobs
Associate Director, Biostatistics
Premier Research

2. The obligatory “big data” quote
Big data is like teenage sex: everyone talks about it, nobody really knows how to do it, everyone thinks everyone else is doing it, so everyone claims they are doing it
Dan Ariely, January 2013

3. Outcomes in multiple sclerosis
Most widely used outcome measure in MS trials is the EDSS
Focused mainly on physical symptoms, particularly walking
But other outcomes are important in MS
4 main domains of MS-related disability:
Ambulation
Dexterity
Visual acuity
Cognition

4. Cognition measures
2 main cognition measures have been used in MS trials:
Paced Auditory Serial Addition Test (PASAT)
Symbol Digit Modalities Test (SDMT)
Both mainly measure processing speed, but PASAT is also affected by working memory capacity
Neither is currently accepted by regulators as a valid outcome measure for MS trials

5. FDA clinical outcome assessment qualification program
“COA qualification is based on a review of the evidence to support the conclusion that the COA is a well-defined and reliable assessment of a specified concept of interest for use in adequate and well-controlled (A&WC) studies in a specified context of use. COA qualification represents a conclusion that within the stated context of use, results of assessment can be relied upon to measure a specific concept and have a specific interpretation and application in drug development and regulatory decision-making and labeling.”

6. Objectives of SDMT qualification
Floor or ceiling effects
Test-retest reliability
Changes in scores over time
Construct and convergent validity
Practice effects
Known group validity
Sensitivity to change
Minimum clinically important difference

7. MSOAC database A database of 16 MS studies from various sponsors
Most (11 studies, 11,843 patients) in RRMS
A variety of different treatments and study designs
Data lovingly crafted into standardised SDTM datasets by the Critical Path Institute
14 studies (N=12,776) included in current analyses: 2 studies did not measure cognition

8. Results

9. Number of Patients (N = 12,776)
Analysis Population
Number of Patients (N = 12,776)
Number of Studies (N = 14)
All studies set
12,776 (100.0%)
14 (100.0%)
PASAT set
11,702 (91.6%)
13 (92.9%)
SDMT and PASAT set
1,512 (11.8%)
1 (7.1%)
SDMT set
2,606 (20.2%)
2 (14.3%)

10. Age at baseline All studies set SDMT Set N 12,727 2,586 Mean 39.5 38.6
Std. Deviation
9.92
9.43
Minimum 17 18
Median
40.0
39.0
Maximum 72 61

11. More demographics at baseline (all studies set)

12. Disease duration at baseline (years)
All studies set
SDMT Set N
6641
2546
Mean
6.5
5.9
Std. Deviation
7.26
5.41
Minimum
Median
4.0
5.0
Maximum 48 40

13. Distribution of SDMT and PASAT pre-dose
2,583
11,630
Min
0.0
5th centile
23.0
25.0
1st quartile
37.0
42.0
Median
48.0
52.0
3rd quartile
57.0
95th centile
72.0
60.0
Max
110.0

14. Assessment of test-retest reliability
Mixed model repeated measures analysis of SDMT or PASAT
Measures taken only from period when EDSS was stable, and no more than 6 months after baseline
Included about 30% of all available data
Statistical model included terms for test number and interval since previous test to allow for practice effects

15. Measures of test-retest reliability
SDMT
PASAT
N (observations)
8,567
24,327
N (patients
2,094
7,962
Intraclass correlation coefficient
0.85
0.86
Within-subject residual SD
6.2
4.5

16. Regression coefficients for practice effects
Test number
SDMT
PASAT 2
-0.02 (-0.09 to 0.05)
0.10 (0.08 to 0.12) 3
0.05 (-0.02 to 0.13)
0.22 (0.20 to 0.24) 4
0.10 (0.03 to 0.18)
0.32 (0.30 to 0.34) 5
0.23 (0.15 to 0.31)
0.44 (0.41 to 0.47) 6
0.33 (0.25 to 0.40)
0.48 (0.45 to 0.51) 7
0.36 (0.28 to 0.44)
0.51 (-0.27 to 1.29)
Regression coefficients are relative to first test and expressed as effect sizes with 95% CI

17. Spearman correlation coefficients: baseline values
Measure
SDMT
PASAT CC
95% CI
EDSS
-0.34
-0.38 to -0.29
-0.21
-0.23 to -0.19
9-HPT
-0.47
-0.51 to -0.43
-0.32
-0.34 to -0.31
T25FW
-0.42
-0.46 to -0.38
-0.29
-0.30 to -0.27
LCVA
0.34
0.30 to 0.39
0.20
0.18 to 0.23
BDI
-0.20
-0.24 to -0.15
-0.19
-0.23 to -0.15
0.54
0.50 to 0.57

18. Spearman correlation coefficients: change from baseline at endpoint
Measure
SDMT
PASAT CC
95% CI
EDSS
-0.12
-0.15 to -0.08
-0.02
-0.04 to 0.00
9-HPT
-0.22
-0.27 to -0.18
-0.04
-0.06 to -0.02
T25FW
-0.21
-0.25 to -0.16
LCVA
0.19
0.14 to 0.23
0.01
-0.02 to 0.04
BDI
-0.30
-0.34 to -0.26
-0.03
-0.07 to 0.02
0.20
0.15 to 0.25

19. Change in SDMT and PASAT scores with age

20. Sensitivity to change of SDMT
Nature of change N
Mean change
95% CI
P value
Last score before relapse to first score during relapse
185
-0.14
[ -1.20; 0.93]
0.8024
First score during relapse to first score after end of relapse
144
2.16
[ 0.63; 3.69]
0.0060
Baseline to first score after EDSS worsening
212
2.96
[ 1.43; 4.49]
0.0002
Baseline to first score after EDSS improvement
224
5.35
[ 3.57; 7.13]
<

21. Some challenges Big data means big datasets: slow to run!
Create subsets of data for debugging programs
Beware of P values! With patients, tiny, trivial effects may be statistically significant
Standardisation of data from many different trials with different designs
Data were not collected for the purpose for which we are using them
SDMT and SDTM!

23. Any questions?