Model Enhanced Classification of Serious Adverse Events SCOPE 2019

Jingshu Liu Lead Data Scientist, Medidata Solutions

Background

“Very high blood pressure” “Bruising on hip due to fall” “Headache for 2 days” “Stroke” “Rash on leg” “Tingling in fingers+toes” Adverse Events (AE) Serious Adverse Events (SAE)

Challenges with SAEs Classification Different standards Within and among organizations

SAE Definition An event is considered serious when the patient outcome is: Death Life-threatening Hospitalization (initial or prolonged) Disability or permanent damage Congenital anomaly or birth defect Required intervention to prevent permanent impairment Other important medical events (that may lead to the above)   Medical and scientific judgement should be exercised in   deciding if an event is ‘serious’ in accordance with these criteria FDA supplements this definition with such categories as “events that may lead to any of the above outcomes, or require intervention to prevent those outcomes, upon appropriate medical judgment”, and “unexpected events for the target condition or population”. A few years ago, in an FDA briefing document on a diabetes drug, it was pointed out that the investigator failed to report 3 stroke events as serious when the patient was not hospitalized.

Challenges with SAEs Classification Different standards Quantitative evidence is frequently lacking Within and among organizations Review is time-consuming Systemic and human error Compounds the problem among over 100 stroke events recorded in those trials, 94% were reported to be serious.

Detection of SAEs is a complex process We have heard from our sponsors that in smaller phase I and II trials, they often review every single adverse event, and in larger phase III trials in which that is infeasible, they review a sampling of all events. Yet fact is serious adverse events only occur a small percentage of the time, so most of the events that get reviewed end up being classified as non-serious.

Data for Medical Review Clinical database External data (e.g. lab data, ECG, imaging data, ePRO, diaries, etc.) Other reports (e.g. admission/discharge reports, autopsy reports, etc.) Important Medical Event (IME) list IME by the Eudravigilance group. A report from the Clinical Trials Transformation Initiative estimated a median review time of 15 minutes per SAE.

Challenges with SAEs Classification Different standards Quantitative evidence is frequently lacking Within and among organizations Review is time-consuming Systemic and human error Many factors to consider Compounds the problem

Generate insights from standardized industry AE data

>15,000 >4,400,000 Trials Trial subjects Medidata Enterprise Data Store >15,000 >4,400,000 Trials Trial subjects

1M 1.8K 130 AE records Trials Sponsors Adverse Events Data - 6% serious Completed Give-to-Get data rights all the records are de-identified. * Data is split by 7:2:1 by patient into training, validation and test set

Seriousness rate increases with severity grade Number of events: 672K 319K 81K 8K 4K Seriousness rate increases with severity grade % serious

AEs with high seriousness rate 98% 98% 96% 94% 94% 94% 94% 93% 92% 91% AEs with high seriousness rate Frequency * Events with more than 100 occurrences in dataset

Classify SAEs with multivariate probabilistic models

Important Medical Event (IME) Age Features Severity Race AE Demographics Sex MedDRA terms ... First event? Event Duration Serious/severity of events Patient Event History ... Multiple events? Outcome Concurrent Events Time between events Study-level features ... Important Medical Event (IME) Sponsor ... Phase Labs* Med Hx* Indication Hospitalization* * These features have not yet been incorporated into our models but will be explored in future work.

How Do We Evaluate Model Performance? Less likely More likely true positives false positives Precision: 0.75 Recall: 0.5 Precision: 0.5 Recall: 0.75 false negatives true negatives SAE Non-SAE How many selected events are SAEs? How many SAEs are selected? Precision = Recall =

Benchmark Model: IME + Severity Rule: Classified as SAE if AE on IME list or Severity = Grade 3 (Severe) or higher Random Model Precision: 0.29 Recall: 0.79

Model 1: Logistic regression Features: Severity, MedDRA PT, indication, average previous occurrence seriousness, age Random Model Precision: 0.43 Recall: 0.90

Model 2: Neural Network Learns interaction between features of each event

Model 2: Neural Network Learns interaction between features of each event Learns complex interactions between concurrent events

Model 2: Neural Network Learns interaction between features of each event Learns complex interactions between concurrent events Learns how to use patient history without manual feature engineering

Model 2: Neural Network Features: All event-, patient-, study-level features (e.g., Severity, MedDRA PT, event on IME or not, etc.) Neural Net Random Model Precision: 0.60 Recall: 0.95 * Area under the PR curve (PR-AUC): Logistic regression - 0.71, Neural Net - 0.77

Medical review: A New Hope

Challenges Addressed Review is time-consuming Human error + different standards Lacking quantitative evidence

Challenges Addressed Review is time-consuming Human error + different standards Lacking quantitative evidence

Challenges Addressed Amount of review needed to capture 99% of SAEs Review is time-consuming Human error + different standards Lacking quantitative evidence IME + Severity Neural Net 96% 41%

Value of Model-Enhanced SAE Classification Improves Accuracy Reduces Time Reduces Workload Our model outperforms the industry baseline of IME + Severity Grade by a large margin Our model can evaluate an event and the corresponding patient profile to make a prediction in seconds Our model prioritizes records for review with higher precision

Thank you. For more information, please contact: jliu@mdsol.com