Clinical Event Classification: Strategies and Practices Mark A Hlatky, MD Stanford University School of Medicine September 27, 2018

Outline Is clinical event classification necessary? Experience in WHI study Possible implications What are best practices when it is done? Some take-aways from this meeting

Why classify events? What does a CEC add? Independent, blinded adjudication of events increases credibility of study results Standardized data collection and definitions of key outcome events Satisfying regulatory requirements for new drug approval Cost, time, and complexity to RCTs Should test value added by CEC process

Comparison of CEC and claims Pragmatic clinical trials may use existing data collection processes to simplify studies The Women’s Health Initiative (WHI), a large NIH RCT, linked the trial database to CMS claims Opportunity to evaluate the accuracy of hospital claims data in assessing CV outcomes: MI, CABG, and PCI Circ Cardiovasc Qual Outcomes 2014;7:157-162.

Methods Women ≥65 yo at WHI enrollment WHI MI = best available adjudication (central > local) CMS MI = ICD.9 code of 410.x1 (acute MI, initial admission) Censored at end WHI follow-up, loss of Medicare Part A, or HMO enrollment Training set = not in hormone RCTs Application to hormone RCTs Circ Cardiovasc Qual Outcomes 2014;7:157-162

Definition of MI WHI process Participant reported a possible event Records requested and reviewed by CEC MI Dx by ECG, biomarkers, symptoms MI categorized as “resulting from a procedure” CMS Process 1º = principal reason for admission, 2º = conditions also present ICD9 Codes: 410 = acute MI, 411 = other acute CHD, 412 = old MI, 413 = angina, 414 = other chronic CHD Circ Cardiovasc Qual Outcomes 2014;7:157-162

CMS MI vs. WHI MI in Training Set 37,397 women ≥65 yo w CMS data 1,345 WHI MI cases (172 procedure related) 1,195 CMS MI cases (1º dx) 306 more with acute MI as 2º dx __ WHI MI_ Kappa Yes No CMS (1º) Yes 914 281 0.71 No 431 35,771 CMS (1º, 2º) Yes 1,062 439 0.74 No 283 35,613 Circ Cardiovasc Qual Outcomes 2014;7:157-16

Reasons for CMS Yes/WHI No 281 with CMS MI = 1º MI dx, no WHI MI No event reported to WHI (49%) Inadequate documentation—no records (5%) or key data missing (9%) Adjudicated, but No CHD outcomes (16%) Another CVD outcome (23%) CHD deaths w/o MI (2%) Circ Cardiovasc Qual Outcomes 2014;7:157-16

Reasons for CMS No/WHI Yes 431 with WHI MI, no CMS MI (1º diagnosis) No CMS admission <30 days (15%) CMS 2º dx of MI (32%) WHI MI procedure related (13%) Other CVD 1º dx (39%) Non-CVD 1º dx (15%) Circ Cardiovasc Qual Outcomes 2014;7:157-16

Intention to treat comparison

Inights from comparing CEC with Claims Claims data identified events not reported by WHI participants Agreement between claims and CEC data was very good for MI, and excellent for PCI and CABG (kappa 0.88 to 0.91) Some disagreements due to: No or incomplete records for adjudication Differences in diagnostic criteria applied by CEC adjudicators vs local physicians MIs due to procedures or non-cardiac illness were often missed in claims

Some reflections on best practices CEC is more valuable when study intervention is applied during hospitalization for acute illness or procedures More difficult to identify MI in this setting CEC more important for drugs in development than for comparative effectiveness studies of well-established drugs Regulatory requirements to meet

More reflections on best practices Application of clear, standardized diagnostic criteria for events decreases variation Interobserver variation is more often due to different dx criteria than to variable application of standardized dx criteria Standardizing CEC processes, however, could impede innovation Could test importance of key processes (e.g. meetings vs independent evaluations) CEC might be applied to a random sample to validate process (e.g. claims) for all events