1. Can administrative data increase the practicality of clinical trials? An example from the Women’s Health Initiative Garnet Anderson Fred Hutchinson Cancer Research Center

2. Pragmatic trial (coined by Schwartz and Lellouch in 1967) A randomized controlled trial designed to inform decisions about practice Used to describe a trial designed to test the effectiveness of the intervention in a broad routine clinical practice setting (as opposed to testing the efficacy in an ideal setting)

3. Pragmatic methods and motivations Broaden eligibility to improve generalizability, increase recruitment yield and perhaps reduce costs Test interventions/delivery mechanisms that better emulate clinical practice and thus improve estimates of population level impact and reduce costs Limit data collection to minimize participant burden and reduce costs

4. Was the original Women’s Health Initiative a pragmatic trial? os

5. WHI transitions in post-intervention phase Protocol streamlined to annual mail follow-up In 2010, documentation and central adjudication of outcomes limited to African American, Hispanic and former HT participants (Medical Records Cohort, n~22,000) Outcomes for remaining participants (Self-Report Cohort, N~71,000) limited to self-report or passive follow-up sources (NDI, Medicare)

6. A Pragmatic Trial : Physical Activity to Improve CV Health in Women Marcia L. Stefanick, Ph.D. Charles L. Kooperberg, Ph.D. Andrea Z. LaCroix, Ph.D. Women’s Health Initiative Strong & Healthy Trial 1 U01 HL122280-01

7. Primary Hypothesis To assess whether aerobic physical activity combined with muscle strengthening, balance and flexibility exercises, and reduced sedentary behavior, will reduce major CV events (MI, stroke, CV death) in older women, compared to “Usual Activity” (Control) over 4-5 years of follow-up Based on: Report of the Physical Activity Guidelines Advisory Committee, 2008. (Chapter 5: Active Older Adults) http://www.health.gov/paguidelines/guidelines/chapter5.aspxhttp://www.health.gov/paguidelines/guidelines/chapter5.aspx Implemented through: National Institutes on Aging (NIA) http://go4life.nia.nih.gov/http://go4life.nia.nih.gov/

8. Eligibility WHI participant, alive and in active follow-up In Medical Records Cohort or in Self Report cohort and enrolled in Medicare Part A/B Exclusions Inability to walk Dementia Residing in nursing home

9. Eligible based on existing data Intervention (n ~ 26,000) Intervention (n ~ 26,000) Consent WHISH PA (Go4Life®) Intervention deliver mail-based [+ website, etc.] ± IVR** (phone) + live advisor, PRN WHISH PA (Go4Life®) Intervention deliver mail-based [+ website, etc.] ± IVR** (phone) + live advisor, PRN Follow, per WHI protocol no yes Randomize Follow, per WHI protocol Control (n ~26,000) Control (n ~26,000) ** Interactive Voice Response System (Consent) Opt Out: no Study Design: Zelen’s randomized consent design Zelen, M. The New England Journal of Medicine 1979; 300: 1242–1245.The New England Journal of Medicine

10. Sources of outcomes data Self-reported health events annually, for all WHI participants Fully documented, adjudicated outcomes for the Medical Records Cohort Outcomes derived from Medicare claims and NDI among women from the Self-Report Cohort

11. Considerations in using Medicare claims data for outcomes assessment Coverage Types of data available for outcomes assessments Quality of inference Logistics

12. Medicare data routinely available Denominator Inpatient (MedPar) Outpatient Home Health Skilled Nursing Hospice Durable Medical Equipment Carrier Part D—Prescription Drug

13. WHI Medicare linkage Submitted 151,116 names, all with valid SSN 140,471 (93%) were age eligible as of 12/31/07 or had already died at age>65 142,195 names returned by Center for Medicare/Medicaid Services (CMS) 132,109 perfect matches on 5 identifiers (94% of submitted, age-eligible) 3,203 “fuzzy” matches (2% of submitted age eligible) Small discrepancies in one identifier 90% linkage of submitted participants 96% linkage of age eligible participants Note: All WHI participants are now over age 65

14. Linkage to Medicare does not mean claims data are available Medicare claims not available for Managed Care (MC) members Included in denominator file but because reimbursement is capitated, MC organizations don’t submit claims MC penetrance varies over time and geographic region Individuals may change their coverage type over time

15. WHI participants’ Medicare enrollment status by calendar year

16. Individuals can change their type of coverage ~98% kept the same type of coverage over a one year period (1/1/2011-1/1/2012) 64% had Fee-For-Service plans (Part A or A+B) 34% were in Managed Care Of the 2.12% that change types of coverage, 0.55% went from Managed Care to Fee-For-Service 1.57% went from FFS to MC

17. Determining “events” in claims data Claims include up to 10 diagnosis codes based on ICD-9-CM One “primary” diagnosis Up to 9 secondary diagnosis “Primary” designation and number of secondary diagnoses included is determined by the submitting institution

18. Assessing agreement rates between WHI adjudicated events and Medicare claims- based diagnoses Eligibility: Age ≥ 65 at WHI enrollment Linked to Medicare and continuously enrolled in fee-for-service Medicare Part A (in-patient) WHI events Documented, centrally adjudicated (hospitalized) cardiovascular outcomes Only first post-enrollment event of each type considered

19. Chronic Disease Warehouse provides a catalog of algorithms for diagnoses Myocardial infarction: hospital discharge codes (410.x0, 410.x1) as either primary or one of 9 secondary discharge diagnosis codes. Coronary revascularization: ICD-9-CM procedures codes for CABG (36.1x, 36.2) or PCTA (00.66, 36.0, 36.00, 36.01, 36.02, 36.05, 36.97) Stroke: ICD-9-CM codes (430.xx, 431.xx, 433.x1, 434.x1, 436.xx, 437.1x, and 437.9x) in any diagnostic position Abdominal aortic aneurysm ICD-9-CM diagnosis (441.3-441.5, 441.9) or procedure (38.34,38.44,39.25,39.52,39.71) or CPT codes (35081, 35082, 35102, 35103, 35091, 0001T, 0002T, 35800, 34802-34805, 34830-34832) Research Data Assistance Center (ResDAC.org) is an excellent resource for working with CMS data

20. Agreement rates for acute MI using principle diagnosis code WHI MedicareYesNoTotal Yes 9142811195 No4313577136202  (95% CI) Total1345360520.71 (0.69-0.73) Sens68% Spec99.2% PPV76% NPV98.8% Overall accuracy 98.1% Hlatky et al., Circulation: Cardiovasc Qual Outcomes (2014)

21. Sensitivity improves but PPV declines using principle or secondary diagnosis codes for aMI WHI MedicareYesNoTotal Yes 10624391501 No2833561335896  (95% CI) Total1345360520.74 (0.72-0.75) Sens79% Spec98.8% PPV71% NPV99% Overall accuracy 98.0% Hlatky et al., Circulation: Cardiovasc Qual Outcomes (2014)

22. Agreement rates for procedures are higher: Coronary bypass graft surgery WHI MedicareYesNoTotal Yes 79596891 No533654936506  (95% CI) Total848365490.91 (0.90-0.93) Sens94% Spec99.7% PPV89% NPV99.9% Overall accuracy 99.6% Hlatky et al., Circulation: Cardiovasc Qual Outcomes (2014)

23. Agreement rates: Stroke events WHI MedicareYesNoTotal Yes 478318796 No1043081730603  (95% CI) Total582308170.91 (0.90-0.93) Sens82% Spec99.0% PPV60.1% NPV99.7% Overall accuracy 98.7% Lakshminarayan et al., Stroke (2014)

24. Medicare up-coding or missing WHI events WHI No/ Medicare Yes PPV after Adjustment All31860.1% No self-reported, hospitalized event 182 (57%)77.9% No adjudication57 (18%)85.8% Adjudication did not identify stroke 79 (25%) Lakshminarayan et al., Stroke (2014)

25. Sensitivity and PPV improve when looking at agreement rates at the person level: Stroke WHI MedicareYesNoTotal Yes 505240745 No773081730603  (95% CI) Total582308170.91 (0.90-0.93) Sens86.8% Spec99.0% PPV67.8% NPV99.7% Overall accuracy 98.7% Lakshminarayan et al., Stroke (2014)

26. Agreement rates for cancer incidence SiteWHI-Yes CMS-Yes WHI-No CMS-Yes WHI-No CMS-Yes WHI-No CMS-No Sens (%) Spec (%) PPV (%) NPV (%) Overall Breast34511562569830298.4 68.899.9498.4 Colorectal11456133910594796.799.465.199.9699.4 Endometrial621146176164297.399.881.099.9799.7 Lung12366346310546895.299.466.199.9999.4 Melanoma3577852210535594.299.331.399.9899.2 Ovarian353372198616894.999.648.799.9899.6 Medicare derived cancers based on presence of relevant ICD-9 cods in MedPar (in patient data), any position, or the first occurring combination of 2 outpatient or carrier claims containing these codes that are 1-365 days apart.

27. Analyzing outcomes found in claims but not in WHI Breast (n=1562) Colorectal (N=613) Endometrial (N=146) Ovarian (N=372) N(%) Related diagnosis888 (56.9)210 (34.3)62 (42.5)199 (53.5) Other cancer or hysterectomy reported 292(18.7)130 (21.2)47 (32.2)91 (24.5) Other non-cancer hospitalization reported 199 (12.7)148 (24.1)11 (7.5)41 (11.0) Cancer self-reported & denied, or not adjudicated 95 (6.1)85 (13.9)19 (13.0)24 (6.5) No information88 (5.6)40 (6.5)7 (4.8)17 (4.6)

28. Correspondence of event dates aMIStrokeAAALE PADCAS Exact same date82%83%89%75%90% Within ± 1 day88%89%79%93% Within ± 7 days95%89%82%96% Within ± 30 days94%96%92%85%96% Exactly ± 365 days0.7%0.4% Hlatky et al., Circulation: Cardiovasc Qual Outcomes (2014) Lakshminarayan et al., Stroke (2014) Mell et al., J of Vascular Surgery (2014)

29. Summary of comparisons between claims- based and protocol-defined events in sample with continuous FFS Medicare enrollment More events counted in Medicare than in WHI across all conditions Agreement rates ranged from good to excellent for clinical diagnoses Excellent agreement rates for procedures Accuracy of dates is acceptable for most failure time analyses Errors in both sources contribute to the disagreements

30. Is outcome misclassification in an RCT benign?

31. Existing methods for mismeasured outcomes focus on discrete proportional hazards Halloran ME and Longini IM. Using validation sets for outcomes and exposure to infection in vaccine field studies. Am J of Epimiol 2001 Meier AS, Richardson BA, and Hughes JP. Discrete proportional hazards models for mismeasured outcomes. Biometrics 2003 Magaret AS. Incorporating validation subsets into discrete proportional hazards models for mismeasured outcomes. Statist in Med 2008

32. Comparing RCT results using adjudicated and claims based outcomes Sample: Women > 65 years of age at randomization in either the HT trial component ITT analyses comparing HT to placebo WHI adjudicated outcomes, with censoring at last-follow-up date or death from other causes Claims-based outcomes, with censoring at end of enrollment in FFS Medicare, death from other causes, or 12/31/2007 (last available claims data)

33. Effect of misclassification of failure time events on inference in a RCT setting: WHI HT trial Hlatky et al., Circulation: Cardiovasc Qual Outcomes (2014) Medicare found fewer clinical diagnoses, more procedures, similar hazard ratios and overall inference

34. Assumptions: Independent measurement error: Errors in outcomes data are comparable across treatment arms May not be defensible if treatment leads to symptoms the result in greater physician contact/diagnostic procedures, etc. Outcomes information from secondary source (claims) does not affect the failure risk, given information on true failure time and treatment assignment

35. Some impracticalities of using claims data for outcomes surveillance

36. Events data collection is not defined by protocol Study outcomes primarily limited to ICD-9 diagnoses or procedures Diagnoses represent the community standard(s) Changes in diagnostic procedures/codes occur outside of the protocol, may be driven by economic factors Data availability is at the mercy of another agency, its policies and practices and those of the institutions who submit data to it

37. Trial monitoring complexities Will data be available soon enough for monitoring purposes? Annual installments, based on calendar year Timeline for release is not guaranteed Changes in data file structure over time Requires additional processing time Difficulty in locking the code for definitions

38. Outcomes collection and analysis plan For participant in the Medical Records Cohort, document and adjudicate all key outcomes For the remaining participants in the Self Report cohort, use Medicare Part A/B Documenting/adjudicating events for those who switch to managed care Analyses based on a stratified Cox Proportional Hazards model with source of data as one of the stratification factors

39. Monitoring plan considerations Intervention is currently available to all (including the comparison group) Ethical requirements to stop for efficacy are reduced Need a full-scale evaluation of this public health program Safety concerns are dominant Medicare may be too tardy, insensitive to monitor safety Options: Supplement Medicare data with self-report, particularly in the early phases

40. Conclusions Claims data provide a key source of selected outcomes data that are standardized and available across the nation for a large segment of the population The correspondence between claims derived outcomes and traditionally documented and adjudicated outcomes are good to excellent but vary by type of outcome Randomized trials using claims data for outcomes may derive valid inference if measurement error assumptions are met Timeliness of obtaining claims data may not be adequate for trial monitoring

41. Dale Burwen Ross Prentice Mary Pettinger Roberta Ray Joseph Larson Charles Kooperberg Andrea LaCroix Marcia Stefanick Acknowledgement of colleagues and collaborators