1. Approaches and Challenges in Accounting for Baseline and Post-Baseline Characteristics when Comparing Two Treatments in an Observational/Non-Randomized Setting
Sept 25, 2019
Joseph Massaro
Boston University School of Public Health
Boston Biomedical Associates

2. Introduction
Studies in an experimental treatment for a serious rare disease for which there are no current available treatments are often single-arm due to:
Desire to maximize number of patients receiving active treatment to obtain as precise estimates as possible for outcomes
Patients may wish to not receive a “placebo” (in addition to best medical care) in this setting

3. Introduction
Despite single-arm nature of such trials, it is still desired to compare treated patients to a “Control” group.
The “Control” group often consists of untreated patients from an observational “natural history” study
Natural history: untreated patients followed longitudinally, treated with best medical care but no experimental treatment, after disease diagnosis

4. Introduction
In such “non-randomized” treatment comparisons, potential issues include difference between experimentally-treated and untreated on:
Distribution of baseline characteristics
Medical care practices, if treated and untreated cohorts are not followed contemporaneously
Post-baseline operational characteristics such as different follow-up visit schedules
And within natural history, patients may not follow consistent follow-up schedules.

5. Introduction
Similar issues exist in post-marketing observational “Registries”:
Similar to “natural history” studies, but where
Some patients receive “experimental treatment” after its regulatory approval, at discretion of physician/patient
Some/all such patients also contribute untreated follow-up to the registry prior to regulatory approval

6. Introduction Post-marketing registries in rare diseases
Often conducted out because experimental product was approved on a surrogate biomarker, but desire is to compare treated to untreated on incidence of clinical outcomes
E.g., mortality, cardiovascular event

7. Treated vs. Natural History
Example: Hutchinson-Gilford Progeria Syndrome
Segmental premature aging
Sporadic, autosomal dominant
Lifespan avg yrs without HGPS-specific treatment
Premature atherosclerosis, CV failure
One of the 5 most rarest diseases (Prevalence: 1 in 20 million)

8. Treated vs. Natural History Example: Progeria
Several treatments have been assessed in clinical trials in children with Progeria
Approximately 30 children in each trial
All received experimental treatment
The experimental treatment for two trials was Lonafarnib (n=63 for the two trials combined)
Biomarkers (e.g., pulse wave velocity, echodensity) were measured in trials
No biomarker information is generally available on untreated patients.

9. Treated vs. Natural History Example: Progeria
Lonafarnib vs. Untreated
Survival information (age at death, last known age alive) is available for untreated patients born 1876 or later (n=195)
Collected by Progeria Research foundation from literature review and/or medical records
Survival information available for all treated patients
Objective: compare treated vs. untreated on survival, accounting for non-randomized nature of comparison

10. Treated vs. Natural History Example: Progeria
Lonafarnib (n=63) vs. Untreated (n=195) on survival
Issue: Patients not enrolled contemporaneously
Untreated patients from 1876 to present
Lonafarnib patients treated 2007 to present
Solution: Include only untreated patients born after 1990 (earliest birth year of treated patients)
Untreated sample sizes reduces from n=195 to n=87
Sensitivity analysis: Include all patients regardless of date of birth

11. Treated vs. Natural History Example: Progeria
Side note: No indication that year of birth affects survival:

12. Treated vs. Natural History Example: Progeria
Lonafarnib (n=63) vs. Contemporaneous Untreated (n=87) on survival
Issue: Groups not necessarily equivalent on distribution of important baseline characteristics
Solution: Match treated patients 1:1 to untreated on sex and on continent of residence (age is discussed on next slide)
Continent was chosen since it may represent available medical care
Country would have been more ideal, not enough untreated patients (n=87) for such a match
“Genotype” is important, but unknown for many untreated patients.
No other baseline characteristics were available for untreated.
If more characteristics were available, how possible would matching be given small available untreated sample size (n=87)?

13. Treated vs. Natural History Example: Progeria
Lonafarnib (n=63) vs. Contemporaneous Untreated (n=87) on survival
Issue: Cannot begin survival analysis follow-up time at birth
Average age of start of lonafarnib was 6.9 years
Many untreated patients died in first 5 years of life
Comparing treated vs. untreated on survival from birth has survival bias toward treated patients.

14. Treated vs. Natural History Example: Progeria
Lonafarnib (n=63) vs. Contemporaneous Untreated (n=87) on survival
Solution: For a treated patient, randomly select an untreated patient (from same sex and continent) alive at age treated patient began lonafarnib.
Once selected, an untreated patient no longer available for matching
“Time 0” for survival analysis for each patient in the matched pair is the age the treated patient began lonafarnib.
Follows approach discussed in Li et al. (2014)
Note: propensity matching difficult to carry out in this setting
How is baseline defined for untreated patients?
Li Y, Schaubel DE, He K. Matching methods for obtaining survival functions to estimate the effect of a time-dependent treatment. Stat Biosci. 2014;6(1):

15. Treated vs. Natural History Example: Progeria
Lonafarnib (n=63) vs. Contemporaneous Untreated (n=87) on survival
Outcome was survival through approximately 2 years post start of lonafarnib.
Groups compared using unadjusted Cox PH regression, conditioned on the matched pair.
Kaplan-Meier curves presented.
Results published in JAMA (2018)
Statistical reviewer was concerned that KM estimates may be biased in this situation, based on Li et al. (2014)

16. Treated vs. Natural History Example: Progeria
Reviewers/Li et al. (2014) 3 reasons as to why KM estimates may be biased in this setting:
Longer “Time to treatment” (T) are more likely to be censored; hence the distribution of T is a biased sample of shorter T values than would exist in the population.
Our Response: Median time to treatment was ~8 years of age. Given (a) age of diagnosis is approximately 1.5 – 2 years, and (b) if lonafarnib is approved it would be given at diagnosis, the observed distribution of T is most likely not less than the distribution in the population.

17. Treated vs. Natural History Example: Progeria
Reviewers/Li et al. (2014) 3 reasons as to why KM estimates may be biased in this setting:
Longer “Times to treatment” (T) and post-treatment time-to-death (D-T) are not usually independent.
Our Response: R2 of T and D-T was <0.01 in treated patients.
Does not necessarily prove T and D-T are independent, but provides support that they may be independent.

18. Treated vs. Natural History Example: Progeria
Reviewers/Li et al. (2014) 3 reasons as to why KM estimates may be biased in this setting:
Matched-yet-untreated patients can receive treatment after being matched.
Our Response: Only patients untreated for entire follow-up were used in matching.
One must carefully consider the appropriate analysis technique in this setting.

19. Treated vs. Natural History Example: Progeria
Results: p=0.04

20. Registry Example: Organ Failure*
Treatment for organ failure approved based on a continuous surrogate endpoint
Potentially low power for time-to-event endpoint in formal clinical trials
Observational study data used to augment database and assess treated vs. untreated difference on time-to-event and again on the continuous outcome
*Analyses still underway so further details cannot be revealed at this time.

21. Registry Example: Organ Failure
Observational studies:
Natural History: untreated patients
Registry: treated patients, untreated patients

22. Registry Example: Organ Failure
Objective: compare untreated patients from natural history to treated patients from Registry on outcomes
Comparisons for time-to-event outcome handled in similar manner as for Progeria
Focus here is treated vs. untreated comparison on the continuous biomarker.
As with time-to-event, issues for analysis need to be considered.

23. Registry Example: Organ Failure
Issue: Treated and untreated patients differ on important baseline characteristics.
Solution: Match on characteristics
Sex, phenotype of disease
Randomly select untreated patient with a biomarker measured within 5 years of age at which treated patient’s baseline biomarker was measured
Follow-up begins at this “matched” age
Ideal to further find untreated patients whose biomarker value was within a caliper of treated patient’s baseline biomarker, but this lead to too few matches.
Performed as a sensitivity analysis.

24. Registry Example: Organ Failure
Issue: More treated than untreated patients.
Solution: Perform two sets of matches:
1:1; for each untreated patient, randomly select one treated patient for analysis.
Mixed model analysis with random intercept and slope used to compare treatments
X:X; untreated patients matched to >1 treated patient; all treated patients included in analysis.
Untreated patients weighted by inverse of number of times patient was matched.

25. Registry Example: Organ Failure
Issue: Biomarker assessed longitudinally, but no consistency between patients with respect to schedule of measurements
Solution: Compare treated vs. untreated on slope of biomarker over time.
Analysis limited to patients with follow-up >= 0.5 years.
Follow-up truncated to 5 years
Complexity: are data “missing” or just not measured.

26. NERDS
FYI, if interested in rare diseases: The 2019 New England Rare Disease Statistics (NERDS) Workshop ( will be held on October 11, 2019, at Seaport Hotel in Boston. 

27. Conclusion
Comparing two treatments for a rare disease in an observational setting poses various challenges
Different demographic distribution between treatments
Definition of baseline for untreated patients
Different follow-up schedules
Small sample size
Care must be taken in analysis
Matching/adjusting as best as possible
Many demographic variables may not be collected on all patients
Small sample size may limit desired effect of matching
Consider effect of no set visit schedule for measuring outcome
Consider various sensitivity analyses
THANK YOU VERY MUCH