Presentations in this series 1.Overview and Randomization 2.Self-matching 3.Proxies 4.Intermediates 5.Instruments 6.Equipoise Avoiding Bias Due to Unmeasured Covariates Alec Walker

T D X Self-matching Proxies Randomization Intermediates Instruments

T D X Self-matching Proxies Randomization Intermediates Instruments

T D X Self-matching Proxies Randomization Intermediates Instruments When strong proxies for the possible confounding determinants of exposure indicate no effect on exposure, there is correspondingly strong evidence for an absence of confounding.

T D X Proxies Other, possibly unmeasured, non- confounding determinants of exposure are the sole determinants of treatment variation. UTUT

Where we’re going When different doctors give different treatments to similar patients, there must be a variety of opinions about therapy in the clinical community. The presence of differing therapeutic opinion has been termed “clinical equipoise” and is a permissive condition for conducting an ethical clinical trial. Clinical equipoise, identified in observational data as “empirical equipoise,” is also a permissive state for comparative effectiveness research.

Σ

Σ Provider

Σ

Σ Rx

Σ Provider ?

Σ ? Patient

?

? complaint

Σ ?

Σ ? training norms colleagues experience

Σ ? complaint symptoms training norms colleagues experience

Σ signs ? complaint symptoms training norms colleagues experience

Σ signs ? complaint symptoms history training norms colleagues experience

Σ signs ? complaint symptoms history illnesses training norms colleagues experience

Σ signs ? medications complaint symptoms history illnesses training norms colleagues experience

Σ signs ? medications complaint symptoms test results history illnesses training norms colleagues experience

Σ signs ? medications complaint symptoms test results history illnesses training norms colleagues experience

Σ signs ? medications complaint symptoms test results history illnesses training norms colleagues experience Rx A

Σ signs ? medications complaint symptoms test results history illnesses training norms colleagues experience Rx A Σ

Σ signs ? medications complaint symptoms test results history illnesses training norms colleagues experience Rx A Σ training norms colleagues experience

Σ signs ? medications complaint symptoms test results history illnesses training norms colleagues experience Rx A Σ training norms colleagues experience Rx B

Σ signs ? medications complaint symptoms test results history illnesses training norms colleagues experience Rx A Σ training norms colleagues experience Rx B Σ training norms colleagues experience Rx C

Σ signs ? medications complaint symptoms test results history illnesses training norms colleagues experience Rx A Σ training norms colleagues experience Rx B Σ training norms colleagues experience Rx C Σ training norms colleagues experience Rx D

Σ signs ? medications complaint symptoms test results history illnesses training norms colleagues experience Rx A Σ training norms colleagues experience Rx B Σ training norms colleagues experience Rx B Σ training norms colleagues experience Rx A

Σ signs ? medications complaint symptoms test results history illnesses training norms colleagues experience Rx A Σ training norms colleagues experience Rx B Σ training norms colleagues experience Rx B Σ training norms colleagues experience Rx A When the same patient has an equal chance of getting Rx A or Rx B, depending only on which doctor he happens to visit, the treating community is in EQUIPOISE.

“If a physician knows that these treatments are not equivalent, ethics requires that the superior treatment be recommended. “Following Fried, I call this state of uncertainty about the relative merits of A and B ‘equipoise.’ ” Reference is to Charles Fried. Medical Experimentation: Personal Integrity and Social Policy. Amsterdam: North-Holland Publishing, 1974 Benjamin Friedman. Equipoise and the ethics of clinical research. N Engl J Med 1987;317:

Threats to personal equipoise Studies that motivated the trial Early results of the trial itself Findings on secondary endpoints: QoL Change in experimenter’s understanding of – Relevant data external to the RCT – Views of other competent observers Benjamin Friedman. Equipoise and the ethics of clinical research. N Engl J Med 1987;317:

Conceal results from investigators through a Data and Safety Monitoring Committee Give the patient responsibility for valuing the relative merits of the treatments Informed Consent Frankly admit the social need for reliable health research. Medical Conscription Benjamin Friedman. Equipoise and the ethics of clinical research. N Engl J Med 1987;317:

Conceal results from investigators through a Data and Safety Monitoring Committee Give the patient responsibility for valuing the relative merits of the treatments Informed Consent Benjamin Friedman. Equipoise and the ethics of clinical research. N Engl J Med 1987;317:

Conceal results from investigators through a Data and Safety Monitoring Committee Give the patient responsibility for valuing the relative merits of the treatments Informed Consent Frankly admit the social need for reliable health research. Medical Conscription Benjamin Friedman. Equipoise and the ethics of clinical research. N Engl J Med 1987;317:

“The standard treatment is A, but some evidence suggests that B will be superior.” “Or there is a split in the clinical community, with some clinicians favoring A and others favoring B …” “… an honest, professional disagreement among expert clinicians …”

“The standard treatment is A, but some evidence suggests that B will be superior.” “Or there is a split in the clinical community, with some clinicians favoring A and others favoring B …” “… an honest, professional disagreement among expert clinicians …” Benjamin Friedman. Equipoise and the ethics of clinical research. N Engl J Med 1987;317: “At this point a state of ‘clinical equipoise’ exists … A state of clinical equipoise is consistent with a decided treatment preference on the part of the investigators.”

Benjamin Friedman. Equipoise and the ethics of clinical research. N Engl J Med 1987;317: “… there is a split in the clinical community ” “There is no consensus within the expert clinical community about the comparative merits…” “… clinical equipoise persists as long as [available] results are too weak to influence the judgement of the community of clinicians.” “As Fried has emphasized, competent (hence, ethical) medicine is social rather than individual in nature.” Reference is to Charles Fried. Medical Experimentation: Personal Integrity and Social Policy. Amsterdam: North-Holland Publishing, 1974

In the absence of knowledge of prescribers’ beliefs about alternative treatments, we might assume that each prescriber’s behavior reflects belief about best treatment for the individual patient, given the constraints of their shared environment. Σ signs ? medications complaint symptoms test results history illnesses training norms colleagues experience Rx A

Σ signs ? medications complaint symptoms test results history illnesses training norms colleagues experience Rx A Σ training norms colleagues experience Rx B Σ training norms colleagues experience Rx B Σ training norms colleagues experience Rx A When the same patient has an equal chance of getting Rx A or Rx B, depending only on which doctor he happens to visit, the treating community is in EMPIRICAL EQUIPOISE. A community of prescribers

Σ signs ? medications complaint symptoms test results history illnesses training norms colleagues experience Rx A Σ training norms colleagues experience Rx B Σ training norms colleagues experience Rx B Σ training norms colleagues experience Rx A We will take empirical equipoise in the population of prescribers as evidence of clinical equipoise in the prescriber community, in just the sense meant by Friedman. A community of prescribers

We can’t observe what different doctors would do with the same patient.

Σ training norms colleagues experience Rx A Σ training norms colleagues experience Σ training norms colleagues experience Σ training norms colleagues experience Rx ARx B In general, the patients who go to different doctors are not identical. Modeling the counterfactual

Σ training norms colleagues experience Rx A Σ training norms colleagues experience Σ training norms colleagues experience Σ training norms colleagues experience Rx ARx B But what if switching patients between doctors made no difference to the treatment assignment? Modeling the counterfactual

Σ training norms colleagues experience Rx A Σ training norms colleagues experience Σ training norms colleagues experience Σ training norms colleagues experience Rx ARx B We’d say that the treatment choices are probably most reflective of prescriber beliefs. Modeling the counterfactual

The patients can’t be switched between doctors in an observational study. Modeling the counterfactual

The patients can’t be switched between doctors in an observational study. Instead of asking whether the patient per se is determinative of doctor’s treatment choice, we can ask whether any observed patient characteristics tend to predict treatment. Modeling the counterfactual

Pr(A|A or B)=f(test results, medications, concurrent illnesses, history, signs, symptoms) Instead of asking whether the patient per se is determinative of doctor’s treatment choice, we can ask whether any observed patient characteristics tend to predict treatment. Modeling the counterfactual

Pr(A|A or B)=f(test results, medications, concurrent illnesses, history, signs, symptoms) If we estimate from the data whether any observed patient characteristics tend to predict treatment, we’ve calculated a propensity score. Propensity

Pr(A|A or B)=f(test results, medications, concurrent illnesses, history, signs, symptoms) If doctor preference is the major determinant of treatment choice, most fitted values from the propensity model estimating Pr(A) will fall near the overall prevalence of A in “A or B” population. Propensity

To make different A-B comparisons easier to compare, we’ve subtracted out the grand mean and renamed the result “Preference.” Preference F Preference S Propensity P Prevalence of A

Alternative treatments in Community Acquired Pneumonia Relative frequency Patients sorted by the preference score corresponding to their clinical and demographic characteristics

Alternative treatments in Community Acquired Pneumonia Relative frequency Patients sorted by the preference score corresponding to their clinical and demographic characteristics A B C D

Alternative treatments in Community Acquired Pneumonia Relative frequency Patients sorted by the preference score corresponding to their clinical and demographic characteristics E F G H

Alternative treatments in Community Acquired Pneumonia Relative frequency Patients sorted by the preference score corresponding to their clinical and demographic characteristics Empirical Equipoise Most people have covariate patterns that are similarly represented in the two treatment groups.

Alternative treatments in Community Acquired Pneumonia Relative frequency Patients sorted by the preference score corresponding to their clinical and demographic characteristics Empirical Equipoise Most people have covariate patterns that are similarly represented in the two treatment groups. A B C D

Alternative treatments in Community Acquired Pneumonia Relative frequency Patients sorted by the preference score corresponding to their clinical and demographic characteristics Patient-driven preferences Most people have covariate patterns that are differently represented in the two treatment groups.

Alternative treatments in Community Acquired Pneumonia Relative frequency Patients sorted by the preference score corresponding to their clinical and demographic characteristics Patient-driven preferences Most people have covariate patterns that are differently represented in the two treatment groups. E F G H

Empirical Equipoise

Empirical equipoise means that the prescribers appear to have diverse opinions. The patients themselves formed very similar groups, so differences in treatments are most likely ascribable to the doctors.

If one treatment has 28% fewer failures, the difference is large enough to be important. Empirical equipoise means that th prescribers appear to have diverse opinions.

Why should balance on observed variables imply balance on the unobserved?

One answer is narrative. The observed variables give a clue as to how prescribers respond to variations in patient characteristics. If we know that they respond little to medical facts that we can account for, we guess that they are insensitive to facts that we have not measured.

Why should balance on observed variables imply balance on the unobserved? Another answer is in the form of a syllogism: Different patients get different treatments. Patient characteristics in empirical equipoise are not determinative of drug choice. Possibly unmeasured factors external to the patient and to his/her prognosis are determinative of drug choice These unmeasured factors are Instruments, whose presence is the only justification for observational CER)

More plausible than “no unmeasured confounding” Treatment variation due to uncontrolled Instruments is much larger than Treatment variation due to uncontrolled Confounders

Where we’ve been When different doctors give different treatments to similar patients, there must be a variety of opinions about therapy in the clinical community. The presence of differing therapeutic opinion has been termed “clinical equipoise” and is a permissive condition for conducting an ethical clinical trial. Clinical equipoise, identified in observational data as “empirical equipoise,” is also a permissive state for comparative effectiveness research.

? medications complaint signs symptoms test results history illnesses A note of caution

? medications complaint signs symptoms test results history illnesses (x) (y) (z) A note of caution

? medications complaint signs symptoms test results history illnesses (x) (y) (z) A note of caution

? medications complaint signs symptoms test results history illnesses (x) (y) (z) ? medications complaint signs symptoms test results history illnesses (a) (b) (c) A note of caution

? medications complaint signs symptoms test results history illnesses (x) (y) (z) ? medications complaint signs symptoms test results history illnesses (a) (b) (c) A note of caution

? medications complaint signs symptoms test results history illnesses (x) (y) (z) ? medications complaint signs symptoms test results history illnesses (a) (b) (c) Because some patient characteristics known to the doctor may not be recorded in the database, Empirical Equipoise might be only Apparent. A note of caution

? medications complaint signs symptoms test results history illnesses (x) (y) (z) ? medications complaint signs symptoms test results history illnesses (a) (b) (c) Because some patient characteristics known to the doctor may not be recorded in the database, Empirical Equipoise might be only Apparent. ! A note of caution

? medications complaint signs symptoms test results history illnesses (x) (y) (z) ? medications complaint signs symptoms test results history illnesses (a) (b) (c) We’ve made arguments as to why the vectors of unmeasured covariates [x,y,z] and [a,b,c] ought to have similar distributions. ! But it is surely a good idea to worry, to check whenever you can, and to be modest. A note of caution