How EBM brings the connection between evidence and measurement into focus. Benjamin Smart

1. Aims and Argument Summary

Aims Outline how, although (arguably) all measurements are theory laden, different categories of evidence in the EBM hierarchy are theory laden in different ways, and to different extents. Show that EBM brings the theory ladenness of measurement into focus, since degrees of ‘observational theory ladenness’ determines the structure of its hierarchy.

Theory Laden Observation Kuhn (1962) argued that observation is theory laden in a number of different ways, including: Saliency: investigators focus on/observe different things depending on what they’re looking for. Some qualities are attended to, and others are ignored (depending on what paradigm they are working in). Perceptual theory loading: Conceptual limitations may result in investigators having different perceptual experiences (paradigm-dependent).

The Evidence in Evidence Based Medicine EBM is a mode of clinical practice that involves “integrating individual clinical expertise with the best available external clinical evidence from systematic research” (Sacket et al, 1996) EBM calls for “careful clinical judgement in evaluating the best available evidence” (Feinstein et al, 1997) The evidence used in evidence based practice often comprises the results of studies (providing morbidity, mortality, and quality of life stats). These statistics are calculated by analysing measurement outcomes, so they, too, are theory laden ((i) in virtue of involving observation, and (ii)in virtue of how the data is organised).

The Evidence in Evidence Based Medicine The key difference between EBM and previous strategies in clinical practice, is what is taken to be “the best available evidence”. Some evidence derives from measurement outcomes, other evidence does not. The evidence hierarchy RCTs Observational studies Mechanistic/pathophysiological reasoning Clinical experience/intuition

Evidence and Measurement in EBM

Pathophysiological Reasoning and Clinical experience Qualitative evidence Using intuition and experience as evidence. Evidence, but it’s ranked less highly in EBM than the quantitative measurement outcomes. Bias is inevitable. What is considered salient by one clinician might not be investigated by another (Kuhn 1962), and whether a disease marker is sufficient for treatment may differ depending on the patient. E.g. Green et al. discovered black patients half as likely to receive treatment (for heart disease) from the same clinicians as white patients, despite the same markers.

Observational Studies Cohort Studies Quantitative measurement outcomes Follow a population “Relate information on risk factor patterns and health states at baseline, to the outcome of interest” (Bhopal 2016) Which risk factors do we measure at baseline?

Observational Studies Case-control Quantitative measurement outcomes “Look for differences and similarities between a series of cases and a control group” (Bhopal 2016) How does one choose the participants in the case and control groups (from within the target population)? Try to rule out known confounders

Theory-ladenness in EBM

Theory-laden Evidence At least 2 different forms: Observational theory ladenness: Saliency, Perceptual Theory Loading etc. results in bias, especially when just a single clinician. Mechanistic theory ladenness: Theory ladenness through study design. Often in an attempt to reduce observational theory ladenness Hypothesis: The hierarchy is fixed by degrees of observational theory ladenness

Theory-laden Evidence The background assumptions made by the investigators play a major role in the conclusions drawn from the study. Mechanistic theory ladenness is explicit in the study design (indeed, it is necessary). Due to the residual observational theory ladenness of measurement in observational studies, the evidential base they provide is fragile But… at least observational studies involve a larger sample, and fixed ‘rules’. Systematic reviews will use lots of studies, with lots of investigators, hopefully reducing the problems with saliency and perceptual theory loading.

Eliminating Theory-Ladenness? Randomised Controlled Trials Where possible, to minimise the observational theory ladenness of effect measurement in epidemiological studies, randomisation is used. As a result, in theory, the two groups should provide the same outcomes were neither subject to the exposure, so any difference after trials can be attributed to the exposure.

Randomised Controlled Trials On the face of it, although RCTs are still mechanistically theory laden, randomisation eliminates some of the observational theory ladenness in observational studies. The evidence provided by RCTs is more robust, and to be viewed as ‘the gold standard’.

Conclusions

Conclusions Any clinical decision based on pathophysiologic is hugely laden with ‘observational theory’. Causal inferences grounded by observational studies are both observationally and mechanistically theory laden. They do, however, provide a better evidential base than pathophysiologic reasoning, since the effect measures provided are less observationally theory laden than experience and intuition.

Conclusions As Kuhn notes, arguably all measurements are theory laden. EBM, however, highlights that theory ladenness comes in different degrees, and different forms. The less observationally theory laden, the better the evidence.

Thank you