G. Biondi Zoccai – Ricerca in cardiologia RICERCA CLINICA IN CARDIOLOGIA INTERVENTISTICA (1/2) Percorso didattico basato sulla revisione ed interpretazione dei dati derivanti dalla letteratura scientifica Giuseppe Biondi Zoccai, Divisione di Cardiologia 1, Ospedale S. Giovanni Battista “Molinette”, Torino –

G. Biondi Zoccai – Ricerca in cardiologia Acknowledgements Guyatt G and Rennie D, editors Users’ Guide to the Medical Literature A Manual for Evidence-Based Clinical Practice 2002: Chicago, AMA Press

G. Biondi Zoccai – Ricerca in cardiologia What to expect? Core modules IntroductionIntroduction Finding out relevant literatureFinding out relevant literature General guidelines for literature appraisalGeneral guidelines for literature appraisal Abstract and Introduction appraisalAbstract and Introduction appraisal Methods and Results appraisal 1 - Patients and proceduresMethods and Results appraisal 1 - Patients and procedures Methods and Results appraisal 2 - Data collection/management and descriptive analysisMethods and Results appraisal 2 - Data collection/management and descriptive analysis Methods and Results appraisal 3 - Inferential analysisMethods and Results appraisal 3 - Inferential analysis Discussion and Conclusions appraisalDiscussion and Conclusions appraisal

G. Biondi Zoccai – Ricerca in cardiologia What to expect? Core modules IntroductionIntroduction Finding out relevant literatureFinding out relevant literature General guidelines for literature appraisalGeneral guidelines for literature appraisal Abstract and Introduction appraisalAbstract and Introduction appraisal Methods and Results appraisal 1 - Patients and proceduresMethods and Results appraisal 1 - Patients and procedures Methods and Results appraisal 2 - Data collection/management and descriptive analysisMethods and Results appraisal 2 - Data collection/management and descriptive analysis Methods and Results appraisal 3 - Inferential analysisMethods and Results appraisal 3 - Inferential analysis Discussion and Conclusions appraisalDiscussion and Conclusions appraisal

G. Biondi Zoccai – Ricerca in cardiologia Reasons for critically reading a paper A clinical research paper can be read for several reasons: -divertissement -comparing our own practice to those of the others -keeping up to date and discovering novel treatment or techniques techniques -keeping abreast of current research in order to design and conduct clinically meaningful and impactful studies conduct clinically meaningful and impactful studies…or… -just to criticize other colleagues’ work!

G. Biondi Zoccai – Ricerca in cardiologia Methods for “studying a study” Regardless of your aim, we suggest to follow this approach: -keep enjoying while reading -bring the paper wherever it suits you and make the habit of reading a paper common practice reading a paper common practice -maintain a global view and at the same time a focused, structured and analytic approach while reading a study structured and analytic approach while reading a study Remember, the more you will read the quicker and accomplished you will become in appraising a study… …indeed, critically reading papers is like PTCA!

G. Biondi Zoccai – Ricerca in cardiologia Keep enjoying while reading BMJ 2003

G. Biondi Zoccai – Ricerca in cardiologia Bring the paper wherever it suits you

G. Biondi Zoccai – Ricerca in cardiologia Maintain a global view as well as a…

G. Biondi Zoccai – Ricerca in cardiologia …focused, structured and analytic approach What is the age of the painting given the claclure? Who was Mona Lisa? Where was the portrait performed? Why is this painting worth so much? It seems as many other 15-16th century portraits! Was Leonardo in love with her?

G. Biondi Zoccai – Ricerca in cardiologia Example: SISR trial Global view Interesting! SES are better than brachytherapy for ISR!or Oh man! Another paper on SES! But we already knew that they work! And who’s using VBT anymore?

G. Biondi Zoccai – Ricerca in cardiologia Example: SISR trial Focused, structured and analytic approach

G. Biondi Zoccai – Ricerca in cardiologia Focused, structured and analytic approach BMJ’s critical appraisal questions: 1 - What is the paper about? 2 - Why was the study done? 3 - What type of study was done? 4 - Was it primary research (RCT, cohort, case-control, cross-sectional, series)? 5 - Was it secondary research (overview, systematic review, meta-analysis, decision analysis, guidelines development, economic analysis)? 6 - Was the design appropriate (for study on treatment, diagnosis, screening, prognosis, or causation)? 7 - Was the study ethical? 8 - Is the design right? A - Does this treatment work? → systematic review, RCT B - How good is a diagnostic test? → (prospective) cohort study C - Should we screen? → RCT D - What causes this disease? → RCT, cohort, case/control (rare diseases) E - What did people think or do? → cohort, cross-sectional survey, qualitative study

G. Biondi Zoccai – Ricerca in cardiologia Focused, structured and analytic approach BMJ’s critical appraisal questions: 1 - What is the paper about? SES vs VBT for BMS ISR 2 - Why was the study done? No RCT yet available comparing SES vs VBT for ISR 3 - What type of study was done? Open RCT 4 - Was it primary research? Yes 5 - Was it secondary research? NA 6 - Was the design appropriate (for study on treatment, diagnosis, screening, prognosis, or causation)? Yes 7 - Was the study ethical? Yes 8 - Is the design right? A - Does this treatment work? → Yes B - How good is a diagnostic test? → Na C - Should we screen? → Na D - What causes this disease? → Na E - What did people think or do? → Na

G. Biondi Zoccai – Ricerca in cardiologia Self-criticisism while “studying a study”

G. Biondi Zoccai – Ricerca in cardiologia Self-criticisism while “studying a study” If you thoroughly and critically read a study, you will be able to summarize it and present it to your colleagues, thus avoiding them the painful experience of reading it again! Schroter JAMA 2006

G. Biondi Zoccai – Ricerca in cardiologia Self-criticisism while “studying a study” GENERAL COMMENTS: The study by Holmes et al is an interesting report on an open RCT comparing SES vs VBT for BMS ISR in 384 patients MAJOR STRENGTHS AND WEAKNESSES: The paper has several strengths, including the novelty, the relatively high angiographic follow-up rate, and the adequate sample size. Among the weaknesses, we should note the inherent lack of blinding, the reliance on target vessel revascularization and target lesion revascularization endpoints that might be driven by the systematic angiographic follow-up, and the lack of truly long-term (>9 months) follow-up to definitively establish the safety of SES for ISR. ORIGINALITY OF THE WORK: The work is original as it is the first RCT to date comparing SES vs VBT for ISR. Another RCT has been recently presented showing the superiority of PES vs VBT for ISR (Stone JAMA 2006). Indeed, the seminal ISAR- DESIRE trial demonstrating that SES outperform PES and both of them outperform POBA for ISR should also be borne in mind (Kastrati JAMA 2005). COMMENTS ON THE AUTHORS’ INTERPRETATION OF THE STUDY FINDINGS: The authors’ interpretation of the results seems satisfactory, even if they probably dismiss the possibility that POBA or BMS implantation associated with systemic antiproliferative therapies might as well be effective. modified from Schroter JAMA 2006

G. Biondi Zoccai – Ricerca in cardiologia Self-criticisism while “studying a study” GENERAL COMMENTS: The study by Holmes et al is an interesting report on an open RCT comparing SES vs VBT for BMS ISR in 384 patients MAJOR STRENGTHS AND WEAKNESSES: The paper has several strengths, including the novelty, the relatively high angiographic follow-up rate, and the adequately sample size. Among the weaknesses, we should note the inherent lack of blinding, the reliance on target vessel revascularization and target lesion revascularization endpoints that might be driven by the systematic angiographic follow-up, and the lack of truly long-term (>9 months) follow-up to definitively establish the safety of SES for ISR. ORIGINALITY OF THE WORK: The work is original as it is the first RCT to date comparing SES vs VBT for ISR. Another RCT has been recently presented showing the superiority of PES vs VBT for ISR (Stone JAMA 2006). Indeed, the seminal ISAR- DESIRE trial demonstrating that SES outperform PES and both of them outperform POBA for ISR should also be borne in mind (Kastrati JAMA 2005). COMMENTS ON THE AUTHORS’ INTERPRETATION OF THE STUDY FINDINGS: The authors’ interpretation of the results seems satisfactory, even if they probably dismiss the possibility that POBA or BMS implantation associated with systemic antiproliferative therapies might as well be effective. modified from Schroter JAMA 2006

G. Biondi Zoccai – Ricerca in cardiologia Self-criticisism while “studying a study” GENERAL COMMENTS: The study by Holmes et al is an interesting report on an open RCT comparing SES vs VBT for BMS ISR in 384 patients MAJOR STRENGTHS AND WEAKNESSES: The paper has several strengths, including the novelty, the relatively high angiographic follow-up rate, and the adequately sample size. Among the weaknesses, we should note the inherent lack of blinding, the reliance on target vessel revascularization and target lesion revascularization endpoints that might be driven by the systematic angiographic follow-up, and the lack of truly long-term (>9 months) follow-up to definitively establish the safety of SES for ISR. ORIGINALITY OF THE WORK: The work is original as it is the first RCT to date comparing SES vs VBT for ISR. Another RCT has been recently presented showing the superiority of PES vs VBT for ISR (Stone JAMA 2006). Indeed, the seminal ISAR- DESIRE trial demonstrating that SES outperform PES and both of them outperform POBA for ISR should also be borne in mind (Kastrati JAMA 2005). COMMENTS ON THE AUTHORS’ INTERPRETATION OF THE STUDY FINDINGS: The authors’ interpretation of the results seems satisfactory, even if they probably dismiss the possibility that POBA or BMS implantation associated with systemic antiproliferative therapies might as well be effective. modified from Schroter JAMA 2006

G. Biondi Zoccai – Ricerca in cardiologia Other important concepts Goals of clinical researchGoals of clinical research Accuracy and precisionAccuracy and precision Random error and systematic error (ie bias)Random error and systematic error (ie bias) Internal and external validityInternal and external validity Alpha (type I) and beta (type II) errorsAlpha (type I) and beta (type II) errors P values and confidence intervalsP values and confidence intervals Sample size computation and statistical powerSample size computation and statistical power Evidence-based medicineEvidence-based medicine

G. Biondi Zoccai – Ricerca in cardiologia Other important concepts Goals of clinical researchGoals of clinical research Accuracy and precisionAccuracy and precision Random error and systematic error (ie bias)Random error and systematic error (ie bias) Internal and external validityInternal and external validity Alpha (type I) and beta (type II) errorsAlpha (type I) and beta (type II) errors P values and confidence intervalsP values and confidence intervals Sample size computation and statistical powerSample size computation and statistical power Evidence-based medicineEvidence-based medicine

G. Biondi Zoccai – Ricerca in cardiologia Ultimate goal: appraisal of causation

G. Biondi Zoccai – Ricerca in cardiologia Other important concepts Goals of clinical researchGoals of clinical research Accuracy and precisionAccuracy and precision Random error and systematic error (ie bias)Random error and systematic error (ie bias) Internal and external validityInternal and external validity Alpha (type I) and beta (type II) errorsAlpha (type I) and beta (type II) errors P values and confidence intervalsP values and confidence intervals Sample size computation and statistical powerSample size computation and statistical power Evidence-based medicineEvidence-based medicine

G. Biondi Zoccai – Ricerca in cardiologia Accuracy and precision Accuracy measures the distance from the true value distance Precision measures the spead in the measurements spread true value measurement

G. Biondi Zoccai – Ricerca in cardiologia Accuracy and precision

G. Biondi Zoccai – Ricerca in cardiologia Accuracy and precision example

G. Biondi Zoccai – Ricerca in cardiologia Accuracy and precision example Schultz et al, Am Heart J 2004

G. Biondi Zoccai – Ricerca in cardiologia Other important concepts Goals of clinical researchGoals of clinical research Accuracy and precisionAccuracy and precision Random error and systematic error (ie bias)Random error and systematic error (ie bias) Internal and external validityInternal and external validity Alpha (type I) and beta (type II) errorsAlpha (type I) and beta (type II) errors P values and confidence intervalsP values and confidence intervals Sample size computation and statistical powerSample size computation and statistical power Evidence-based medicineEvidence-based medicine

G. Biondi Zoccai – Ricerca in cardiologia Random and systematic errors Thus Precision expresses the extent of RANDOM ERROR Accuracy expresses the extent of SYSTEMATIC ERROR (ie bias)

G. Biondi Zoccai – Ricerca in cardiologia Bias Bias is a systematic DEVIATION from the TRUTH -in itself it cannot be ever recognized -there is a need for external gold standard and/or permanent surveillance

G. Biondi Zoccai – Ricerca in cardiologia An incomplete list of bias Simplest classification: 1. Selection bias 1. Selection bias 2. Information bias 2. Information bias · Selection bias · Information bias · Confounders · Observation bias · Investigator’s bias (enthusiasm bias) · Patient’s background bias · Distribution of pathological changes bias · Selection bias · Small sample size bias · Reporting bias · Referral bias · Variation bias · Recall bias · Statistical bias · Selection bias · Confounding · Intervention bias · Measurement or information · Interpretation bias · Publication bias · Subject selection/sampling bias Sackett, J Chronic Dis 1979

G. Biondi Zoccai – Ricerca in cardiologia Selection bias

G. Biondi Zoccai – Ricerca in cardiologia Information bias

G. Biondi Zoccai – Ricerca in cardiologia Other important concepts Goals of clinical researchGoals of clinical research Accuracy and precisionAccuracy and precision Random error and systematic error (ie bias)Random error and systematic error (ie bias) Internal and external validityInternal and external validity Alpha (type I) and beta (type II) errorsAlpha (type I) and beta (type II) errors P values and confidence intervalsP values and confidence intervals Sample size computation and statistical powerSample size computation and statistical power Evidence-based medicineEvidence-based medicine

G. Biondi Zoccai – Ricerca in cardiologia Validity Internal validity entails both PRECISION and ACCURACY (ie does a study provide a truthful answer to the research question?) External validity expresses the extent to which the results can be applied to other contexts and settings. It corresponds to the distinction between SAMPLE and POPULATION)

G. Biondi Zoccai – Ricerca in cardiologia Validity

Validity Example biondi

G. Biondi Zoccai – Ricerca in cardiologia Validity

Validity One-year events in the ARTS trials Event ARTS II, n=607 (%) ARTS I CABG, n=602 (%) ARTS I PCI, n=600 (%) All-cause mortality CVE MI CABG* PCI* Any MACCE *First or repeat. CVE=cerebrovascular event. MACCE=major adverse cardiac and cerebrovascular events Serruys et al, ACC 2005

G. Biondi Zoccai – Ricerca in cardiologia Validity

Validity

Validity Hill et al, Eur Heart J 2004

G. Biondi Zoccai – Ricerca in cardiologia Validity

Validity Rothwell, Lancet 2005

G. Biondi Zoccai – Ricerca in cardiologia Other important concepts Goals of clinical researchGoals of clinical research Accuracy and precisionAccuracy and precision Random error and systematic error (ie bias)Random error and systematic error (ie bias) Internal and external validityInternal and external validity Alpha (type I) and beta (type II) errorsAlpha (type I) and beta (type II) errors P values and confidence intervalsP values and confidence intervals Sample size computation and statistical powerSample size computation and statistical power Evidence-based medicineEvidence-based medicine

G. Biondi Zoccai – Ricerca in cardiologia Alpha and type I error Whenever I perform a test, there is thus a risk of a FALSE POSITIVE result, ie REJECTING A TRUE null hypothesis This error is called type I, is measured as alpha and its unit is the p value The lower the p value, the lower the risk of falling into a type I error (ie the HIGHER the SPECIFICITY of the test)

G. Biondi Zoccai – Ricerca in cardiologia Alpha and type I error Type I error is like a MIRAGE Because I see something that does NOT exist that does NOT exist

G. Biondi Zoccai – Ricerca in cardiologia Beta and type II error Whenever I perform a test, there is also a risk of a FALSE NEGATIVE result, ie NOT REJECTING A FALSE null hypothesis This error is called type II, is measured as beta and its unit is a probability The complementary of beta is called power The lower the beta, the lower the risk of missing a true difference (ie the HIGHER the SENSITIVITY of the test)

G. Biondi Zoccai – Ricerca in cardiologia Type II error is like being BLIND Because I do NOT see something that exists Beta and type II error

G. Biondi Zoccai – Ricerca in cardiologia Summary of errors Experimental study H 0 accepted H 0 rejected Truth H 0 true Type I error H 0 false Type II error

G. Biondi Zoccai – Ricerca in cardiologia Type I error Pitt et al, Lancet 1997

G. Biondi Zoccai – Ricerca in cardiologia Type I error Pitt et al, Lancet 2000

G. Biondi Zoccai – Ricerca in cardiologia Type II error

G. Biondi Zoccai – Ricerca in cardiologia Type II error Kastrati JAMA 2005

G. Biondi Zoccai – Ricerca in cardiologia An example of small sample size To aim for an 80% power with a 4 vs 6% event rate you need at least 3700 patients!

G. Biondi Zoccai – Ricerca in cardiologia Other important concepts Goals of clinical researchGoals of clinical research Accuracy and precisionAccuracy and precision Random error and systematic error (ie bias)Random error and systematic error (ie bias) Internal and external validityInternal and external validity Alpha (type I) and beta (type II) errorsAlpha (type I) and beta (type II) errors P values and confidence intervalsP values and confidence intervals Sample size computation and statistical powerSample size computation and statistical power Evidence-based medicineEvidence-based medicine

G. Biondi Zoccai – Ricerca in cardiologia Descriptive statistics AVERAGE

G. Biondi Zoccai – Ricerca in cardiologia Inferential statistics If I become a scaffolder, how likely I am to eat well every day? Pvalues ConfidenceIntervals

G. Biondi Zoccai – Ricerca in cardiologia Inferential statistics P values tell you whether there is a DIFFERENCE and its DIRECTION DIFFERENCE and its DIRECTION Confidence intervals tell you what is the MAGNITUDE (or SIZE) of such difference

G. Biondi Zoccai – Ricerca in cardiologia 2.1 vs 2.4% Difference and direction 0.3 vs 2.8%

G. Biondi Zoccai – Ricerca in cardiologia Size of the difference 7.2 vs 2.4%

G. Biondi Zoccai – Ricerca in cardiologia Other important concepts Goals of clinical researchGoals of clinical research Accuracy and precisionAccuracy and precision Random error and systematic error (ie bias)Random error and systematic error (ie bias) Internal and external validityInternal and external validity Alpha (type I) and beta (type II) errorsAlpha (type I) and beta (type II) errors P values and confidence intervalsP values and confidence intervals Sample size computation and statistical powerSample size computation and statistical power Evidence-based medicineEvidence-based medicine

G. Biondi Zoccai – Ricerca in cardiologia Power and sample size Whenever designing a study or analyzing a dataset, it is important to estimate the sample size or the power of the comparison SAMPLE SIZE Setting a specific alpha and a specific beta, you calculate the necessary sample size given the average inter-group difference and its variation POWER Given a specific sample size and alpha, in light of the calculated average inter-group difference and its variation, you obtain an estimate of the power (ie 1-beta)

G. Biondi Zoccai – Ricerca in cardiologia Sample size calculation Ardissino et al, JAMA 2004 To compute the sample size for a study we thus need: 1. Preferred alpha value 2. Preferred beta value 3. Control event rate or average value (with measure of dispersion if appliable) 4. Expected relative reduction in experimental group

G. Biondi Zoccai – Ricerca in cardiologia Other important concepts Goals of clinical researchGoals of clinical research Accuracy and precisionAccuracy and precision Random error and systematic error (ie bias)Random error and systematic error (ie bias) Internal and external validityInternal and external validity Alpha (type I) and beta (type II) errorsAlpha (type I) and beta (type II) errors P values and confidence intervalsP values and confidence intervals Sample size computation and statistical powerSample size computation and statistical power Evidence-based medicineEvidence-based medicine

G. Biondi Zoccai – Ricerca in cardiologia Evidence-based medicine Excerpt from a 1990 leaflet for internal medicine resident at McMaster University (Hamilton, Canada): “…goal of evidence-based medicine is to be aware of the evidence on which one’s practice is based, the soundness of the evidence, and the strength of inference the evidence permits. The strategy employed requires a clear delineation of the relevant question(s); a thorough search of the literature relating to questions; a critical appraisal of the evidence, and its applicability to the clinical situation; and a balanced application of the conclusions to the clinical problem.” Guyatt and Rennie, Users’ guide to the medical literature, 2002

G. Biondi Zoccai – Ricerca in cardiologia Evidence-based medicine Definition: The coscientious, explicit, and judicious use of current best evidence in making decisions about the care of individual patients. The practice of evidence-based medicine requires integration of individual clinical expertise and patient preferences with the best available external clinical evidence from systematic search. Guyatt and Rennie, Users’ guide to the medical literature, 2002

G. Biondi Zoccai – Ricerca in cardiologia Rebuttal to critics of EBM Evidence alone is never sufficient to make a clinical decision: decision makers must always trade the benefits and risks, inconvenience, and costs associated with alternative management strategies, and in doing so consider the patient’s values! Guyatt and Rennie, Users’ guide to the medical literature, 2002

G. Biondi Zoccai – Ricerca in cardiologia EBM hierarchy of evidence 1.N of 1 randomized controlled trial 2.Systematic reviews of randomized trials 3.Single randomized trial 4.Systematic review of observational studies addressing patient-important outcomes 5.Single observational study addressing patient-important outcomes 6.Physiologic studies (eg blood pressure, cardiac output, exercise capacity, bone density, and so forth) 7.Unsystematic clinical observations Guyatt and Rennie, Users’ guide to the medical literature, 2002

G. Biondi Zoccai – Ricerca in cardiologia Parallel hierarchy of CV research Biondi-Zoccai, Ital Heart J 2003 Qualitative reviews Systematic reviews Meta-analyses from individual studies Meta-analyses from individual patient data Case reports and series Observational studies Observational controlled studies Randomized controlled trials Multicenter randomized controlled trials

G. Biondi Zoccai – Ricerca in cardiologia Take home messages The most important questions to remember when reading a clinical research paper are:

G. Biondi Zoccai – Ricerca in cardiologia Take home messages The most important questions to remember when reading a clinical research paper are: 1 – does the study provide an accurate and precise answer to the clinical issue it claims to address? answer to the clinical issue it claims to address?

G. Biondi Zoccai – Ricerca in cardiologia Take home messages The most important questions to remember when reading a clinical research paper are: 1 – does the study provide an accurate and precise answer to the clinical issue it claims to address? answer to the clinical issue it claims to address? 2 – are its conclusions worth of real-life application?