Introduction to Critical Appraisal Yulia Lin, MD, FRCPC Transfusion Medicine Specialist, Dept of Clinical Pathology Sunnybrook Health Sciences Centre University of Toronto TMR Journal Club, September 4, 2008

Why is Critical Appraisal important?

Vast amount of literature there is a vast amount of literature out there

New Studies in Transfusion – there are the new studies in transfusion medicine being published all the time

Past Studies in Transfusion New Studies in Transfusion - There are all of the old studies that were published before we even knew that we were going to be specialists in transfusion medicine Past Studies in Transfusion

Past Studies in Transfusion New Studies in Transfusion And because TM specialists are consultants to physicians in many other different subspecialties, there are studies in other fields that we haven’t traditionally been familiar with. Studies in other Sub-specialties Past Studies in Transfusion

Why critical appraisal is important? Impossible to keep up with all of the literature Enables us to distinguish stronger evidence from weaker evidence Allows us to appropriately incorporate the evidence into our practice so as to improve patient care It is impossible to keep up with all of the literature – there just isn’t enough time in the day. And I won’t spend too much time on this occasion discussing how to keep up with the literature. But this is where critical appraisal comes in – when you’re presented with a study, critical appraisal is a tool that enables us to distinguish good evidence from bad evidence. It allows us to hopefully appropriately incorporate the existing evidence into our daily practice so as to improve patient care Unfortunately, though we talk about the importance of EBM, there isn’t a whole lot of evidence for EBM. It would be difficult to randomized to practising EBM vs. not as well as unethical. But we do know that adherence to guidelines at least in some fields (i.e. beta-blockers in MI) does improve outcome.

Medical Expert Scholar Apply principles to evaluate quality of research publications Scholar Critically appraise sources of medical information Understand the concepts of clinical research design Demonstrate knowledge of basic statistics and epidemiology And of course, critical appraisal is mentioned a number of times in the royal college objectives.

Purpose of TMR Journal Club To present a recent article in the transfusion literature To learn how to critically appraise the medical literature The purpose of TMR journal club is not only to present a recent article in the tranfusion literature so as to help us all keep up to date with the literature but more important to learn how to go through the process of critical appraisal

Steps to a successful JC Choosing the article Presenting the article Critically appraising the article And so in my mind, 3 steps to a successful JC are:

Choosing an article… Where to find an article? Recent table of contents NEJM, Lancet, JAMA Transfusion, Vox Sanguinis, Blood, etc. Transfusion Medicine Reviews – Journal Club Transfusion Medicine Community List of interesting articles

Choosing the article… Which article? What type of article? Relevant Interesting to you What type of article? Any article But some are easier than others…

Type of articles Randomized controlled trials – Therapy Observational studies – Harm Systematic Reviews Guidelines Diagnosis Prognosis

First, a review…

Randomized Controlled Trial Intervention A Population selection Outcomes measured Randomization Intervention B

Randomized Controlled Trial Intervention A Population Outcomes measured Randomization Intervention B Is aprotinin superior to lysine analogs in high-risk cardiac surgery? Fergusson DA et al. NEJM May 29, 2008.

Observational Case-Control Exposed Exposure to Risk Factor Population Cases (+Disease) Yes No Control (No disease) But unfortunately, it’s not feasible to conduct a RCT for every intervention. In some cases, the intervention would be unethical (eg. transfusion) Especially in cases of rare diseases or events, one can look to observational case-control studies. Yes No

Observational Case-Control Exposed Exposure to Risk Factor Population Cases (+Disease) Yes No Control (No disease) Yes No Retrospective

Observational Case-Control Exposed Exposure to Risk Factor Population Cases (+Disease) Yes No Control (No disease) Yes No Blood transfusion, anesthesia, surgery and risk of NHL in a population-based case-control study. Cerhan JR et al. Int J Cancer 2008 Aug 15.

Observational Cohort Cohort without Exposure to Disease Disease Population Cohort without Disease Exposure to Risk Factor Disease Yes No Cohort Exposed Yes No A stronger observational study is the cohort study. Attempting to ensure that the participants are the same at the beginning of the study is challenging. Not exposed

Cohort Studies Past Present Future Cohort Follow-up assembled Retrospective Cohort Study

Cohort Studies Past Present Future Cohort Follow-up assembled Retrospective Cohort Study The ratio of fibrinogen to red cells transfused affects survival in casualties receiving massive transfusions at an army combat support hospital. Stinger HK et al., J Trauma February 2008;64:S79-85. Red blood cell to plasma ratios transfused during massive transfusion are associated with mortality in severe multiply Injury: a retrospective analysis from Deutsche Gesellschaft. Maegele M et al. Vox Sang August 2008;95:112-119.

Cohort Studies Past Present Future Cohort Follow-up assembled Retrospective Cohort Study Cohort assembled Follow-up Prospective Cohort Study

Cohort Studies Past Present Future Cohort Follow-up assembled Retrospective Cohort Study Cohort assembled Follow-up Prospective Cohort Study Prospective evaluation of a transfusion policy of D+ red blood cells into D- patients. Gonzalez-Porras JR et al. Transfusion July 2008.

Systematic Review Overview Systematic review Meta-analysis Summary of the medical literature that attempts to address a focused clinical question Systematic review Using methods designed to reduce the likelihood of bias Meta-analysis Review that uses quantitative methods to summarize the results

Systematic Review Systematic review Meta-analysis Using methods designed to reduce the likelihood of bias Meta-analysis Review that uses quantitative methods to summarize the results Comparing the efficacy and safety of apheresis and whole-blood derived platelet transfusions: a systematic review. Heddle NM et al. Transfusion July 2008.

Guidelines “Systematically developed statements to assist practitioner and patient decisions about appropriate health care for specific clinical circumstances” Attempt to address all issues and values relevant to a clinical decision Attempt to distill a large body of medical expertise into a convenient, readily usable format Make explicit recommendations with definite intent to influence what clinicians do Field MJ. Clinical practice guidelines. 1990 Hayward et al. JAMA 1995;274:570-4 Cook et al. Ann Int Med 1997;127:210-6

Guidelines “Systematically developed statements to assist practitioner and patient decisions about appropriate health care for specific clinical circumstances” Guidelines for policies on alternatives to allogeneic blood transfusion: Predeposit autologous blood donation and transfusion Boulton FE et al. Transfusion Medicine 2007;17:354-65

How to do a critical appraisal?

Critical appraisal methods Most are based on Users’ Guides to the Medical Literature Available in a textbook form (2nd ed) but also on-line at http://ugi.usersguides.org/usersguides/hg/hh_start.asp Centre for Health Evidence Website: http://www.cche.net/usersguides/main.asp For guidelines, different instruments AGREE instrument: http://www.agreetrust.org/instrument.htm GRADE instrument available at Blood Depot

Steps to Critical Appraisal Are the results valid? Have the results been influenced in a systematic fashion so as to lead to a false conclusion? What are the results? How can I apply the results to patient care?

RCT Intervention A Population Outcomes Randomization measured Intervention B

RCT - Are the results valid? Intervention A Population Outcomes measured Randomization Intervention B Did experimental & control groups begin the study with a similar prognosis? Were patients randomized? Was randomization concealed? Were patients analyzed in the groups to which they were randomized? Were groups similar with respect to known prognostic variables?

RCT - Are the results valid? Intervention A Population Outcomes measured Randomization Intervention B Did experimental & control groups retain a similar prognosis after the study started? Were patients, clinicians, outcome assessors blinded? Was follow-up complete?

RCT - What were the results? Intervention A Population Outcomes measured Randomization Intervention B How large was the treatment effect? How precise was the estimate of the treatment effect?

RCT - How can I apply the results to patient care? Intervention A Population Outcomes measured Randomization Intervention B Were the study patients similar to the patient in my practice?

RCT - How can I apply the results to patient care? Intervention A Population Outcomes measured Randomization Intervention B Were the study patients similar to the patient in my practice? Were all clinically important outcomes considered? Are the likely treatment benefits worth the potential harm and costs?

Observational Cohort Cohort without Exposure to Disease Disease Population Cohort without Disease Exposure to Risk Factor Disease Yes No Cohort Exposed Yes No Not exposed

Cohort - Are the results valid? Population Cohort without Disease Exposure to Risk Factor Disease Yes No Exposed Cohort Yes No Not exposed Did expt and control groups begin the study with a similar prognosis? Did the investigators demonstrate similarity in all known determinants of outcome? Did they adjust for differences in the analysis?

Cohort - Are the results valid? Population Cohort without Disease Exposure to Risk Factor Disease Yes No Exposed Cohort Yes No Not exposed Did expt and control groups retain a similar prognosis after the study started? Were the outcomes measured in the same way in the groups being compared? Was follow-up sufficiently complete?

Cohort - What are the results? Population Cohort without Disease Exposure to Risk Factor Disease Yes No Exposed Cohort Yes No Not exposed How strong is the association between exposure and outcome? Consider magnitude and dose response? How precise is the estimate of the risk?

Cohort - How can I apply the results? Population Cohort without Disease Exposure to Risk Factor Disease Yes No Exposed Cohort Yes No Not exposed Were the study patients similar to my patients?

Cohort - How can I apply the results? Population Cohort without Disease Exposure to Risk Factor Disease Yes No Exposed Cohort Yes No Not exposed Were the study patients similar to my patients? Was the duration of follow-up adequate? Was was the magnitude of the risk? Should I attempt to stop the exposure?

Putting it all together…

Tips for TMR Journal Club Choose the article carefully Present the article briefly (10 slides) The main objective is the critical appraisal Internal validity: Focus on 1-2 key points Try explaining a new concept (bias, statistical concept) External validity: Take a stand Allow ½ the time for discussion