1. Introduction to Critical Appraisal
Yulia Lin, MD, FRCPC
Transfusion Medicine Specialist, Dept of Clinical Pathology
Sunnybrook Health Sciences Centre
University of Toronto
TMR Journal Club, September 4, 2008

2. Why is Critical Appraisal important?

3. Vast amount of literature
there is a vast amount of literature out there

4. New Studies in Transfusion
– there are the new studies in transfusion medicine being published all the time

5. Past Studies in Transfusion
New Studies in
Transfusion
- There are all of the old studies that were published before we even knew that we were going to be specialists in transfusion medicine
Past Studies in Transfusion

6. Past Studies in Transfusion
New Studies in
Transfusion
And because TM specialists are consultants to physicians in many other different subspecialties, there are studies in other fields that we haven’t traditionally been familiar with.
Studies in other
Sub-specialties
Past Studies in Transfusion

7. Why critical appraisal is important?
Impossible to keep up with all of the literature
Enables us to distinguish stronger evidence from weaker evidence
Allows us to appropriately incorporate the evidence into our practice so as to improve patient care
It is impossible to keep up with all of the literature – there just isn’t enough time in the day. And I won’t spend too much time on this occasion discussing how to keep up with the literature.
But this is where critical appraisal comes in – when you’re presented with a study, critical appraisal is a tool that enables us to distinguish good evidence from bad evidence. It allows us to hopefully appropriately incorporate the existing evidence into our daily practice so as to improve patient care
Unfortunately, though we talk about the importance of EBM, there isn’t a whole lot of evidence for EBM. It would be difficult to randomized to practising EBM vs. not as well as unethical. But we do know that adherence to guidelines at least in some fields (i.e. beta-blockers in MI) does improve outcome.

8. Medical Expert Scholar
Apply principles to evaluate quality of research publications
Scholar
Critically appraise sources of medical information
Understand the concepts of clinical research design
Demonstrate knowledge of basic statistics and epidemiology
And of course, critical appraisal is mentioned a number of times in the royal college objectives.

9. Purpose of TMR Journal Club
To present a recent article in the transfusion literature
To learn how to critically appraise the medical literature
The purpose of TMR journal club is not only to present a recent article in the tranfusion literature so as to help us all keep up to date with the literature but more important to learn how to go through the process of critical appraisal

10. Steps to a successful JC
Choosing the article
Presenting the article
Critically appraising the article
And so in my mind, 3 steps to a successful JC are:

11. Choosing an article… Where to find an article?
Recent table of contents
NEJM, Lancet, JAMA
Transfusion, Vox Sanguinis, Blood, etc.
Transfusion Medicine Reviews – Journal Club
Transfusion Medicine Community
List of interesting articles

12. Choosing the article… Which article? What type of article? Relevant
Interesting to you
What type of article?
Any article
But some are easier than others…

13. Type of articles Randomized controlled trials – Therapy
Observational studies – Harm
Systematic Reviews
Guidelines
Diagnosis
Prognosis

14. First, a review…

15. Randomized Controlled Trial
Intervention A
Population
selection
Outcomes
measured
Randomization
Intervention B

16. Randomized Controlled Trial
Intervention A
Population
Outcomes
measured
Randomization
Intervention B
Is aprotinin superior to lysine analogs in
high-risk cardiac surgery?
Fergusson DA et al. NEJM May 29, 2008.

17. Observational Case-Control
Exposed
Exposure to
Risk Factor
Population
Cases
(+Disease)
Yes No
Control
(No disease)
But unfortunately, it’s not feasible to conduct a RCT for every intervention. In some cases, the intervention would be unethical (eg. transfusion) Especially in cases of rare diseases or events, one can look to observational case-control studies.
Yes No

18. Observational Case-Control
Exposed
Exposure to
Risk Factor
Population
Cases
(+Disease)
Yes No
Control
(No disease)
Yes No
Retrospective

19. Observational Case-Control
Exposed
Exposure to
Risk Factor
Population
Cases
(+Disease)
Yes No
Control
(No disease)
Yes No
Blood transfusion, anesthesia, surgery and risk of NHL
in a population-based case-control study.
Cerhan JR et al. Int J Cancer 2008 Aug 15.

20. Observational Cohort Cohort without Exposure to Disease Disease
Population
Cohort without
Disease
Exposure to
Risk Factor
Disease
Yes No
Cohort
Exposed
Yes No
A stronger observational study is the cohort study. Attempting to ensure that the participants are the same at the beginning of the study is challenging.
Not exposed

21. Cohort Studies Past Present Future Cohort Follow-up assembled
Retrospective
Cohort Study

22. Cohort Studies Past Present Future Cohort Follow-up assembled
Retrospective
Cohort Study
The ratio of fibrinogen to red cells transfused affects survival in
casualties receiving massive transfusions
at an army combat support hospital.
Stinger HK et al., J Trauma February 2008;64:S79-85.
Red blood cell to plasma ratios transfused during massive
transfusion are associated with mortality in severe multiply
Injury: a retrospective analysis from Deutsche Gesellschaft.
Maegele M et al. Vox Sang August 2008;95:

23. Cohort Studies Past Present Future Cohort Follow-up assembled
Retrospective
Cohort Study
Cohort
assembled
Follow-up
Prospective
Cohort Study

24. Cohort Studies Past Present Future Cohort Follow-up assembled
Retrospective
Cohort Study
Cohort
assembled
Follow-up
Prospective
Cohort Study
Prospective evaluation of a transfusion policy of
D+ red blood cells into D- patients.
Gonzalez-Porras JR et al. Transfusion July 2008.

25. Systematic Review Overview Systematic review Meta-analysis
Summary of the medical literature that attempts to address a focused clinical question
Systematic review
Using methods designed to reduce the likelihood of bias
Meta-analysis
Review that uses quantitative methods to summarize the results

26. Systematic Review Systematic review Meta-analysis
Using methods designed to reduce the likelihood of bias
Meta-analysis
Review that uses quantitative methods to summarize the results
Comparing the efficacy and safety of apheresis and
whole-blood derived platelet transfusions: a systematic review.
Heddle NM et al. Transfusion July 2008.

27. Guidelines
“Systematically developed statements to assist practitioner and patient decisions about appropriate health care for specific clinical circumstances”
Attempt to address all issues and values relevant to a clinical decision
Attempt to distill a large body of medical expertise into a convenient, readily usable format
Make explicit recommendations with definite intent to influence what clinicians do
Field MJ. Clinical practice guidelines. 1990
Hayward et al. JAMA 1995;274:570-4
Cook et al. Ann Int Med 1997;127:210-6

28. Guidelines
“Systematically developed statements to assist practitioner and patient decisions about appropriate health care for specific clinical circumstances”
Guidelines for policies on alternatives to allogeneic blood transfusion:
Predeposit autologous blood donation and transfusion
Boulton FE et al. Transfusion Medicine 2007;17:354-65

29. How to do a critical appraisal?

30. Critical appraisal methods
Most are based on Users’ Guides to the Medical Literature
Available in a textbook form (2nd ed) but also on-line at
Centre for Health Evidence Website:
For guidelines, different instruments
AGREE instrument:
GRADE instrument available at Blood Depot

34. Steps to Critical Appraisal
Are the results valid?
Have the results been influenced in a systematic fashion so as to lead to a false conclusion?
What are the results?
How can I apply the results to patient care?

35. RCT Intervention A Population Outcomes Randomization measured
Intervention B

36. RCT - Are the results valid?
Intervention A
Population
Outcomes
measured
Randomization
Intervention B
Did experimental & control groups begin the study with a similar prognosis?
Were patients randomized?
Was randomization concealed?
Were patients analyzed in the groups to which they were randomized?
Were groups similar with respect to known prognostic variables?

37. RCT - Are the results valid?
Intervention A
Population
Outcomes
measured
Randomization
Intervention B
Did experimental & control groups retain a similar prognosis after the
study started?
Were patients, clinicians, outcome assessors blinded?
Was follow-up complete?

38. RCT - What were the results?
Intervention A
Population
Outcomes
measured
Randomization
Intervention B
How large was the treatment effect?
How precise was the estimate of the treatment effect?

39. RCT - How can I apply the results to patient care?
Intervention A
Population
Outcomes
measured
Randomization
Intervention B
Were the study patients similar to the patient in my practice?

40. RCT - How can I apply the results to patient care?
Intervention A
Population
Outcomes
measured
Randomization
Intervention B
Were the study patients similar to the patient in my practice?
Were all clinically important outcomes considered?
Are the likely treatment benefits worth the potential harm and costs?

41. Observational Cohort Cohort without Exposure to Disease Disease
Population
Cohort without
Disease
Exposure to
Risk Factor
Disease
Yes No
Cohort
Exposed
Yes No
Not exposed

42. Cohort - Are the results valid?
Population
Cohort without
Disease
Exposure to
Risk Factor
Disease
Yes No
Exposed
Cohort
Yes No
Not exposed
Did expt and control groups begin the study with a similar prognosis?
Did the investigators demonstrate similarity in all known
determinants of outcome? Did they adjust for differences
in the analysis?

43. Cohort - Are the results valid?
Population
Cohort without
Disease
Exposure to
Risk Factor
Disease
Yes No
Exposed
Cohort
Yes No
Not exposed
Did expt and control groups retain a similar prognosis after the study
started?
Were the outcomes measured in the same way in the groups being
compared?
Was follow-up sufficiently complete?

44. Cohort - What are the results?
Population
Cohort without
Disease
Exposure to
Risk Factor
Disease
Yes No
Exposed
Cohort
Yes No
Not exposed
How strong is the association between exposure and outcome?
Consider magnitude and dose response?
How precise is the estimate of the risk?

45. Cohort - How can I apply the results?
Population
Cohort without
Disease
Exposure to
Risk Factor
Disease
Yes No
Exposed
Cohort
Yes No
Not exposed
Were the study patients similar to my patients?

46. Cohort - How can I apply the results?
Population
Cohort without
Disease
Exposure to
Risk Factor
Disease
Yes No
Exposed
Cohort
Yes No
Not exposed
Were the study patients similar to my patients?
Was the duration of follow-up adequate?
Was was the magnitude of the risk?
Should I attempt to stop the exposure?

47. Putting it all together…

48. Tips for TMR Journal Club
Choose the article carefully
Present the article briefly (10 slides)
The main objective is the critical appraisal
Internal validity:
Focus on 1-2 key points
Try explaining a new concept (bias, statistical concept)
External validity: Take a stand
Allow ½ the time for discussion