1. Can we teach critical appraisal ‘the GATE approach’
in 30 minutes!
Rod Jackson
University of Auckland, NZ
September 2010

3. GATE: a Graphic Appraisal Tool for Epidemiology
1 picture, 2 formulas & 3 acronyms

4. I was inspired by my experience learning the krebs cycle – one of the few things I remembered from medical school, – and unfortunately not a very useful thing – to develop the krebs cycle of epidemiology. What made it memorable was that it was taught primarily as a picture and as Confucius is meant to have said – a picture is worth 1000 words.

5. A picture: the GATE frame
the shape of every epidemiological study

6. The PECOT acronym: the 5 parts of every epidemiological study
Participants
Exposure Group
Comparison Group
E C
Outcomes O
Time T
All epidemiological studies can be hung on the GATE frame

7. 2 formulas: study analyses
Occurrence of disease
= Numerator ÷ Denominator ÷ Time
2. Random error (95% Confidence Interval)
= 1.96 x Standard Error

8. The RAMMbo* acronym: assessing study biasP
Recruitment
Allocation
Maintenance
Measurement of outcomes blind or objective P E C O T E C
Cover RAMMbo when I introduce critical appraisal O T
* Paul Glasziou

9. GATE frame picture & PECOT acronym
Participants
Exposure Group
Comparison Group
E C
Outcomes O
Time T
Describe a study’s design by hanging PECOT on the frame

10. Participants
Study Setting
Eligible Participants P
Participants

11. DVT in long-haul flights.
Fill in for
DVT in long-haul flights.
Lancet 2001; 357:1485-9
Page 95 in Glasziou et al

12. Participants DVT in long-haul flights: Lancet 2001;357:1485-9 P
Study Setting: volunteers, UK, ? 1990s
Eligible Participants: no previous DVT, > 50 yrs, planned economy air travel 2 sectors > 8 hours P
Participants: 200, mean age years

13. Exposure & Comparison Groups
Exposure or Intervention Group (EG)
Comparison or Control Group (CG) EG CG

14. DVT in long-haul flights.
Fill in for
DVT in long-haul flights.
Lancet 2001; 357:1485-9
Page 95 in Glasziou et al

15. Exposure & Comparison Groups
115
116
Exposure or Intervention Group (EG):
Below knee compression stockings
Comparison or Control Group (CG):
no stockings
100
100

16. Outcomes (O) a b c d
yes no
‘Dis-ease’ O
Outcomes (O)

17. DVT in long-haul flights.
Fill in for
DVT in long-haul flights.
Lancet 2001; 357:1485-9
Page 95 in Glasziou et al

18. Outcomes (O)
100
100
a= 0
b= 12 c d
yes O
Outcomes (O)
DVT no

19. Time (T)
incidence T
prevalence

20. DVT in long-haul flights.
Fill in for
DVT in long-haul flights.
Lancet 2001; 357:1485-9
Page 95 in Glasziou et al

21. T= from take-off to 2 days post final flight
Time (T)
Outcome: e.g. death
incidence
T= from take-off to 2 days post final flight

22. Time (T) prevalence Outcome: number with DVTs 0 12
prevalence
T = at post-flight assessment (<48 hrs)

23. Study design: GATE frame & PECOT
Participants P
Exposure Group
Comparison Group
E C
Outcomes O
Time T
Every epidemiological study hangs on the GATE frame

24. DVT in long-haul flights: Lancet 2001;357:1485-9
Setting: UK volunteers – 479 considered
Eligible: > 50 yrs, no DVT, flying > 8 hrs
Participants
231
Exposure Group:
stockings
Comparison Group:
no stockings
Outcomes:
DVT
Time:
at post flight assess

25. 2 formulas: study analyses
Occurrence of disease
= Numerator ÷ Denominator ÷ Time
2. Random error (95% Confidence Interval)
= 1.96 x Standard Error

26. O = N÷D D Denominator (Participants) Numerator (Outcomes)
All epidemiological studies involve measuring the OCCURRENCE of disease Occurrence = Numerator ÷ Denominator
Denominator (Participants) D
O = N÷D N
Numerator (Outcomes)

27. Describe a study’s analyses by hanging the numbers on the frame
GATE study analyses P
Overall Denominator
Denominator 1:
Denominator 2:
EG CG
Exposure Group (EG)
Comparison Group (CG)
Numerator 1: a
Numerator 2: b a b c d O
Describe a study’s analyses by hanging the numbers on the frame

28. Occurrence = N ÷ D P O Denominator 1: Exposure Group EG Denominator 2:
Comparison Group CG
EG CG
Numerator 1: a
Numerator 2: b a b c d O
Exposure Group Occurrence:
EGO = a ÷ EG
Comparison Group Occurrence:
CGO = b ÷ CG

29. Calculate EGO & CGO for the outcome ‘DVT’ post long-haul flight
Denominator 1:
Exposure Group
EG = 100
Numerator 1:
a = 3*
Denominator 2:
Comparison Group
CG = 100
Numerator 2:
b = 12
EGO = 3/100 people at post flight assessment
CGO = 12/100= people at post flight assessment
* a = 0 but have used 3 to illustrate calculations on next slide
EG CG a b c d O

30. Describing differences between occurrences
Relative difference or Relative Risk = EGO ÷ CGO
Absolute Difference or Risk Difference = EGO - CGO
Number Needed To Treat (NNT) = 1 ÷ RD

31. DVT in long-haul flights.
Fill in for
DVT in long-haul flights.
Lancet 2001; 357:1485-9
Page 95 in Glasziou et al

32. Describing differences between occurrences
Relative difference or Relative Risk = EGO ÷ CGO
Absolute Difference or Risk Difference = EGO - CGO
Number Needed To Treat (NNT) = 1 ÷ RD
= 3/100 ÷ 12/100 = 3/12 = 0.25
= 3/ /100 = - 9/12 = /100
= 1 ÷ (- 7.5 /100) = - 100/7.5 = 13.3

33. Analyses
it’s all about EGO & CGO

34. The RAMMbo acronym: assessing study biasP
Recruitment
Allocation
Maintenance
Measurement of outcomes blind or objective P E C O T E C
Cover RAMMbo when I introduce critical appraisal O T

35. appropriate Recruitment processes?
Study setting
RAMMbo
Eligible people
appropriate Recruitment processes? P P
Study setting & eligibility criteria well described?
e.g. Recruit random/representative sample OR consecutive eligibles OR volunteers from advertisements
Participants representative of eligibles?
Prognostic/risk profile appropriate to study question?
E C O T

36. appropriate Recruitment processes?
Study setting
RAMMbo
Eligible people
appropriate Recruitment processes? P P
Study setting & eligibility criteria well described?
Recruited volunteers from advertisements; eligibility criteria well described
Participants representative of eligibles?
479 considered, 248 excluded - probably
Prognostic/risk profile appropriate to study question? yes
E C O T

37. RAMMbo: A is for AllocationP P
RCT: Allocate randomly by investigators (e.g drugs)
Cohort: Allocate by measurement (e.g. smoking)
were EG & CG similar at baseline?
EG CG
EG CG O O T T

38. RAMMbo: A is for AllocationP
RCT: Allocate randomly by investigators using sealed envelopes
EG & CG similar at baseline except more women in stocking group
EG CG O T

39. did most participants remain in allocated groups (EG & CG)
RAMMbo P
good Maintenance?
did most participants remain in allocated groups (EG & CG)
EG CG
Participants &/or investigators blind to exposure (and comparison exposure)?
Compliance high & similar in EG & CG?
Contamination low & similar in EG & CG?
Co-interventions low & similar in EG & CG?
Completeness of follow-up high & similar in EG & CG? O T

40. did most participants remain in allocated groups (EG & CG)
RAMMbo P
good Maintenance?
did most participants remain in allocated groups (EG & CG)
115
116
Participants &/or investigators blind to exposure (and comparison exposure)? no
Compliance high & similar in EG & CG? dk*
Contamination low & similar in EG & CG? dk
Co-interventions low & similar in EG & CG? dk
Completeness of follow-up high & similar in EG & CG? probably ok, lost < 15%
* dk = don’t know O T

41. Measurement of outcomes
RAMMbo P
Measurement of outcomes
blind or objective?
EG CG
If outcome measurements not Objective (eg. automated or definitive) were investigators blind to exposure (and comparison exposure) O T

42. Measurement of outcomes
RAMMbo P
Measurement of outcomes
blind or objective?
EG CG
Outcome measurements reasonably objective (ultra-sound)
Technician was blind to exposure (although some participants may have had a stocking line) O T

43. The 4 (GATE) study biasesP
Recruitment bias
Allocation bias
E C
Maintenance bias
Measurement (of outcomes) bias O T

44. Measurement (of outcomes) ok
DVT in long-haul flights: Lancet 2001;357:1485-9 P
Recruitment ok
Allocation ok
E C
Maintenance ok
Measurement (of outcomes) ok O T

45. 2 formulas: study analyses
Occurrence of disease
= Numerator ÷ Denominator ÷ Time
2. Random error (95% Confidence Interval)
= 1.96 x Standard Error

46. 2 formulas: study analyses
Occurrence of disease
= Numerator ÷ Denominator ÷ Time
2. Random error (95% Confidence Interval)
= 1.96 x Standard Error
EGO = 0%; 95% CI = 0% - 3%
CGO = 10%; 95% CI = 4.8% - 16%

47. Excel CATs & paper Gate-lites
There is a GATE for every study design

48. the 3rd acronym: SRs & meta-analyses
Find appropriate studies?
Appraise selected studies?
Include only valid studies?
Total-up (synthesise) appropriately?
Heterogeneity adequated addressed?

49. In the 19th century we made great advances in health through the provision of clean, clear water; in the 21st century we will make the same advances through clean, clear (systematically reviewed) information.
Muir Gray
Systematic reviews are one of the greatest ideas of modern thought. They should be celebrated.
Ben Goldacre

50. Muhammad Shafique Sajid et al
Muhammad Shafique Sajid et al. Knee-length graduated compression stockings for thromboprophylaxis in air travellers: A meta-analysis. Int J Angiol. 2008; 17: 119–123.
Nine trials studying participants using KL stockings were analyzed. Forty-six of 1261 participants randomly assigned to the control group developed deep vein thrombosis (DVT), compared with two of 1237 participants (0.16%) in the KL stockings group. …..There was an absolute difference of 3.4% in the incidence of DVT, in favour of KL stockings. The number needed to treat with KL stockings to avoid one case of DVT was However, there was significant heterogeneity among trials. The RR for DVT was 0.08 in high-risk participants and 0.14 in low- to medium-risk participants.

51. GATE can also be used to frame the first 4 steps of EBP

52. The 4 steps of EBP 1. Ask a focused question.
2. Access (systematically search for) epidemiological evidence to help answer question.
3. Appraise evidence found for its validity, effect size, precision (ideally all the relevant evidence)
4. Apply the evidence:
a. amalgamate the valid evidence with other relevant information (patient/community values, clinical/health issues, & policy context) to make a good decision; and
b. Act (implement) the decision in practice

53. the PECOT acronym: the 5 parts of every epidemiological study
Participants
Exposure Group
Comparison Group
E C
Outcomes O
Time T
All epidemiological studies can be hung on the GATE frame

54. EBP Step 1: ASK - turn your question into 5 parts (PECOT)
Participants (the patient problem)
Exposure (e.g. a therapy)
Comparison (there is always an alternative! - another therapy or no treatment…
Outcome (e.g. a disease you want to prevent or manage)
Time frame (over which you expect a result)

55. EBP Step 2: ACCESS the evidence – use PECOT to identify search terms
Participants (the patient problem)
Exposure (e.g. a therapy)
Comparison (there is always an alternative! - another therapy or no treatment…
Outcome (e.g. a disease you want to prevent or manage)
Time frame (over which you expect a result)

56. EBP Step 3: APPRAISE the evidence – using PECOT & RAMMbo
Recruitment
Allocation
Maintenance
Measurement of outcomes blind or objective P E C O T E C
Cover RAMMbo when I introduce critical appraisal O T

58. EBP Step 4: APPLY the evidence by: a
EBP Step 4: APPLY the evidence by: a. AMALGAMATING the relevant information & making an evidence-based decision:’ the X-factor

59. X-factor: making evidence-based decisions
Epidemiologic evidence
Clinical / health considerations
Patient / community preferences
Policy issues
expertise: ‘putting it all together’ the art of practice

60. Step 4b
ACT

61. Step 5: EBP 360° = EBP + Quality Improvement Ask a focused question.
Access (systematically search for) epidemiological evidence to help answer question.
3. Appraise evidence AND then meta-analyse (systematically review) ALL relevant valid evidence.
4. Apply the best evidence:
amalgamate the valid evidence with other relevant information to make a good decision; and
ACT on your decision
5. AUDIT your practice (i.e. check your actual practice – ‘actions’ - against ‘best’ evidence-based practice)
= EBP + Quality Improvement