Can we teach critical appraisal ‘the GATE approach’ in 30 minutes! Rod Jackson University of Auckland, NZ September 2010

GATE: a Graphic Appraisal Tool for Epidemiology 1 picture, 2 formulas & 3 acronyms

I was inspired by my experience learning the krebs cycle – one of the few things I remembered from medical school, – and unfortunately not a very useful thing – to develop the krebs cycle of epidemiology. What made it memorable was that it was taught primarily as a picture and as Confucius is meant to have said – a picture is worth 1000 words.

A picture: the GATE frame © the shape of every epidemiological study

The PECOT acronym: the 5 parts of every epidemiological study Participants Exposure Group Comparison Group E C Outcomes O Time T All epidemiological studies can be hung on the GATE frame

2 formulas: study analyses Occurrence of disease = Numerator ÷ Denominator ÷ Time 2. Random error (95% Confidence Interval) = 1.96 x Standard Error

The RAMMbo* acronym: assessing study bias P Recruitment Allocation Maintenance Measurement of outcomes blind or objective P E C O T E C Cover RAMMbo when I introduce critical appraisal O T * Paul Glasziou

GATE frame picture & PECOT acronym Participants Exposure Group Comparison Group E C Outcomes O Time T Describe a study’s design by hanging PECOT on the frame

Participants Study Setting Eligible Participants P Participants

DVT in long-haul flights. Fill in for DVT in long-haul flights. Lancet 2001; 357:1485-9 Page 95 in Glasziou et al

Participants DVT in long-haul flights: Lancet 2001;357:1485-9 P Study Setting: volunteers, UK, ? 1990s Eligible Participants: no previous DVT, > 50 yrs, planned economy air travel 2 sectors > 8 hours P Participants: 200, mean age 61-62 years

Exposure & Comparison Groups Exposure or Intervention Group (EG) Comparison or Control Group (CG) EG CG

DVT in long-haul flights. Fill in for DVT in long-haul flights. Lancet 2001; 357:1485-9 Page 95 in Glasziou et al

Exposure & Comparison Groups 115 116 Exposure or Intervention Group (EG): Below knee compression stockings Comparison or Control Group (CG): no stockings 100 100

Outcomes (O) a b c d yes no ‘Dis-ease’ O Outcomes (O)

DVT in long-haul flights. Fill in for DVT in long-haul flights. Lancet 2001; 357:1485-9 Page 95 in Glasziou et al

Outcomes (O) 100 100 a= 0 b= 12 c d yes O Outcomes (O) DVT no

Time (T) incidence T prevalence

DVT in long-haul flights. Fill in for DVT in long-haul flights. Lancet 2001; 357:1485-9 Page 95 in Glasziou et al

T= from take-off to 2 days post final flight Time (T) Outcome: e.g. death incidence 0 0 T= from take-off to 2 days post final flight

Time (T) prevalence Outcome: number with DVTs 0 12 0 12 prevalence T = at post-flight assessment (<48 hrs)

Study design: GATE frame & PECOT Participants P Exposure Group Comparison Group E C Outcomes O Time T Every epidemiological study hangs on the GATE frame

DVT in long-haul flights: Lancet 2001;357:1485-9 Setting: UK volunteers – 479 considered Eligible: > 50 yrs, no DVT, flying > 8 hrs Participants 231 115 116 Exposure Group: stockings Comparison Group: no stockings 100 100 0 12 Outcomes: DVT Time: at post flight assess

2 formulas: study analyses Occurrence of disease = Numerator ÷ Denominator ÷ Time 2. Random error (95% Confidence Interval) = 1.96 x Standard Error

O = N÷D D Denominator (Participants) Numerator (Outcomes) All epidemiological studies involve measuring the OCCURRENCE of disease Occurrence = Numerator ÷ Denominator Denominator (Participants) D O = N÷D N Numerator (Outcomes)

Describe a study’s analyses by hanging the numbers on the frame GATE study analyses P Overall Denominator Denominator 1: Denominator 2: EG CG Exposure Group (EG) Comparison Group (CG) Numerator 1: a Numerator 2: b a b c d O Describe a study’s analyses by hanging the numbers on the frame

Occurrence = N ÷ D P O Denominator 1: Exposure Group EG Denominator 2: Comparison Group CG EG CG Numerator 1: a Numerator 2: b a b c d O Exposure Group Occurrence: EGO = a ÷ EG Comparison Group Occurrence: CGO = b ÷ CG

Calculate EGO & CGO for the outcome ‘DVT’ post long-haul flight Denominator 1: Exposure Group EG = 100 Numerator 1: a = 3* Denominator 2: Comparison Group CG = 100 Numerator 2: b = 12 EGO = 3/100 people at post flight assessment CGO = 12/100= people at post flight assessment * a = 0 but have used 3 to illustrate calculations on next slide EG CG a b c d O

Describing differences between occurrences Relative difference or Relative Risk = EGO ÷ CGO Absolute Difference or Risk Difference = EGO - CGO Number Needed To Treat (NNT) = 1 ÷ RD

DVT in long-haul flights. Fill in for DVT in long-haul flights. Lancet 2001; 357:1485-9 Page 95 in Glasziou et al

Describing differences between occurrences Relative difference or Relative Risk = EGO ÷ CGO Absolute Difference or Risk Difference = EGO - CGO Number Needed To Treat (NNT) = 1 ÷ RD = 3/100 ÷ 12/100 = 3/12 = 0.25 = 3/100 - 12/100 = - 9/12 = - 7.5 /100 = 1 ÷ (- 7.5 /100) = - 100/7.5 = 13.3

Analyses it’s all about EGO & CGO

The RAMMbo acronym: assessing study bias P Recruitment Allocation Maintenance Measurement of outcomes blind or objective P E C O T E C Cover RAMMbo when I introduce critical appraisal O T

appropriate Recruitment processes? Study setting RAMMbo Eligible people appropriate Recruitment processes? P P Study setting & eligibility criteria well described? e.g. Recruit random/representative sample OR consecutive eligibles OR volunteers from advertisements Participants representative of eligibles? Prognostic/risk profile appropriate to study question? E C O T

appropriate Recruitment processes? Study setting RAMMbo Eligible people appropriate Recruitment processes? P P Study setting & eligibility criteria well described? Recruited volunteers from advertisements; eligibility criteria well described Participants representative of eligibles? 479 considered, 248 excluded - probably Prognostic/risk profile appropriate to study question? yes E C O T

RAMMbo: A is for Allocation P P RCT: Allocate randomly by investigators (e.g drugs) Cohort: Allocate by measurement (e.g. smoking) were EG & CG similar at baseline? EG CG EG CG O O T T

RAMMbo: A is for Allocation P RCT: Allocate randomly by investigators using sealed envelopes EG & CG similar at baseline except more women in stocking group EG CG O T

did most participants remain in allocated groups (EG & CG) RAMMbo P good Maintenance? did most participants remain in allocated groups (EG & CG) EG CG Participants &/or investigators blind to exposure (and comparison exposure)? Compliance high & similar in EG & CG? Contamination low & similar in EG & CG? Co-interventions low & similar in EG & CG? Completeness of follow-up high & similar in EG & CG? O T

did most participants remain in allocated groups (EG & CG) RAMMbo P good Maintenance? did most participants remain in allocated groups (EG & CG) 115 116 100 100 Participants &/or investigators blind to exposure (and comparison exposure)? no Compliance high & similar in EG & CG? dk* Contamination low & similar in EG & CG? dk Co-interventions low & similar in EG & CG? dk Completeness of follow-up high & similar in EG & CG? probably ok, lost < 15% * dk = don’t know O T

Measurement of outcomes RAMMbo P Measurement of outcomes blind or objective? EG CG If outcome measurements not Objective (eg. automated or definitive) were investigators blind to exposure (and comparison exposure) O T

Measurement of outcomes RAMMbo P Measurement of outcomes blind or objective? EG CG Outcome measurements reasonably objective (ultra-sound) Technician was blind to exposure (although some participants may have had a stocking line) O T

The 4 (GATE) study biases P Recruitment bias Allocation bias E C Maintenance bias Measurement (of outcomes) bias O T

Measurement (of outcomes) ok DVT in long-haul flights: Lancet 2001;357:1485-9 P Recruitment ok Allocation ok E C Maintenance ok Measurement (of outcomes) ok O T

2 formulas: study analyses Occurrence of disease = Numerator ÷ Denominator ÷ Time 2. Random error (95% Confidence Interval) = 1.96 x Standard Error

2 formulas: study analyses Occurrence of disease = Numerator ÷ Denominator ÷ Time 2. Random error (95% Confidence Interval) = 1.96 x Standard Error EGO = 0%; 95% CI = 0% - 3% CGO = 10%; 95% CI = 4.8% - 16%

Excel CATs & paper Gate-lites There is a GATE for every study design www.epiq.co.nz

the 3rd acronym: SRs & meta-analyses Find appropriate studies? Appraise selected studies? Include only valid studies? Total-up (synthesise) appropriately? Heterogeneity adequated addressed?

In the 19th century we made great advances in health through the provision of clean, clear water; in the 21st century we will make the same advances through clean, clear (systematically reviewed) information. Muir Gray Systematic reviews are one of the greatest ideas of modern thought. They should be celebrated. Ben Goldacre

Muhammad Shafique Sajid et al Muhammad Shafique Sajid et al. Knee-length graduated compression stockings for thromboprophylaxis in air travellers: A meta-analysis. Int J Angiol. 2008; 17: 119–123. Nine trials studying participants using KL stockings were analyzed. Forty-six of 1261 participants randomly assigned to the control group developed deep vein thrombosis (DVT), compared with two of 1237 participants (0.16%) in the KL stockings group. …..There was an absolute difference of 3.4% in the incidence of DVT, in favour of KL stockings. The number needed to treat with KL stockings to avoid one case of DVT was 29.4. However, there was significant heterogeneity among trials. The RR for DVT was 0.08 in high-risk participants and 0.14 in low- to medium-risk participants.

GATE can also be used to frame the first 4 steps of EBP

The 4 steps of EBP 1. Ask a focused question. 2. Access (systematically search for) epidemiological evidence to help answer question. 3. Appraise evidence found for its validity, effect size, precision (ideally all the relevant evidence) 4. Apply the evidence: a. amalgamate the valid evidence with other relevant information (patient/community values, clinical/health issues, & policy context) to make a good decision; and b. Act (implement) the decision in practice

the PECOT acronym: the 5 parts of every epidemiological study Participants Exposure Group Comparison Group E C Outcomes O Time T All epidemiological studies can be hung on the GATE frame

EBP Step 1: ASK - turn your question into 5 parts (PECOT) Participants (the patient problem) Exposure (e.g. a therapy) Comparison (there is always an alternative! - another therapy or no treatment… Outcome (e.g. a disease you want to prevent or manage) Time frame (over which you expect a result)

EBP Step 2: ACCESS the evidence – use PECOT to identify search terms Participants (the patient problem) Exposure (e.g. a therapy) Comparison (there is always an alternative! - another therapy or no treatment… Outcome (e.g. a disease you want to prevent or manage) Time frame (over which you expect a result)

EBP Step 3: APPRAISE the evidence – using PECOT & RAMMbo Recruitment Allocation Maintenance Measurement of outcomes blind or objective P E C O T E C Cover RAMMbo when I introduce critical appraisal O T

EBP Step 4: APPLY the evidence by: a EBP Step 4: APPLY the evidence by: a. AMALGAMATING the relevant information & making an evidence-based decision:’ the X-factor ©

X-factor: making evidence-based decisions Epidemiologic evidence Clinical / health considerations Patient / community preferences Policy issues expertise: ‘putting it all together’ the art of practice

Step 4b ACT

Step 5: EBP 360° = EBP + Quality Improvement Ask a focused question. Access (systematically search for) epidemiological evidence to help answer question. 3. Appraise evidence AND then meta-analyse (systematically review) ALL relevant valid evidence. 4. Apply the best evidence: amalgamate the valid evidence with other relevant information to make a good decision; and ACT on your decision 5. AUDIT your practice (i.e. check your actual practice – ‘actions’ - against ‘best’ evidence-based practice) = EBP + Quality Improvement