Anwar Showail

 Critical appraisal is the systematic evaluation of clinical research papers in order to establish: 1. Does this study address a clearly focused question? 2. Did the study use a valid methods to address this question? 3. Are the valid results of this study important? 4. Are these valid, important results applicable to my patient or population?

 If the answer to any of these questions is “no”, you can save yourself the trouble of reading the rest of it.

 Therapy / RCT  Systematic Review  Diagnosis  Prognosis

I. Relevance II. Validity III. Results IV. Applicability

1. Relevance  The topic addresses a common problem in my practice  POEM outcomes  Results (if valid) will change my practice

VALIDITY (RABI) Therapy Systematic Review DiagnosisPrognosis R Randomization Criteria (Inclusion and exclusion - PICOTT) Reproducibility Reference standard Representative spectrum Representative sample (Homogeneous patients with respect prognostic risk) A Allocation (concealed) Attrition Articles search Articles selection Articles abstraction Ascertained (Verification) Adjustment of prognostic factors B Blindness (3Cs) Blind Articles quality assessment Blind Comparison Blind (Outcome assessor should e blind to prognostic factors – outcome need to be Objective not subjective) I Intention to treatInconsistency (Heterogeneity) Independent test not part of Gold Standard

 Randomization:  Each patient has an equal (50%) chance to be in control or intervention group (Best computer generated randomization list).  Also expressed as: Sequence generated

 Concealed Allocation (assignment):  No one from research team or patients know where next patient will be allocated.  Best is remote central call. ▪ Other ways: ▪ Serially numbered opaque sealed envelope ▪ computerized with protected folder.  Also expressed as: Random allocation / Assignment

 Proper randomization and concealment prevent SELECTION BIAS  Attrition Bias

 Blinding:  Research team and patient are unaware whether patients are receiving the intervention therapy or control therapy.

 PLEASE: Do Not confuse Blinding with Allocation concealment

 Five groups should be blind: 1. Patients 2. Clinicians 3. Data collectors 4. Outcome assessors 5. Data analyst

 And the 3 Cs : 1. Co-intervention: Any extra intervention other than study treatment (either to control or intervention). 2. Contamination: Any member from one group who received intervention from other group. 3. Compliance: Adherence of participants to the intervention.

 Effective blinding and the 3 Cs (Co- intervention, Contamination and Compliance ) prevent PERFORMANCE BIAS

 Intention to treat principle:  The calculation of the outcome according to the initial involved number of study’s subjects ( i.e. : including non-adherent to treatment and drop out).  Once randomized should be analyzed ▪ These patients should e followed up for outcome occurrence.

 Another weak way Called: Per protocol analysis : Calculation of outcome after excluding the drop out.  The drop out should be < 20 % to maintain power of the study.

 Composite outcome: multiple outcomes combined together, the earliest one appears, will stop looking for other outcomes for that particular patient.  It is easier magnify the result, increase number of events, but not reflecting the real situation.

 Post hoc test: means further testing after the experiment has concluded for patterns not specified before.  The more statistical tests are done post hoc, the more chance researchers will find effective results.

 Types of bias in therapy: 1. Selection bias : if randomization or allocation was not proper. 2. Performance bias: no blinding, there is contamination or co intervention. 3. Attrition (drop out): lost follow up. 4. Measurement (detection) bias: outcome assessor not blind to risk factor when measuring outcome.

RESULTS TherapySystematic Review DiagnosisPrognosis Magnitude and Precision ARR & NNT RR,RRR and OR Pooled results: Forest Plot (RR, OR) Accuracy of the test: Sn & Sp PPV,NPV LR SnOUT SpPIN Kaplan Meier curve: -1 st year survival rate -5 yr survival rate -Median survival

APPLICAILITY TherapySystematic Review DiagnosisPrognosis Can the results be applied to my patients care? IPPP -Intervention -Patient-similar -Preference -Potential harm IPPPO -Intervention -Patient-similar -Preference -Potential harm -All outcomes considered TPPP -Test -Patient similar -Preference -Care for Patient PF -Patient similar -Follow up sufficient

VALIDITY (RABI) Therapy Systematic Review DiagnosisPrognosis R Randomization Criteria (Inclusion and exclusion - PICOTT) Reproducibility Reference standard Representative spectrum Representative sample (Homogeneous patients with respect prognostic risk) A Allocation (concealed) Attrition Articles search Articles selection Articles abstraction Ascertained (Verification) Adjustment of prognostic factors B Blindness (3Cs) Blind Articles quality assessment Blind Comparison Blind (Outcome assessor should e blind to prognostic factors – outcome need to be Objective not subjective) I Intention to treatInconsistency (Heterogeneity) Independent test not part of Gold Standard

2. Validity  Representative sample (criteria) ▪ PICOTT ▪ P: population ▪ I: Intervention ▪ C: Comparison ▪ O: Outcome ▪ T: Type & Quality of included study ▪ T: Time of F/U (of Study) ▪ Inclusion and exclusion

 Reproducibility (The way of process described in a sufficient way to allow reproducibility)  Article (Search / Selection / Appraisal)  Blind (quality assessment of included studies)  Inconsistency (Heterogeneity)

1) Electronic database (pubmed, medline, chocrane, embase) 2) No language restriction 3) Reference of reference 4) Conference proceedings 5) Grey literature and unpublished articles 6) Contact drug company / expert 7) Hand search on none cited articles

 At least 2 independent reviewers for quality assessment of all studies included  Can be expressed as: ▪ Risk of bias ▪ Critically evaluated ▪ We assessed / we evaluated  Can be done by using Cochrane criteria (ROB) or JADAD score

 Disagreement should be resolved (inter- observer agreement) by: 1. Third party 2. Consensus 3. Kappa test

 Interpretation of Kappa Kappa %Agreement < 40Poor Moderate 61 – 80Substantial > 80Almost perfect

RESULTS TherapySystematic Review DiagnosisPrognosis Magnitude and Precision ARR & NNT RR,RRR and OR Pooled results: Forest Plot (RR, OR) Accuracy of the test: Sn & Sp PPV,NPV LR SnOUT SpPIN Kaplan Meier curve: -1 st year survival rate -5 yr survival rate -Median survival

APPLICAILITY TherapySystematic Review DiagnosisPrognosis Can the results be applied to my patients care? IPPP -Intervention -Patient-similar -Preference -Potential harm IPPPO -Intervention -Patient-similar -Preference -Potential harm -All outcomes considered TPPP -Test -Patient similar -Preference -Care for Patient PF -Patient similar -Follow up sufficient

 To compare between the new test and (index) && gold standard (Reference).

VALIDITY (RABI) Therapy Systematic Review DiagnosisPrognosis R Randomization Criteria (Inclusion and exclusion - PICOTT) Reproducibility Reference standard Representative spectrum Representative sample (Homogeneous patients with respect prognostic risk) A Allocation (concealed) Attrition Articles search Articles selection Articles abstraction Ascertained (Verification) Adjustment of prognostic factors B Blindness (3Cs) Blind Articles quality assessment Blind Comparison Blind (Outcome assessor should e blind to prognostic factors – outcome need to be Objective not subjective) I Intention to treatInconsistency (Heterogeneity) Independent test not part of Gold Standard

 Validity:  R ▪ Reference standard “gold” ; (acceptable, validated and practical) ▪ Representative – spectrum (sample); all levels of disease, care ;etc. ▪ Inclusion & exclusion criteria

▪ Replication (to be conducted by other people in different setting) ▪ Reproducible (is the degree to which repeated measurements under unchanged conditions show same results): ▪ Preparation of the patients ▪ Performance of the test ▪ Analysis and interpretation ▪ Precision and correctness

 A ▪ Ascertained “verification” ; I have to verify the result of the new test by using the gold standard, (Ideally, all patients should have both the index test and the gold standard)  B ▪ Blind ; when interpreting the results, investigators should not know the results of the gold standard or the index test.

 I ▪ Independent; the new test is not part of the gold standard.

 Types of bias in diagnosis:  Selection bias : did the patient sample include an appropriate spectrum of patients to whom the diagnostic test will be applied in clinical practice. (choosing only severe cases will affect the sensitivity … falsely increased)

 Verification bias: did the results of the test being evaluated influence the decision to perform the reference standard. ▪ i.e. : not all index test results will be verified by gold standard test. ▪ Solution follow up participants who were not subjected the gold standard and the duration should be based on the nature of the disease.

RESULTS TherapySystematic Review DiagnosisPrognosis Magnitude and Precision ARR & NNT RR,RRR and OR Pooled results: Forest Plot (RR, OR) Accuracy of the test: Sn & Sp PPV,NPV LR SnOUT SpPIN Kaplan Meier curve: -1 st year survival rate -5 yr survival rate -Median survival

1-

APPLICAILITY TherapySystematic Review DiagnosisPrognosis Can the results be applied to my patients care? IPPP -Intervention -Patient-similar -Preference -Potential harm IPPPO -Intervention -Patient-similar -Preference -Potential harm -All outcomes considered TPPP -Test -Patient similar -Preference -Care for Patient PF -Patient similar -Follow up sufficient

VALIDITY (RABI) Therapy Systematic Review DiagnosisPrognosis R Randomization Criteria (Inclusion and exclusion - PICOTT) Reproducibility Reference standard Representative spectrum Representative sample (Homogeneous patients with respect prognostic risk) A Allocation (concealed) Attrition Articles search Articles selection Articles abstraction Ascertained (Verification) Adjustment of prognostic factors B Blindness (3Cs) Blind Articles quality assessment Blind Comparison Blind (Outcome assessor should e blind to prognostic factors – outcome need to be Objective not subjective) I Intention to treatInconsistency (Heterogeneity) Independent test not part of Gold Standard

 Validity  R ▪ Representative sample ▪ Setting : 1˚, 2 ˚ or 3 ˚ care, if it is more from 3 ˚ car setting (more severe cases (Referred); so we will have (Referral filter bias)). ▪ Zero point (homogenous sample): all patients should be started at the same point of the clinical course/ natural history of the disease (consistent clinical course… i.e. same stage).

 A ▪ Adjustment of prognostic factor / confounders by: ▪ Matching / risk stratification  Subgrouping based on important factors  Age, gender,… ▪ Regression analysis (multi-variables)

 Prognostic factors: factor that predict which patients do better or worse (predict the outcome). ▪ i.e. association not causation  Confounder factor: independent factor not intended to be studied and will affect the result.

▪ Attrition (follow up complete and long) ▪ Duration of study enough or not (MI ≥ 3years, Cancer ≥ 5 years ) ▪ Drop out and how large  > 20 % drop out generally can affect study  How large (depend on the percentage of drop out compared to percentage of outcome)

Drop outOutcome Worst case scenario Study 11%10% 11% (minimal effect) Study21% 2% (Doubled) Example If the outcome is low and drop out is same or higher the affect will be higher (study 2)and vice versa

 B ▪ Blinding ▪ Outcome assessor should be blind to the prognostic factors ▪ Outcome clearly defined

Outcome criteria: ▪ Objective ▪ Reproducible ▪ Accurate ▪ Not composite (not combined)

 Types of bias in prognosis:  Attrition bias: (% lost to follow up or duration was not proper).  Referral filter bias: (severe cases were referred – filtered- to tertiary care because the need for special expertise).

 Diagnostic bias: outcome assessor not blind to the prognostic factors or inconsistent measurement (no clearly defined outcome criteria)  Lead time bias : survival (as measured from the time of diagnosis ) may be increase not because patients live longer but because screening lengthens the time that they have disease. E.g. prostate cancer

Study types:  Cohort (the best)  Case control

RESULTS TherapySystematic Review DiagnosisPrognosis Magnitude and Precision ARR & NNT RR,RRR and OR Pooled results: Forest Plot (RR, OR) Accuracy of the test: Sn & Sp PPV,NPV LR SnOUT SpPIN Kaplan Meier curve: -1 st year survival rate -5 yr survival rate -Median survival

APPLICAILITY TherapySystematic Review DiagnosisPrognosis Can the results be applied to my patients care? IPPP -Intervention -Patient-similar -Preference -Potential harm IPPPO -Intervention -Patient-similar -Preference -Potential harm -All outcomes considered TPPP -Test -Patient similar -Preference -Care for Patient PF -Patient similar -Follow up sufficient

 CRQ made easy,NGH  Cebm.net