Question 1 A new ‘Super test’ claims to have a superb capability to diagnose disease X. Its sensitivity is 99% and specificity is 90%. Which of the following statement is most correct? The ‘Super test’ is definitely good as it has high sensitivity and specificity. The ‘Super test’ is definitely better because its sensitivity is higher than specificity. Specificity is the probability for a test to show a negative results in those without a particular disease. The ‘Super test’ is not good as its specificity is below 95% Legend: WBC = White Blood Cell Sens.= Sensitivity Spec.= Specificity

80 30 20 70 Question 2 Positive Negative Yes WBC count = or > 15K? Patients with bacteraemia (Blood culture) Positive Negative 80 30 20 70 Yes WBC count = or > 15K? No The sensitivity of “WBC” in detecting bacteraemia is: 70 / (70+30) 70 / (20+70) 80 / (80+30) 80 / (80+20) Legend: WBC = White Blood Cell Sens.= Sensitivity Spec.= Specificity

60 25 40 75 Question 3 Positive Negative Positive “New Scan” Negative Renal artery stenosis Positive Negative 60 25 40 75 Positive “New Scan” Negative The specificity of the “New Scan” is: 75 / (75+25) 75 / (75+40) 60 / (60+40) 60 / (60+25) Legend: WBC = White Blood Cell Sens.= Sensitivity Spec.= Specificity

What is the ‘number needed to treat’ to prevent one leg ulcer? 20 10 5 Question 4 A study randomised diabetic patients into treatment X (placebo) and treatment Y. It reported that leg ulcer occurred in 22% of the patients in the placebo group (treatment X) and 2% of the patients given treatment Y. What is the ‘number needed to treat’ to prevent one leg ulcer? 20 10 5 15 Legend: WBC = White Blood Cell Sens.= Sensitivity Spec.= Specificity

Question 5 A researcher reported that there was significant difference of waist circumference between male and female. The male’s waist circumference is 3cm larger than the female’s and the p-value was 0.12. Which of the following statements is most correct? The study was not significant because the absolute difference in waist circumference was only 3 cm. (B) The study was significant because the p-value is greater than 0.05 (C) P value is the probability that an observed difference occurred by chance. (D) The study was not significant because the p-value must be smaller than 0.001 Legend: WBC = White Blood Cell Sens.= Sensitivity Spec.= Specificity

Positive Predictive Value Negative Predictive Value Sensitivity Specificity Positive Predictive Value Negative Predictive Value

Definition Sensitivity: The probability of a positive test in those with the disease Specificity: The probability of a negative test in those without the disease

a b c d Sensitivity Patients with the condition Yes No Positive Test Result Negative Legend: a = True positive b = False positive c = False negative d = True negative Sensitivity = a / [a + c]

a b c d Specificity Patients with the condition Yes No Positive Test Result Negative Legend: a = True positive b = False positive c = False negative d = True negative Specificity = d / [b + d]

a b c d Positive Predictive Value (PPV) Patients with the condition PPV is the probability that a positive test is a true positive. Patients with the condition Yes No Positive a b c d Test Result Negative Legend: a = True positive b = False positive c = False negative d = True negative Positive predictive value= a / [a + b]

a b c d Negative Predictive Value (NPV) Patients with the condition NPV is the probability that a negative test is a true negative. Patients with the condition Yes No Positive a b c d Test Result Negative Legend: a = True positive b = False positive c = False negative d = True negative Negative predictive value= d / [c + d]

55 275 5 177 Patients with bacteraemia (blood culture) Example 1: Sensitivity Patients with bacteraemia (blood culture) Positive Negative Yes 55 275 5 177 WBC count = or > 15K? No Sensitivity = 55/ [55+5] = 0.91 Legend: WBC = White Blood Cell Sens.= Sensitivity Spec.= Specificity Reference: Bass JW, et al. Antimicrobial treatment of occult bacteremia: a multicenter cooperative study. Pediatric Infect Dis J 1993; 12: 466 – 73.

55 275 5 177 Patients with bacteraemia (blood culture) Example 1: Specificity Patients with bacteraemia (blood culture) Positive Negative Yes 55 275 5 177 WBC count = or > 15K? No Specificity = 177/ [275+177] = 0.39 Legend: WBC = White Blood Cell Sens.= Sensitivity Spec.= Specificity Reference: Bass JW, et al. Antimicrobial treatment of occult bacteremia: a multicenter cooperative study. Pediatric Infect Dis J 1993; 12: 466 – 73.

55 275 5 177 Example 1 (continued) Positive Predictive Value (PPV) PPV is the probability that a positive test is a true positive. Patients with bacteraemia (Blood culture) Positive Negative 55 275 5 177 Yes WBC count = or > 15K No Legend: WBC = White Blood Cell Sens.= Sensitivity Spec.= Specificity Positive Predictive Value: 55 / [55 + 275] = 0.17

55 275 5 177 Example 1 (continued) Negative Predictive Value (NPV) NPV is the probability that a negative test is a true negative. Patients with bacteraemia (Blood culture) Positive Negative 55 275 5 177 Yes WBC count = or > 15K No Legend: WBC = White Blood Cell Sens.= Sensitivity Spec.= Specificity Negative Predictive Value: 177 / [5 + 177] = 0.97

Example 2 Consider HIV screening in the general population Imagine Western blot has a sensitivity of 99.99% but a specificity of 95% Say in a city of 100,000 has an HIV incidence of 1/10,000 What will be the true & false positives as a result of screening the city?

Example 2 (continue) TP FP FN TN HIV Yes No Positive HIV Screening With that HIV incidence of 1/10,000 we may expect 10 people have HIV in that city of 100,000 people. Number of estimated HIV patient= 100 000 x 1/10 000 = 10. HIV Yes No TP FP FN TN TP= True Positive FP= False Positive TN= True Negative FN= False Negative Positive HIV Screening Negative Total 10 99,990

Example 2 (continue) 10 FP TN HIV Yes No Positive HIV Screening With a sensitivity of 99.99%, the True Positive= 99.99/100 x 10 =9.999 which is rounded to 10. HIV Yes No 10 FP TN TP= True Positive FP= False Positive TN= True Negative FN= False Negative Positive HIV Screening Negative Total 10 99,990

Example 2 (continue) 10 94,990 HIV Yes No Positive HIV Screening With a specificity of 95%, the True Negative= 95/100 * (100 000-10) = 95/100 * 99990 = 94990 HIV Yes No 10 94,990 TP= True Positive FP= False Positive TN= True Negative FN= False Negative 5,000 Positive HIV Screening Negative Total 10 99,990

Example 2 (continue) 10 98,990 HIV Yes No Positive HIV Screening If the specificity of Western Blot improves to 99%, the True Negative will be = 99/100 * 99990 = 98990; and the False Positive = 99990 - 98990= 1000 HIV Yes No 10 98,990 TP= True Positive FP= False Positive TN= True Negative FN= False Negative 1,000 Positive HIV Screening Negative Total 10 99,990

Example 2(continued) In the above example, the false positive reduces from 5000 to 1000 by improving the specificity of the test! Higher specificity will decrease the number of false positive especially in a disease of low incidence.

Revisit Question 1 A new ‘Super test’ claims to have a superb capability to diagnose disease X. Its sensitivity is 99% and specificity is 90%. Which of the following statement is most correct? The ‘Super test’ is definitely good as it has high sensitivity and specificity. The ‘Super test’ is definitely better because its sensitivity is higher than specificity. Specificity is the probability for a test to show a negative results in those without a particular disease. The ‘Super test’ is not good as its specificity is below 95% Legend: WBC = White Blood Cell Sens.= Sensitivity Spec.= Specificity

80 30 20 70 Revisit Question 2 Positive Negative Yes WBC count Patients with bacteraemia (Blood culture) Positive Negative 80 30 20 70 Yes WBC count = or > 15K? No The sensitivity of “WBC” in detecting bacteraemia is: 70 / (70+30) 70 / (20+70) 80 / (80+30) 80 / (80+20) Legend: WBC = White Blood Cell Sens.= Sensitivity Spec.= Specificity

60 25 40 75 Revisit Question 3 Positive “New Scan” Negative Renal artery stenosis Positive Negative 60 25 40 75 Positive “New Scan” Negative The specificity of the “New Scan” is: 75 / (75+25) 75 / (75+40) 60 / (60+40) 60 / (60+25) Legend: WBC = White Blood Cell Sens.= Sensitivity Spec.= Specificity

Absolute Risk Reduction Relative Risk Absolute Risk Reduction Number needed to treat

Relative Risk (RR) RR: ratio of the risk of the outcome in the treatment group compared with the control group. RR is usually reported in a prospective study. The equivocal value (cut-off value) for RR is “1”. Example 3: The Diabetes Control and Complication Trial investigated the effect of intensive diabetes therapy on the development and progression of neuropathy. The study reported that neuropathy occurred in 9.6% of the patients randomized to control-group and 2.8% of the patients randomized to intensive therapy. RR= [0.028 / 0.096] = 0.29 The RR is less than 1 which means that treatment reduces the risk of neuropathy.

Example 3 (continued) Absolute Risk Reduction (ARR): The absolute difference in the rates between the treatment and control groups. Number needed to treat (NNT): The number of people who need to be treated to achieve one good outcome or prevent one adverse event. ARR= 0.096 – 0.028 = 0.068 NNT= 1/ 0.068 = 15 15 diabetic patients need to be treated by the ‘intensive therapy’ to prevent one neuropathy. Reference: The effect of intensive diabetes therapy on the development and progression of neuropathy. Annals of Internal Medicine 1995; 122:561-8

What is the ‘number needed to treat’ to prevent one leg ulcer? 20 10 5 Revisit Question 4 A study randomised diabetic patients into treatment X (placebo) and treatment Y. It reported that leg ulcer occurred in 22% of the patients in the placebo group (treatment X) and 2% of the patients given treatment Y. What is the ‘number needed to treat’ to prevent one leg ulcer? 20 10 5 15 Legend: WBC = White Blood Cell Sens.= Sensitivity Spec.= Specificity

Example 4 The absolute risk for DVT in women aged 50-60yr is 1 in 10,000. Studies show Oral Combined HRT for 5 years has a relative risk (RR) of 2. In a cohort of 10,000 women aged 50-60yr, how many additional cases would be found in the HRT group? Which of the following is most correct? (A) 1 (B) 2 (C) 4 (D) 6 (E) 8

Example 4 The absolute risk for DVT in women aged 50-60yr is 1 in 10,000. Studies show Oral Combined HRT for 5 years has a relative risk (RR) of 2. => hrt/ [1/10000] =2 => hrt = 2 x [1/10000] ==> hrt = 2/ 10000 Hence, in a cohort of 10,000 women aged 50-60yr who take oral combined HRT for 5 years, 2 may have the risk of DVT. Additional case: 2-1 = 1.

Example 4 The absolute risk for DVT in women aged 50-60yr is 1 in 10,000 [or 0.1 in 1000] Studies show Oral Combined HRT for 5 years has a relative risk (RR) of 2. In a cohort of 10,000 women aged 50-60yr who take oral combined HRT for 5 years (RR=2), additional case of DVT is 2 - 1= 1 So The most correct answer: (A) 1 (B) 2 (C) 4 (D) 6 (E) 8

P-value Confidence interval

An analogy for confidence interval (CI): Passing mark for an examination is set at 65 marks Student A scores 85 marks (95% CI 75marks – 95 marks) 75 85 95 Student B scores 78 marks (95%CI 60 marks – 96 marks) 60 78 96 Student A performs better than student B When we are making comparison, look at the confidence intervals. 65 65

P value, confidence interval Example 5: In a case control study to investigate the potential association between lung cancer and smoking Group 1: people with lung cancer Group 2: people without lung cancer The smoking history of both groups are compared. The Null Hypothesis: there is no significant difference between the groups and their smoking history The Alternative Hypothesis: there is significant difference between the groups and their smoking history Legend: df = Degree of Freedom OR = Odds Ratio CI = Confidence Interval

= [Smoking/Non-smoking]cancer/ [Smoking/Non-smoking]noncancer Example 5 (continued) The odds ratio O.R. = [Smoking/Non-smoking]cancer/ [Smoking/Non-smoking]noncancer = [12/5]/ [4/9] = 5.4 Odds ratio is usually reported in a retrospective study or case-control study. The equivocal value(cut-off value) for odds ratio is “1”. Legend: df = Degree of Freedom OR = Odds Ratio CI = Confidence Interval

The odds ratio =5.4, p=0.03, 95% confidence interval 1.2-- 26 Example 5 (continued) The odds ratio =5.4, p=0.03, 95% confidence interval 1.2-- 26 What is your interpretation of the above results? People who have lung cancer are 5.4 times more likely to have been smokers The above finding is significant as the confidence interval does not include the value of 1.0 P value is the probability that an observed difference occurred by chance. All of the above Legend: df = Degree of Freedom OR = Odds Ratio CI = Confidence Interval

Confidence Interval: Another example If the odds ratio equal 1: equivocal i.e. there is no significant association between smoking and lung cancer. Study A: Odds Ratio for smoker to have lung cancer is 6 and the 95% confidence interval for the odds ratio is between 3 and 10 Study B: Odds Ratio for smoker to have lung cancer is 2 and the 95% confidence interval for the odds ratio is between 0.5 and 3.5

Confidence Interval: Another example Study A Odds Ratio: 6 (95% CI 3 – 10) 3 6 10 Study B Odds Ratio: 2 (95%CI 0.5 – 3.5) 0.5 2 3.5 Results from Study A is more significant Hence, when we read a research report, we should not just read the single value of odds ratio but we should read the confidence interval for the odds ratio too. 1 1

Revisit Question 5 A researcher reported that there was significant difference of waist circumference between male and female. The male’s waist circumference is 3cm larger than the female’s and the p-value was 0.12. Which of the following statements is most correct? The study was not significant because the absolute difference in waist circumference was only 3 cm. (B) The study was significant because the p-value is greater than 0.05 (C) P value is the probability that an observed difference occurred by chance. (D) The study was not significant because the p-value must be smaller than 0.001 Legend: WBC = White Blood Cell Sens.= Sensitivity Spec.= Specificity

How did you go in the Quiz? Questions?

An example of exam question A new screening blood test for Beeper Phobia Disorder (BPD) was developed which detects the presence of the Beeper Phobia gene. The surgical residents at Bedpan University Hospital were screened for BPD with this new test to determine how well it identifies early cases of this dreaded disease which, if left untreated, can result in permanent mental impairment. The goal is to be able treat BPD while the residents are still asymptomatic before they are in danger of mutilating their beepers. The results were obtained and are displayed in next slide. Legend: df = Degree of Freedom OR = Odds Ratio CI = Confidence Interval

(+) BPD (-) BPD Total (+) Test 90 30 120 (-) Test 10 70 80 100 200 An example of exam question (continue) (+) BPD (-) BPD Total (+) Test 90 30 120 (-) Test 10 70 80 100 200 Legend: df = Degree of Freedom OR = Odds Ratio CI = Confidence Interval What is the sensitivity of the screening test for BPD? A. 88% B. 70% C. 50% D. 90% E. 75%

EBM in OSCE You may have an OSCE station in which you are given a research paper for critical appraisal. EBM is usually integrated in the study. Read the abstract (objectives/ hypothesis, methods, results, conclusion) Is the objective/hypothesis clinically relevant / valid? Methods: Randomised controlled trial (RCT)? Retrospective? Prospective? .Sampling method (randomised, systematic, convenience...)? Sample size? Representative samples? Bias? Double-blinded? Single-blinded? Pilot run? Ethics approval? Will the researchers publish the paper if the results are not supportive of their hypothesis? Drop out rate? Reasons for drop out? Adverse events?

EBM in OSCE - continue Continue.. Results, Analysis & Discussion: Apply your knowledge of ‘p value’, ‘confidence interval’, ‘NNT’.......... Are the results supportive of their hypothesis? Applicable to your patients? (compare the profile of the sample/population with the profile of your patient) The approach/finding of the study: educational to you +/- your patients? Sponsor for the study: conflict of interest? Reference: RACGP Curriculum “Critical Thinking and Research”, Dr Andrew Moreton’s Mock OSCE DVD.

Thank You