1. Diagnostic Tests Studies 87/3/2 “How to read a paper” workshop Kamran Yazdani, MD MPH

2. Diagnostic tests When looking at a paper about a diagnostic test we ask ourselves three questions.

3. Diagnostic tests Is this test useful?

4. Diagnostic tests Is this test useful? Is it reliable?

5. Diagnostic tests Is this test useful? Is it reliable? Is it valid?

6. Is this test useful? The test should have been researched in a study population relevant to the individual or population in whom it is to be used.

7. Reliability Reliability refers to the repeatability or reproducibility of a test. It can be assessed by repeating the test using the same or different observers.

8. Validity Relates to whether the test measures what it purports to measure. Is the result true? It can be assessed by comparing the test results with a Gold Standard.

9. Validity For example if you measure blood pressure in an obese patient and use a cuff that is too small you are likely to get a falsely high reading. The reading maybe reliable (you get the same blood pressure if you do it again) but it lacks validity.

10. Gold standard The gold standard is the test or battery of tests that will most accurately diagnose a particular disease or condition. The OGTT for diabetes Fluoroscein angiography for diabetic retinopathy (too expensive or invasive) The Jones criteria for rheumatic fever (a battery of tests or symptoms)

11. What is your variable?

12. Sensitivity and specificity

14. Ability of a test to accurately diagnose diseased and healthy individuals Sensitivity Specificity Likelihood Ratio …

15. Sensitivity Disease No Disease Test Result Positive Negative TP FP FN TN Sensitivity: The capacity of the test to correctly identify diseased individuals in a population; “TRUE POSITIVES”. Gold Standard

16. Specificity Specificity: The capacity of the test to correctly exclude individuals who are free of the disease; “TRUE NEGATIVES”. Disease No Disease Positive Negative TP FP FN TN Test Result Gold Standard

17. Sensitivity and Specificity Disease No Disease Positive Negative TP FP FN TN Sensitivity Specificity TP/TP+FN TN/FP+TN Test Result Gold Standard

18. Example Disease No Disease Positive Negative 75 25 20 180 100 200 300 95 205 Sensitivity = 75/100 = 75% Specificity = 180/200 = 90% Test Result Gold Standard

19. Accuracy of the test (a+d)/(a+b+c+d) Disease No Disease Positive Negative a c b d a+c b+d 300 a+b c+d Test Result Gold Standard

20. Positive Predictive Value Disease No Disease Positive Negative TP FP FN TN PPV: The probability of the disease being present, among those with positive diagnostic test results PPV = TP/TP+FP Test Result Gold Standard

21. Negative Predictive Value Disease No Disease Positive Negative TP FP FN TN NPV: The probability that the disease was absent, among those whose diagnostic test results were negative NPV = TN/TN+FN Test Result Gold Standard

22. The effect of Sens, Spec, and P on PPV and NPV PPVNPV Prevalence SensitivitySpecificity 90%50%10%90%50%10% 70%60% 94%64%16%18%67%95% 70%90% 98.4%88%44%25%75%96% 80%90% 98.6%89%47%33%82%98% 90% 98.7%90%50% 90%99% 100%5% 2%51%10%100% 5%100% 98%51%90%

23. There are some predictors other than the prevalence: What do we do in clinic?

24. Likelihood ratio Likelihood of (+) test in diseased persons Likelihood of (+) test in diseased persons LR Positive = Likelihood of (+) test in healthy persons Likelihood of (+) test in healthy persons Likelihood of (-) test in diseased persons Likelihood of (-) test in diseased persons LR Negative= Likelihood of (-) test in healthy persons Likelihood of (-) test in healthy persons Sensitivity Sensitivity LR Positive = 1 - Specificity 1 - Specificity 1 - Sensitivity 1 - Sensitivity LR Negative= Specificity Specificity

25. Likelihood ratio Sensitivity = 90% Specificity = 90% Sensitivity 0.90 Sensitivity 0.90 LR Positive == = 9 1 – Specificity 1 – 0.90 1 – Specificity 1 – 0.90 1 – Sensitivity 1 – 0.90 1 – Sensitivity 1 – 0.90 LR Negative== = 1/9 Specificity 0.90 Specificity 0.90

26. LR nomogram

27. Example 5000 pregnant women underwent a test for blood glucose at 24 weeks, following a glucose load. 243 women were found to have a blood glucose greater than 6.8 mmol/L and were referred for an OGTT. 186 were found to have gestational diabetes. Four women who initially had tested negative were diagnosed as having diabetes later in their pregnancy.

28. DiabetesNo diabetesTotal Positive18657243 Negative447534757 Total19048105000

29. Example Prevalence Sensitivity Specificity Positive predictive value Negative predictive value Likelihood ratio + test Likelihood ratio - test Accuracy

30. Example Prevalence Sensitivity Specificity Positive predictive value Negative predictive value Likelihood ratio + test Likelihood ratio - test Accuracy 190/5000 186/190 4753/4810 186/243 4753/4757 (186/190)/(57/4810) (4/190)/(4753/4810) (186+4753)/5000

31. Example Prevalence Sensitivity Specificity Positive predictive value Negative predictive value Likelihood ratio + test Likelihood ratio - test Accuracy 3.8% 97.9% 98.8% 76.5% 99.9% 82.6.02 98.8%

32. Continuous Measurements Diseased Cutoff Value for Positive Test TN TP FP FN Healthy IOP Proportion

33. Continuous Measurements Healthy Diseased IOP Proportion Cutoff Value for Positive Test TN TP FP FN

34. Continuous Measurements Healthy Diseased IOP Proportion Cutoff Value for Positive Test TN TP FP FN

36. Receiver Operator Characteristic Curve ROC Curve 1 - Specificity Sensitivity 0 1

37. ROC Curve Analysis

39. CASP checklist

40. Improve the accuracy and completeness of research reporting and allow readers to assess the “potential for bias” in the study reported. Standards for Reporting of Diagnostic Accuracy (STARD) Always use:  FLOW CHART or Diagram  CHECKLIST

41. FLOW CHART or Diagram

42. STARD checklist Section & TopicItem # TITLE/ABS…./KEYW ORDS 1Identify the article as a study of diagnostic accuracy (recommend MeSH Heading ‘sensitivity and specificity’). INTRODUCT… 2 State the research questions or study aims, such as estimating diagnostic accuracy or comparing accuracy between tests or across participant groups.

43. Section & Topic Item # METHODS Participants3 Describe the study population: The inclusion and exclusion criteria, setting and locations where the data were collected. 4 Describe participant recruitment : Was recruitment based on presenting symptoms, results from previous tests, or the fact that the participants had received the index tests or the reference standard? 5 Describe participant sampling: Was the study population a consecutive series of participants defined by the selection criteria in items 3 and 4? If not, specify how participants were further selected. 6Describe data collection: Was data collection planned before the index test and reference standard were performed (prospective study) or after (retrospective study)? Test methods7 Describe the reference standard and its rationale. 8Describe technical specifications of material and methods involved including how and when measurements were taken, and/ or cite references for index tests and reference standard. 9Describe definition of and rationale for the units, cutoffs and/or categories of the results of the index tests and the reference standard. 10Describe the number,training and expertise of the persons executing and reading the index tests and the reference standard. 11 Describe whether or not the readers of the index tests and reference standard were blind (masked) to the results of the other tests and describe any other clinical information available to the readers. STARD checklist

44. Section & Topic Item # METHOD S Statistical methods 12Describe methods for calculating or comparing measures of diagnostic accuracy, and the statistical methods used to quantity uncertainly (e.g. 95% confidence intervals). 13Describe methods for calculating test reproducibility, if done. STARD checklist ( METHODS Continued….)

45. Section & Topic Item # RESULTS Participants14Report when study was done, including beginning and ending dates of recruitment. 15 Report clinical and demographic characteristics of the study population (e.g. age,sex, spectrum of presenting symptoms, comorbidity, current treatments, recruitment centers). 16Report the number of participants satisfying the criteria for inclusion that did or did not undergo the index tests and/or the reference standard; describe why participants failed to receive either test ( a flow diagram is strongly recommended). Test results17Report time interval from the index tests to the reference standard, and any treatment administered between. 18Report distribution of severity of disease (define criteria) in those with the target condition; other diagnoses in participants without the target condition. 19Report a cross tabulation of the results of the index tests (including indeterminate and missing results) by the results of the reference standard; for continuous results, the distribution of the test results by the results of the reference standard. 20Report any adverse events from performing the index tests or the reference standard. STARD checklist

46. Estimates21Report estimates of diagnostic accuracy and measures of statistical uncertainty (e.g. 95% confidence intervals). 22Report how indeterminate results, missing responses and outliers of the index tests were handled. 23Report estimates of variability of diagnostic accuracy between subgroups of participants, readers or centers, if done. 24Report estimates of test reproducibility, if done.

47. DISCUS….. 25 Discuss the clinical applicability of the study findings. STARD checklist

48. Thank You