1. Course overview, the diagnostic process, and measures of interobserver agreement Thomas B. Newman, MD, MPH September 20, 2007

2. Overview n Administrative stuff n Overview of the course n The diagnostic process n Interobserver agreement –Continuous variables –Categorical variables – Kappa Regular Weighted

3. Administrative stuff n Introductions n Basic structure of course –New material each week in lecture –Read material before lecture if possible –HW on that material due the following week in section –Exceptions: Penultimate class 11/29/07 – Chapter 12 (Clinicians and Probability) and course review: take-home exam; no HW on Ch 12 Last lecture 12/7: review of take-home exam

4. Homework n Required –key way of learning material n Which problems are assigned announced in SECTION and (later) posted on web n Not graded if late, but can still be turned in; answers on web n Use fresh sheets of paper with your name on each, not syllabus pages, not e-mail. (You can download and word-process if you want.) n Will be read by section leaders and returned the following week

5. Getting help n Classmates, then section leaders, then faculty n Ambiguous/confusing problems – send e-mail to section leader or me –Unless you indicate otherwise, we will assume we can cc the whole class when we respond if we think question is of general interest

6. Textbook n TBN and MAK are turning the syllabus into a book, “Evidence-based Diagnosis” (Cambridge University Press, 2008) n Other texts listed in syllabus and on web n Several copies of each in bookstore

7. Course overview n Diagnosis –Theory –Inter-rater reliability, accuracy, usefulness –Dichotomous tests –Multilevel tests –Combining tests n Screening and prognostic tests n Treatments: randomized trials n Alternatives to randomized trials n P-values and confidence intervals; Bayes theorem n Clinicians and probability

8. Diagnostic process n Why do we want to assign a name to this person’s illness? n Different reasons lead to different classification schemes

9. Examples n Acute nephrotic syndrome n Acute leukemia n Attention deficit disorder n Dysuria worth a course of antibiotics n SLUBI=Self-limited undiagnosed benign illness

10. Simplified Generic Decision Problem n Patient either has the disease or not n If D+, net benefit of treatment n If D-, better not to treat n (“Treat” could include doing more tests)

11. Simplifying assumptions (often wrong) n Test results are dichotomous –Most tests have more than two possible answers n Disease states are dichotomous –Many diseases occur on a spectrum –There are many kinds of “nondisease”!

12. Evaluating diagnostic tests n Reliability n Accuracy n Usefulness n Today we do reliability

13. Types of variables n Categorical –Dichotomous – 2 values –Nominal – no intrinsic ordering –Ordinal – intrinsic ordering n Continuous (infinite number of values) vs Discrete (limited number)

14. Measuring interobserver agreement for categorical variables What is agreement?

15. Concordance rate n What percent of the time do the 2 observers agree (exactly) n Advantage: easy to understand n Disadvantage: may be misleading if observers agree on prevalence of abnormality

16. Concordance rate problem

17. n Definition of Kappa n Calculation of expected agreement from marginals n Practice with the Kappa formula n Impact of marginals and balanced and unbalanced disagreements n Weighted kappa –Linear –Quadratic –Custom

18. GCS Eye opening- Observed

19. GCS Eye Opening: Expected

20. Real-life illustration: Rating of neurological examination n Types of weights, Stata illustration.. tab ex1 ex2. kap ex1 ex2, w(w). kap ex1 ex2, w(w2) n (See Appendix 2.1)

21. What does Kappa depend upon? n How well people agree n SPECTRUM within classifications –E.g., re the abnormal ones VERY abnormal? –Difficult cases can be excluded or over-sampled n PREVALENCE of classifications by the various observers (and whether they agree) n Chance (random error; people can get lucky/unlucky) n Weighting scheme used

22. Wireless Internet Access n Key is n2xa8!wr