1. 1 © 2008 Emmett Keeler RAND When to do a Diagnostic Test Expected Value of Information Triage to testing Tree flipping and strategic form of trees

2. 2 © 2008 Emmett Keeler RAND Three Problems These cover first two of my classes. Check your answers with solutions, ask me or a pal if you disagree with solutions and don’t understand why. 6.2, 7.2 in Hunink I have made up a challenging 3rd problem I will give out next time. When you hand it that one, also state that you have done 6.2 and 7.2, if that is true.

3. 3 © 2008 Emmett Keeler RAND Key Points Bayes theorem says how to interpret test results, but not whether to test. Triage: typically recommended actions are split into three parts by initial p(disease)  don’t treat, test, always treat  or in public health, none, some, everyone gets it. Use EVCI to decide whether to do a costly test  Use EVPI as a quick screen for collecting data generally  If no test result changes actions EVCI = 0

4. 4 © 2008 Emmett Keeler RAND Solved simple Tree

5. Sensitivity Analysis, p(Appendicitis)

6. 6 © 2008 Emmett Keeler RAND Harm and Benefit of treatment Harm = health cost of treatment over no treatment when patient does not have disease = (.65-0) in our case. Benefit = health gain of treatment when patient has disease = 27-1 = 26. Note that treatment threshold = where lines for treatment, no treatment cross. In our case 27 p + 0 (1-p) = mortality cost of waiting line 1 p +.65 (1-p) = mortality cost of treating line. (27-1) p = (.65 - 0) (1-p) is where they cross, I.e. Benefit p = harm (1-p) at threshold

7. 7 © 2008 Emmett Keeler RAND Clinical Decision Paradigm Treat Wait and See Get more information Problem Should we use a dangerous perfect test?

8. Appendicitis NSAP 1.0.05 # Operate 0 Wait If you had perfect information, value =.05 EVPI = Do test if cost < But what if test is dangerous and imperfect, should we use it? Old tree: operate, value =.67 Appendicitis tree: perfect test No test cost(test)

9. Appendicitis Table 1 VariableBase case Value Range for Sens. Analysis Source probability.050 - 0.1EBK made it up Benefit of treatment 27 - 1cost of not treating needy Neutra, 19xx Harm of unneeded Tx 0.65 - 0cost of treating unneeded Neutra, 19xx test TPR, sensitivity 0.8EBK made it up test TNR, specificity 0.9EBK made it up Harm of test0.2EBK made it up

10. Solved Appendicitis safe tests tree 0.37 = payoff with C t = 0 What if C t =.2?

11. 11 © 2008 Emmett Keeler RAND Expected value of Information EVI = Expected payoff after new information - expected payoff from current best decision  Hunink calls this “EVCI” with C for clinical  by convention does not include cost of test If no test result changes actions, then EVI = 0.  ignores reassurance Diminishing returns to precision of information  Actions may depend on several parameters  Weak links drive uncertainty e.g. External vs internal validity of trials

12. 12 © 2008 Emmett Keeler RAND Expected Value of Perfect Information (EVPI) With a perfect test, EVI calculations are easy  Actions clear; Bayes revision is trivial  Can use the “Rabbi’s Rule”: choose the action that turns out to be best. EVPI is useful quick screen for deciding whether to try to learn more. Rabbi operates just on those with appendicitis, kills.05/3000 Expert with no test operates on everyone, kills.05 x 1 +.95 x.65 =.67, SO, EVPI =.67-.05 =.62. Should we do perfect but dangerous test that kills 1/3000? Should we do imperfect dangerous test that kills.6/3000?

13. 13 © 2008 Emmett Keeler RAND What happened? Test results split patients into two groups The test found a group that we treat differently, and EVI =.67-.37 =.3 Weak tests would not split them much, so would not change actions, so outcome stays at 0.67. 0 Probability of Disease 1 After Test Test + Test - PriorAfter test oper. threshold

14. Graph shows our test with cost.2, TPR =.8, TNR=.9 Hunink, p169 gives exact formulas for testing thresholds. Triage to testing with C t =.2

15. Triage Thresholds Decision rule splits prior into 3 segments  Wait  Get more information (in the middle)  Operate now Triage means sorting: like Katrina medics.  in NPR story, victims got colored tags: too late, can wait, needs immediate care.

16. Triage in Public Health interventions Suppose we are interested in intervening on children’s obesity to reduce subsequent bad health. Useful trick of ranking them in order of benefit What if costs of the program turn out to be A, B or C? value of health gains, costs of program per child Kids BMI level A B C

17. Probability Trees and tree-flipping Another way to do Bayes. Branches from each chance node add to 100%. Each tip has P(A&B) = P(B|A) P(A) along its path. Male Big male big big and F small and F.6.36 big.5 How to flip a tree. Fill in tips on flipped tree Fill in first stage prob. divide for cond. prob. p(M|B) = 36/56 etc.

18. Tree-flipping exercise p(dis) P(t+ | dis) p(dis & test+) Test +Test -prior prob % TreatWait Disease 0.8 0.2 20 5 30 no Dis. 0.1 0.9 80 5 0 payoffs 1.fill out 1st tree 2.flip the tree 3.calculate payoffs on nodes of 1st tree p( T given D) Dis. T+

19. Tree-flipping to simplify Treeage Trees MD finds out order tricky order test + Dis p (t+ | dis) p(dis & test+) p(dis | test+) Don’t need to do Bayes calculations

20. 20 © 2008 Emmett Keeler RAND Strategic form of trees To see if testing helps, just show “strategy” where we obey test : treat if test +, don’t treat if test negative.  let software tell you if acting without testing is better.  gets rid of branches where you act against the test. If your action is the same no matter how test comes out, don’t order test!  in that case, test branch leads to outcome without testing + the cost of the test.  so software will pick “don’t test”

21. Strategic form example

22. 22 © 2008 Emmett Keeler RAND End of class questions What was most confusing in this session? What would you like to hear more about? How was the pace? too fast, too slow or ok.

23. Test region EVCI & EVPI Benefit*FNR Harm*FPR Harm of unneeded Tx Benefit of needed Tx Mortality Cost of errors P(disease) Note that both EVIs depend on p(disease) biggest at treatment threshold (where harm = benefit) We could talk about “net” EVCI (after subtracting C t ) What happens to net EVCI as C t gets bigger?

24. 24 © 2008 Emmett Keeler RAND Test regions in general Ben.*FNR + test harm Harm*FPR + test harm Harm of unneeded Tx Benefit of needed Tx Mortality cost of errors P(disease)