1. Jason H.D. Cho 1,2, Parikshit Sondhi 1, Chengxiang Zhai 1, Bruce R. Schatz 1,2,3 1 Department of Computer Science, 2 Institute of Genomic Biology, 3 Department of Medical Information Science, University of Illinois at Urbana-Champaign, Urbana, IL Resolving Healthcare Forum Posts via Similar Thread Retrieval

2. 72% of internet users looked online for health information within the past year 18% of internet users have gone online to find others who might have health concerns similar to theirs Improving health information retrieval and similar case retrieval will improve quality of search for vast majority of users Not many posts are answered in timely manner! * Pew Research http://www.pewinternet.org/ Motivation 2

3. 3

4. Envisioned Response The following threads discuss similar problems:  Doritos Allergy Very Severe and New  Certain Foods + Beer = Flushing and Head Pounding…Help!  Peanut/Food Allergies …………………… 4

5. Traditionally defined as retrieving relevant cases doctors may be interested in Doctors may want to compare cases that are similar to the current patient In online domain, we define this as retrieving forum posts written by patients We tackled cases where we do not know user’s background Case Retrieval Task 5

6. Query Characteristics Queries meant for human experts not automated systems Simple non-technical language Presence of emotional statements 6

7. Document Characteristics 7

8. How can we improve case retrieval search task? How should we represent queries? Entity-based search, or context-based search? Which posts are most informative in a given thread? Can we utilize forum categories? Our Goal 8

9. Evaluation via Pooling 350K threads and 20 queries from HealthBoards 2 judges first judged 100 query-thread pairs 88% agreement (κ=0.76) 730 total judged query-thread pairs 324 relevant 406 irrelevant 9

10. Method Summary Baseline weighting First Post BM-25 Thread BM-25 Semantic weighting Medical term extraction Shallow Information Extraction Post weighting Monotonic weighting Parabolic weighting Forum Category weighting Uniform weighting (FCUW) Feedback weighting (FCFW) Q: How should we represent queries? 10

11. State of the Art Baseline Baseline BM-25 formula: c(w,t): Count of word w in thread t c(w,q): Count of word w in query q FPBM-25: Consider only the content of first post to represent the thread document TBM-25: Consider content of entire thread to represent the thread document 11

12. Results: Query Representation Comparison RunMethodP@5Recall@30MAP B1Baseline TBM-25 0.30000.28460.1977 B2Baseline FPBM-25 0.4700 (56.6%)0.4975 (74.8%)0.3316 (67.7%) Representing first post as query is better than utilizing all of the posts 12

13. Method Summary Baseline weighting First Post BM-25 Thread BM-25 Semantic weighting Medical term extraction Shallow Information Extraction Post weighting Monotonic weighting Parabolic weighting Forum Category weighting Uniform weighting (FCUW) Feedback weighting (FCFW) Q: Which one works better? Entity- based search, or context-based search? 13

14. Medical Entity Extraction Applied ADEPT toolkit (MacLean and Heer 2013) High precision but low recall 14

15. MedicalEx: Relevance Scoring Count of occurrences labeled as med entity Count of occurrences not labeled as med entity Modified query frequency 15

16. Background (BKG) Neither PE nor MED I am severly allergic to some product that is found in both Tostitos and Doritos, as well as random other types of chips. I know the solution is "don't eat chips" but what could the product be? I don't want to accidentally consume it. When I eat this, I get very bad stomach cramps and it ruins the rest of my day/night - the only solution is to go to sleep so I can't feel it. Help! Any ideas on this? Shallow Information Extraction Physical Examination (PE) Disease, Symptoms Medication (MED) Treatment, Prevention Sondhi, 2010 16

17. ShallowEx: Relevance Scoring Give higher importance to PE and MED sentences Modified Query Count Word count in PE sentences Word count in MED sentences Word count in BKG sentences 17

18. Results: Semantic Methods RunMethodP@5Recall@30MAP B2Baseline FPBM-25 0.47000.49750.3316 S1 B2+MedEx 0.46000.42830.2918 S2B2+ShallowEx 0.53 (12.7%)0.4847 (-2.5%)0.3481 (4.9%) Shallow extraction is better than medical entity extraction 18

19. Method Summary Baseline weighting First Post BM-25 Thread BM-25 Semantic weighting Medical term extraction Shallow Information Extraction Post weighting Monotonic weighting Parabolic weighting Forum Category weighting Uniform weighting (FCUW) Feedback weighting (FCFW) Q: Which posts are most informative in a given thread? 19

20. Not all posts are equally representative Post Weighting Sondhi, 2013 20

21. Post Weighting : gives the weight of post i in a thread with K posts 21

22. Monotonic Post Weighting Post Position i Relative Post Weight for K=10 22

23. Parabolic Post Weighting 23

24. Post Weighting Methods Evaluation 24

25. Results: Post Weighting RunMethodP@5Recall@30MAP B2Baseline FPBM-25 0.47000.49750.3316 P1Monotonic 0.5100 (8.5%)0.5240 (5.3%)0.3631 (9.5%) P2Parabolic 0.5100 (8.5%)0.50400.3494 Both post weighting schemes outperform the baseline 25

26. Method Summary Baseline weighting First Post BM-25 Thread BM-25 Semantic weighting Medical term extraction Shallow Information Extraction Post weighting Monotonic weighting Parabolic weighting Forum Category weighting Uniform weighting (FCUW) Feedback weighting (FCFW) Q: Can we utilize forum categories? 26

27. Forum Categories 27

28. Relevance feedback based on top k retrieved categories Forum Category Uniform weighting (FCUW) Forum Category Feedback weighting (FCFW) Forum Category Weighting Randomly selecting forum ID Ratio of current forum ID amongst retrieved documents 28

29. Forum Category Weighting Scoring New Score Forum Category Feedback weighting Weights for forum category weighting 29

30. Results: Forum Category Weighting RunMethodP@5Recall@30MAP B2Baseline FPBM-25 0.47000.49750.3316 P1Uniform weighting 0.5200 (10.6%) 0.4678 (-7.0%) 0.3334 (0.5%) P2Feedback weighting 0.5100 (8.5%) 0.4610 (-7.3%) 0.3389 (2.2%) Uniform weighting and Feedback weighting similar performance, but FCFW less parameters to tune. 30

31. Results: Method Combinations RunMethodP@5Recall@30MAP B2Baseline FPBM-25 0.47000.49750.3316 S2 Baseline FPBM-25 + ShallowEx 0.530.48470.3481 C2 Monotonic + ShallowEx 0.5400 (14.9%)0.5354 (7.6%)0.3745 (12.9%) C3 Parabolic +ShallowEx 0.51000.51550.3573 C4 Monotonic + ShallowEx + FCFW 0.52000.5625 (13.1%)0.3702 Monotonic + ShallowEx performs the best 31

32. Conclusion Fairly high P@5 accuracy is achievable Treating first post as query performed the better than utilizing all posts in thread Shallow information extraction is better for query understanding Incorporates contextual information Utility of posts drops steadily with position Easy extension of baseline method 32

33. Future Work Recommending relevant forum posts for doctors Various online forums have ‘ask a doctor’ section Doctors will save time by recommending forum posts Intent-based case retrieval Identifying intents for both the end user and the existing posts will improve search quality Examples: Cause of symptom, managing disease, adverse effects 33

34. This work is supported in part by the National Science Foundation under Grant Number CNS-1027965. We would also like to thank the anonymous reviewers for their invalu- able feedback, and Institute of Genomic Biology for their computing resources. Acknowledgements 34

35. Questions? Thank you! 35

36. J. H. D. Cho and V. Q. Liao and Y. Jiang and B. Schatz, Aggregating Personal Health Messages for Scalable Comparative Effectiveness Research. ACM BCB, 2013 J. H. D. Cho and P. Sondhi and C. Zhai and B. Schatz, Resolving Healthcare Forum Posts via Similar Thread Retrieval. ACM BCB, 2014 K. Pattabiraman and P. Sondhi and C. Zhai, Exploiting Forum Thread Structures to Improve Thread Clustering. ICTIR 2013. P. Sondhi and M. Gupta and C. Zhai and J. Hockenmaier, Shallow Information Extraction from Medical Forum Data. COLING 2010. B. W. Chee and R. Berlin and B Schatz, Predicting Adverse Drug Events from Personal Health Messages, AMIA 2011 Diana L. MacLean and Jeffrey Heer. Identifying medical terms in patient- authored text: a crowdsourcing-based approach. Journal of the American Medical Informatics Association, pages amiajnl–2012–001110+, May 2013. References 36

37. Features & Performance of Shallow Information Extraction Method 37

38. We use the best performing SVM based classifier (Posts: 175, Sentences: 1494) ShallowEx: Extraction Model 38