1. Text Mining in Biomedicine Michael Krauthammer Department of Pathology Yale University School of Medicine

2. Definition Text mining is –the process of automatically extracting knowledge from large text collections –data mining applied to text documents / knowledge discovery from text –a modular process similar to reading, where facts from different articles / books are combined for novel inference (de Bruijn 2002)

3. Examples in Biomedicine Protein A activates Protein B Protein C triggers Apoptosis Protein B activates Protein C Text Mining System Protein A Protein B Apoptosis Protein C

4. Signal Transduction © Max Planck Institute of Molecular Physiology

5. Signal Transduction - Apoptosis © Daniel Focosi / Molecular Medicine

6. Signal Transduction - Apoptosis © Daniel Focosi / Molecular Medicine

7. Signal Transduction - Apoptosis © Daniel Focosi / Molecular Medicine

8. Mining Molecular Interactions

9. Information Explosion

10. Mining Molecular Interactions Protein A activates Protein B Protein C triggers Apoptosis Protein B activates Protein C GeneWays System Protein A Protein B Apoptosis Protein C

14. Network-based Candidate Gene Prediction

18. Text Mining - Components

19. Information Extraction Information Extraction: “the activity of populating a structured information source (or database) from an unstructured, or free text, information source” (Gauzuskas & Wilks 1998)

20. Information Extraction Many information sources are free text: Law (Court Orders) Academic Research (Research Articles) Finance (Quarterly Reports) Medicine (Discharge Summaries) Biology (Molecular Interactions) Data analysis on free text is difficult Transformation of free text into structured data (machine-readable)

21. Information Extraction DISCHARGE SUMMARY (free text) PATIENT DATABASE (structured data) NameSmith Symptomfever Symptomweight loss Patient Smith reports fever and weight loss INFORMATION EXTRACTION

22. Information Extraction SCIENTIFIC ARTICLE (free text) RESEARCH DATABASE (structured data) SubstanceProtein A Interactionactivation SubstanceProtein B INFORMATION EXTRACTION We observed the activation of protein A by protein B

23. Information Extraction SCIENTIFIC ARTICLE (free text) RESEARCH DATABASE (structured data) SubstanceProtein A Interactionactivation SubstanceProtein B INFORMATION EXTRACTION We observed the activation of protein A by protein B Natural Language Processing

24. Information Extraction SCIENTIFIC ARTICLE (free text) RESEARCH DATABASE (structured data) SubstanceProtein A Interactionactivation SubstanceProtein B INFORMATION EXTRACTION We observed the activation of protein A by protein B Statistical methods Pattern matching Full/Shallow parsing

25. Statistical Methods Stapley (2000): Measuring gene associations Venn diagram of a set of Medline documents showing the Intersection of documents containing both genes i and j. BioBibliometric distance: dij=(|i|+|j|) / (|ij|) gene i gene j Stapley, B. J. and G. Benoit (2000). “Biobibliometrics: information retrieval and visualization from co- occurrences of gene names in Medline abstracts.” Pac Symp Biocomput: 529-40.

26. Pattern Matching Pattern matching (~regexp) to extract protein- protein interactions Blaschke, C., M. A. Andrade, et al. (1999). “Automatic extraction of biological information from scientific text: protein-protein interactions.” Proc Int Conf Intell Syst Mol Biol: 60-7. Ng, S. K. and M. Wong (1999). “Toward Routine Automatic Pathway Discovery from On-line Scientific Text Abstracts.” Genome Inform Ser Workshop Genome Inform 10: 104-112. Ono, T., H. Hishigaki, et al. (2001). “Automated extraction of information on protein-protein interactions from the biological literature.” Bioinformatics 17(2): 155-61.

27. Full Parsing Parsing: Detect sequence of grammar rules that describe internal structure of sentence Grammar rule: S -> NP VP [The house] NP [was demolished] VP. Syntax parse tree:

28. Full Parsing Language Parsing in Biomedicine MedLEE and GENIES semantic grammar parsers Columbia University, Dr. Carol Friedman MedLEE: Clinical medicine parser: discharge summaries, radiology reports, pathology reports the patient has a family history of coronary artery disease /bodyloc>

29. Full Parsing GENIES: parser for molecular domain. Extracts molecular interactions. Frame representation: Each frame is a list beginning with the elements type, value, possibly followed by additional frames: [protein, Il-2, [state, active]] For example, the parse of Raf-1 activates Mek-1 is [action, activate, [protein, Raf-1], [protein, Mek-1]]

30. Full Parsing Handles nested sentences (context free language): mediation of sonic hedgehog-induced expression of Coup-Tfii by a protein phosphatase [action,promote,[geneorprotein, phosphatase], [action,activate,[geneorprotein,sonic hedgehog], [action,express,X,[geneorprotein,Coup-Tfii]]]]

31. Full Parsing Hafner, C. D., K. Baclawski, et al. (1994). “Creating a knowledge base of biological research papers.” Proc Int Conf Intell Syst Mol Biol 2: 147-55. Friedman, C., P. Kra, et al. (2001). GENIES: A Natural-Language System for the Extraction of Molecular Pathways from Complete Journal Articles. Proc Int Conf Intell Syst Mol Biol, Kopenhagen. Yakushiji A, Tateisi Y, Miyao Y, Tsujii J. Event extraction from biomedical papers using a full parser.Pac Symp Biocomput. 2001:408-19. McDonald DM, Chen H, Su H, Marshall BB. Extracting gene pathway relations using a hybrid grammar: the Arizona relation parser.Bioinformatics. 2004 Jul 15 Leroy G, Chen H, Martinez JD. A shallow parser based on closed-class words to capture relations in biomedical text.J Biomed Inform. 2003 Jun;36(3):145-58. Koike A, Niwa Y, Takagi T. Automatic extraction of gene/protein biological functions from biomedical text.Bioinformatics. 2004 Oct 27 Daraselia N, Yuryev A, Egorov S, Novichkova S, Nikitin A, Mazo I. Extracting human protein interactions from MEDLINE using a full-sentence parser.Bioinformatics. 2004 Mar 22;20(5):604-11. Epub 2004 Jan 22

32. Shallow Semantic Parsing Medical Abstracts Zocor (Arg0) reduced cholesterol (Arg1) “The article discussed that Zocor reduced cholesterol in the intervention group.” Medicine action blood test DATABASE What medicine decreased a blood test? How did a medicine affect a blood test?

33. Shallow Semantic Parsing Shallow Semantic Parsing Technique (SSPT) –Successfully applied in non-medical domain* –“Predicate-centric” –Dissect sentences into simple WHAT did WHAT to WHOM/WHAT, and Modifiers (WHEN, WHERE, WHY and HOW) The article discussed that Zocor (What) reduced (did What) cholesterol (to What) in the intervention group (modifiers). –Thus two core arguments, “Zocor” (Argument 0) and “cholesterol” (Argument 1), are related by the predicate “reduce(d)” –Modifier “in the intervention group” –“The article discussed that” is a null argument, i.e. it is not part of the predicate arguments. * S. Pradhan, D. Jurafsky, et al. In Proc. Of NAACL-HLT 2004.

34. Treebank contains the Wall Street Journal (WSJ) corpus annotate with syntactic information Propbank annotates the same WSJ corpus found in Treebank with semantic information Given the syntactic and semantic features, we can build a machine learning-based Information Extraction (IE) system, using shallow semantic parsing Advantage of using Treebank and Propbank is its re-use of an existing corpora to do ‘free’ information extraction in the medical domain Treebank and Propbank

35. “Pierre Vinken, 61 years old, will join the board as a nonexecutive director Nov. 29.” ( (S (NP-SBJ (NP (NNP Pierre) (NNP Vinken) ) (,,) (ADJP (NP (CD 61) (NNS years) ) (JJ old) ) (,,) ) (VP (MD will) (VP (VB join) (NP (DT the) (NN board) ) (PP-CLR (IN as) (NP (DT a) (JJ nonexecutive) (NN director) )) (NP-TMP (NNP Nov.) (CD 29) ))) (..) )) Introduction: Treebank \\treebank\parsed\mrg\wsj_0001.mrg

36. wsj/00/wsj_0001.mrg 0 8 gold join.01 vf--a 0:2-ARG0 7:0 ARGM-MOD 8:0-rel 9:1-ARG1 11:1-ARGM-PRD 15:1-ARGM-TMP Verb ‘Join’ Location in Treebank Argument 0 Argument 1Argument M Introduction: Propbank Pierre Vinken, 61 years old, will join the board as a nonexecutive director Nov. 29.

37. Overall idea Syntax From Treebank

38. Overall idea Syntax From Treebank Arg0- the eater Arg1- the thing eaten predicts Predicate Arguments From Propbank

39. Problem: WSJ corpus = business domain In order to use WSJ, we have to make sure that the predicate distribution is “representative” for medical sentences. We found that 99 out of top 100 predicates in medical abstracts can be found in the WSJ corpus.

40. Results: Verb Frequency 10 most frequently found verbs in medical abstracts #OccurrencesVerbCumulative frequency 11238reduce0.036 21163improve0.070 31056suggest0.100 4963increase0.129 5888use0.155 6808associate0.178 7742compare0.200 8733show0.221 9718provide0.242 10593appear0.260

41. Methods: ML Training set and Intra-Domain Testing Set WSJ Extract sentences with top 5 verbs 15,424 words Training Set 12,500 words Test Set 2,924 words

42. Methods: ML Training & Testing (Intra-domain) ML Training ML Testing WSJ Training Set SVMTorch* * http://www.idiap.ch/machine_learning.php?content=Torch/en_SVMTorch.txt Extraction of syntactic features from Treebank and semantic categories from Propbank Extraction of syntactic features WSJ Testing Set Build classifier for semantic categories Predict semantic categories Pierre Vinken, 61 years old, will join [the board]_Arg1 as a nonexecutive director Nov. 29.

43. Syntactic Features S NP VP The Article discussed SBAR that S NP VP Zocor reduced NP cholesterol PP in NP the intervention group Null Argument 0 Verb Argument 1

44. Syntactic Features Predicate of the sentences Syntactic path from a word to the sentence predicate –For the word Zocor, the paths are NP  S  VP  VBD and S  VP  VBD Phrase Type –The syntactic category of the constituent –NP and S for Zocor * S. Pradhan, D. Jurafsky, et al. In Proc. Of NAACL-HLT 2004.

45. Syntactic Features Position of the word relative to the predicate Head Word POS The POS tag of the syntactic head of the constituent Sub-categorization Phrase structure expanding the predicate’s parent node in the parse tree. VP  VBD-NP for the predicate reduced

46. Results: Intra-domain performance ArgumentRecallPrecisionFn NULL0.840.86 1574 00.550.480.52 236 10.850.760.81 936 20.930.450.61 152 30.00 N/A 9 40.780.640.71 17 Weighted Avg.0.820.770.80

47. Results: Comparison with Prior Work * (Intra-domain) *Table 1: Performance on WSJ test set ArgPrecisionRecallF ID (null)0.860.840.86 ID + Class0.770.820.80 * S. Pradhan, D. Jurafsky, et al. In Proc. Of NAACL-HLT 2004.

48. Methods: ML Cross-Domain Testing Set Medline Abstracts Test set (6373 Words) 250 Sentences with 5 target verbs Manual annotated by 2 Medical Experts Hand annotated test set

49. Methods: ML Testing (cross-domain) SVMTorch Extraction of syntactic features ML Training ML Testing RCT Abstracts Propbank (WSJ) Extraction of syntactic and semantic categories WSJ Training set Medical Abstracts Testing set Predict semantic categories

50. Results: Cross-domain performance ArgRecallPrecisionF n NULL0.810.700.75 3351 00.720.330.45 745 10.670.860.75 1952 20.600.240.34 325 3 0 4 0 Weighted Avg.0.750.680.71

51. Results: Comparison with prior work* (cross-domain) Table 15*: Performance on the AQUAINT test set. AQUAINT: collection of text from the NY Times Inc., AP Inc., and Xinhua News Service ArgPrecisionRecallF ID (null)0.700.810.75 ID + Class0.680.750.71

52. Discussion Our ML classifier for null arguments –Intra-domain F = 86%, and cross-domain F = 75%, difference = 11% Pradhan and Jurafsky article for null arguments –Intra-domain F = 92%, and cross-domain F = 81%, difference = 11% Reuse of Propbank and Treebank information to automatically annotate medical abstract by using SSPT and ML classifier is feasible

53. Discussion - Limitations Limitation –The results are based on a small medical testing set Future directions –Improve the performance by addition of: Verb sense feature found in Propbank was not used Lack of lexical features Verb Clustering Temporal cue words –Test the performance using much larger medical abstract test set

54. Summary Literature is an important resource for biomedical knowledge Text mining = framework for accessing the free text in the literature, and transforming it to structured data Machine Learning = essential element in the text mining process

55. Appendix: Sentence Predicate Extraction Perl module Lingua::EN::Sentence -> Identified sentences Charniak parser 1 -> Identified Parts of Speech –Based on WSJ corpus Extracted terminals with VB* POS tags Program morpha 2 -> Normalization of verbs 1.Charniak, E., A Maximum-Entropy-Inspired Parser. 1999, Brown University. 2.Minning, G., J. Carroll, and P. D., Applied morphological processing of English. Natural Language Engineering, 2001. 7(3): p. 207-223.