1. Discovering Severity and Body Site Modifiers Dmitriy Dligach, Ph.D. Boston Children’s Hospital and Harvard Medical School

2. Acknowledgements Relation Extraction team:  Steve Bethard  Lee Becker  Wei-Te Chen  Guergana Savova Annotation team  David Harris, Glenn Zaramba, Donna Ihrke  Dann Albright and his team

3. Motivation  Clinical Element Model template has attributes/modifiers for body site and severity –Critical to discover these modifiers to normalize to and populate a CEM template  Body site modifiers: –“diverticulosis of sigmoid colon” –“LUNGS: Equal AE bilaterally, no rales, no rhonchi.”  Severity: –“low-grade fever” –“severe headache”

4. Relation Extraction  Cast as Relation Extraction for two types of UMLS relations LocationOf(Anatomical Site, Disease/Disorder) DegreeOf(Modifier, Disease/Disorder)  Example: “LUNGS: Equal AE bilaterally, no rales, no rhonchi.” LocationOf(LUNGS, rhonchi) LocationOf(LUNGS, rales)

5. Prerequisites: Entities  LocationOf –Automatic entity discovery –cTAKES extract entities of these UMLS semantic types:  Drug  Disorder  Sign/Symptom  Procedure  Anatomical Site  DegreeOf –Modifiers –Entities

6. Prerequisites: Modifiers  Modifier discovery module –Implemented in cTAKES –BIO (Begin, Inside, Outside) representation –Word features –Algorithm: SVM  Informal evaluation results –All automatically discovered modifiers appear to be valid

7. Approach  Supervised learning –Input: a pair of entities –Output: relation / no relation label  Sample sentence –“LUNGS: Equal AE bilaterally, no rales, no rhonchi.” ?(LUNGS, rhonchi) ?(LUNGS, rales) ?(rales, rhonchi)

8. Learning  Training –Pair up all entity pairs –Assign a gold relation label (including NONE) –Downsample –Train an SVM model  Testing –Pair up all entities in test set –Pass to the model –Assign label

9. Features  Word features –Words of mentions –Context words –Distance  Named entity features –Entity types –Entity context  POS features –POS tags of entities –POS tags between entities  Dependency features –Distance to common ancestor –Dependency path features  Chunking features –Head words of phrases between entities –Phrase head context  Wikipedia features –Entity similarity –Article titles

10. ClearTK Integration  Tutorial by Steve Bethard today  Feature extraction –Common interface for feature extractors –Many commonly used feature extractors available  Training –Commonly used machine learning packages –Training data writers  Evaluation framework –N-fold cross validation –Training and testing

11. Gold Annotated Data  SHARP –All entity types –Total notes: 80 –Total instances of LocationOf: 1852 –Total instances of DegreeOf: 308  ShARe (Shared Annotation Resources, PIs: Chapman, Elhadad, Savova) –Anatomical Sites and Disease/Disorders –Total notes: 130 –Total instances of LocationOf: 2190 –Total instances of DegreeOf: 702

12. Evaluation  Two-fold cross validation  LibSVM –Linear kernel  Parameter search –Kernel (Linear/RBF) –SVM Cost parameter –RBF gamma parameter –Probability of keeping a negative example  Evaluation is on gold entities as input to the relation classifier

13. Current Results  SHARP data –LocationOf: F1 = 0.72 –DegreeOf: F1 = 0.93  SHARE data –LocationOf: F1 = 0.88 –DegreeOf: F1 = 0.94  Best parameters –Linear kernel –Downsampling rate: 0.5  Best features –Entity features –Word features

14. Future Directions  Evaluation on a held-out test set  Evaluate on cTAKES-generated entities  Other types of relations relevant for CEM  Detailed error analysis  Release cTAKES module (July 2012) with trained models  Publish our findings

15. Funding  The Strategic Health IT Advanced Research Projects (SHARP) Program (90TR002) administered by the Office of the National Coordinator for Health Information Technology  Integrating Informatics and Biology to the Bedside (i2b2) NCBO U54LM008748

16. Thank you!