1. Creating and Evaluating a Consensus for Negated and Speculative Words in a Swedish Clinical Corpus Hercules Dalianis Maria Skeppstedt Stockholm University Department of Computer and Systems Sciences

2. Dalianis & Skeppstedt, NeSp-NLP July 10, 2010 Intro and Contents An experiment with annotated clinical text 1Background 2Creation of a consensus 3Automatic detection of cues and the class 4Comparison with the BioScope Corpus 5Conclusion and next step 2

3. Dalianis & Skeppstedt, NeSp-NLP July 10, 2010 What is special about clinical text? 3

4. Dalianis & Skeppstedt, NeSp-NLP July 10, 2010 Kvinna med hjärtsvikt, förmaksflimmer, angina pectoris. Ensamstående änka. Tidigare CVL med sequelae högersidig hemipares och afasi. Tidigare vårdad för krampanfall misstänkt apoplektisk. Inkommer nu efter att ha blivit hittad på en stol och sannolikt suttit så över natten. Inkommer nu för utredning. Sonen Johan är med. Example of clinical text (Swedish) 4

5. Dalianis & Skeppstedt, NeSp-NLP July 10, 2010 Woman with heart failures, atrial fibrillation, and angina pectoris. Single, widow. Former CVL with sequele, right hemiparesis and aphasia. Prior hosp. care for seizures, apoplectic suspected. Arrive to hospital after being found in a chair and probably been sitting there over night. Arrive for further investigation and care. Accompanied by her son Johan. Example of clinical text 5

6. Dalianis & Skeppstedt, NeSp-NLP July 10, 2010 Related research: Negation and speculation detection in clinical text Both rule-based systems and machine learning systems Precision and recall from just above 80% to just below 100% Most on English text 6

7. Dalianis & Skeppstedt, NeSp-NLP July 10, 2010 The Stockholm EPR Corpus Clinics in Stockholm 2006-2008 >800 clinics, >1 million patients In Swedish 7

8. Dalianis & Skeppstedt, NeSp-NLP July 10, 2010 The annotation Three annotators The assessment part of health records 6 740 sentences Annotated: –Cues for negation and speculation –Classify the sentence as either certain or uncertain, or break it up the into sub-clauses 8

9. The annotation Dalianis & Skeppstedt, NeSp-NLP July 10, 2010 Not really much worse than before 9

10. Construction of a consensus General idea: Choose the majority annotation Discarded: The first annotation rounds discarded (16%) 2% too different to be resolved, also discarded In the resulting consensus: 92% identically annotated by at least two persons 6% identically annotated by at least two persons for class. (For cues, only identical when disregarding the scope. Ex. could perhaps) 2% only identical for class, only when scope of class disregarded. 10

11. Dalianis & Skeppstedt, NeSp-NLP July 10, 2010 Differences between the individual annotations and the consensus 1.Fewer uncertain expressions 2.Fewer cues for speculation 3.Fewer sentences that were divided into sub- clauses 11

12. Dalianis & Skeppstedt, NeSp-NLP July 10, 2010 The BioScope Corpus 1.Cues for speculation and negation 2.The scope of speculation and negation Correlation with the patient's height and weight may be some value. 12

13. Dalianis & Skeppstedt, NeSp-NLP July 10, 2010 Comparison between the BioScope Corpus and our corpus Type of wordOur ConsensusBioScope Unique negation cues 1319 Negation cues occurring only once 510 Unique speculation cues 40879 Speculation cues occurring only once 29419 13

14. Dalianis & Skeppstedt, NeSp-NLP July 10, 2010 Our corpus/the BioScope Corpus 1.Not so detailed guidelines/More detailed guidelines 2.Consensus with majority decision/Resolving differences with chief annotator (also higher inter-annotator agreement) 3.Assessment part from many clinics/Radiology reports 14

15. Dalianis & Skeppstedt, NeSp-NLP July 10, 2010 Experiment with the Stanford Named Entity Recognizer Based on Conditional Random Fields Detections of cues and certain/uncertain Comparison between our corpus and the BioScope Corpus 15

16. Dalianis & Skeppstedt, NeSp-NLP July 10, 2010 Result of automatic detection of cues for negation PrecisionRecall Our corpus0.8790.917 The BioScope corpus 0.9760.967 16

17. Dalianis & Skeppstedt, NeSp-NLP July 10, 2010 Result of automatic detection of cues for speculation PrecisionRecall Our corpus0.6740.354 The BioScope corpus 0.9460.908 17

18. Dalianis & Skeppstedt, NeSp-NLP July 10, 2010 Result of automatic detection of class and scope PrecisionRecall Our corpus (Uncertain expression) 0.4940.371 PrecisionRecall BioScope (Scope for either negation or speculation) 0.8380.812 18

19. Dalianis & Skeppstedt, NeSp-NLP July 10, 2010 Conclusion and next step 1.Low results for detecting cues for speculation and class in our constructed corpus 2.Simplifying the task can hopefully result in: Higher inter-annotator agreement Easier to automatically learn to detect speculation 19

20. Thank you! Questions? Hercules Dalianis hercules@dsv.su.se Maria Skeppstedt mariask@dsv.su.se