1. Acknowledgements Contact Information Objective An automated annotation tool was developed to assist human annotators in the efficient production of a high quality, reference standard of part-of-speech tags. It was important to reduce demand effect bias that could be caused by system generated part-of-speech cues being accepted by human annotators even when inaccurate. Background For an ongoing research study exploring domain adaptation of a Natural Language Processing (NLP), part-of- speech tagging system, a reference standard for the target domain of healthcare was created using clinical text reports. One of the challenges faced in clinical NLP is the limited availability of high quality annotated clinical text required for training NLP machine learning models, as in this case, a Transformation Based Learner. Methods (con’t)... was not prompted based on a tag being displayed multiple times. A web- interface was created to display each sentence to be annotated. For each word, the auto-generated cue tags along with a drop-down containing the complete tag-set were displayed, as illustrated in Figure 1.The drop-down was populated with a default only when all systems generated the same tag, otherwise, the human annotator needed to actively select a tag. Regardless of default, any tag could be manually selected as the appropriate tag for each word. Providing Annotation Options, Without Revealing Too Much Jeffrey P Ferraro MS, 1-2 Scott L DuVall PhD, 3-4 Peter Haug MD 1-2 1 Intermountain Healthcare, Salt Lake City, UT –– 2 Department of Biomedical Informatics University of Utah, Salt Lake City, UT 3 Internal Medicine University of Utah, Salt Lake City, UT –– 4 VA Salt Lake City Health Care System, Salt Lake City, UT Methods Six (6) part-of-speech tagging systems were used with their out-of-the-box tagging models to initially tag the clinical text corpus. These include the Stanford Log-linear Part-Of-Speech Tagger 1, OpenNLP pos–tagger 2, MorphAdorner 3, LingPipe 4, the Illinois Part of Speech Tagger 5, and the Specialist Lexicon 6 modified to only contain non–ambiguous terms. Tag output from each system was mapped to a normalized tag-set. To control for demand effect, tags generated by the six taggers were filtered for duplicates leaving only distinct tag choices so that selection... Results A clinical text corpus of 212 randomly selected sentences from the 10 most common clinical report types from Intermountain Medical Center were annotated by two annotators using the annotation tool. An Inter-annotator agreement of 0.95 (p-value < 0.0001) was achieved over 3,672 tagged words using Fleiss’ kappa. Demand effect was analyzed by evaluating the kappa between the human annotators and the six taggers. We would expect to see a relatively high kappa score between human annotators and the taggers as these taggers are on average 80-86% accurate on clinical texts. We would not, however, expect to see scores between human annotator and tagger reaching the same levels of which are seen between human annotators. This would be indicative of passive system agreement and demand effect. We would expect, and did see, higher kappa scores between human annotators reflecting reliance on their expert knowledge of the target domain. Conclusion Using the approach described, we were able to leverage existing, moderately accurate, NLP tools to assist human annotators in producing a high quality reference standard without the introduction of bias due to the semi- automated support for the manual annotation process. References 1.The Stanford Natural Language Processing Group. Stanford Log-linear Part-Of-Speech Tagger. http://nlp.stanford.edu/software/tagger.shtml 2.OpenNLP. OpenNLP Tools. http://opennlp. sourceforge.net// 3. Academic and Research Technologies, Northwestern University. MorphAdorner. http://morphadorner.northwestern.edu/ 4. Alias-i. LingPipe. http://alias-i.com/lingpipe/index.html 5.Cognitive Computation Group. University of Illinois at Urbana-Champaign. Illinois Part of Speech Tagger. http://l2r.cs.uiuc.edu/~cogcomp/asoftware.php?skey=FL BJPOS 6.The Lexical Systems Group. National Library of Medicine. Specialist NLP Tools. http://lexsrv3.nlm.nih.gov/Specialist/Home/index.html AnnotatorsKappa HA1 vs. HA20.953 HA1 vs. Illinois Tagger0.859 HA2 vs. Illinois Tagger0.852 HA1 vs. Stanford Tagger0.850 HA2 vs. Stanford Tagger0.843 HA1 vs. OpenNLP0.850 HA2 vs. OpenNLP0.841 HA1 vs. MorphAdorner0.822 HA2 vs. MorphAdorner0.820 HA1 vs. LingPipe0.810 HA2 vs. LingPipe0.807 HA1 vs. Spec. Lexicon0.803 HA2 vs. Spec.Lexicon0.815