1. kevin.cohen@gmail.com http://compbio.ucdenver.edu/Hunter_lab/Cohen Test suite design for biomedical ontology concept recognition systems Kevin Bretonnel Cohen Christophe Roeder William A. Baumgartner Jr. Lawrence Hunter Karin Verspoor

2. Concept recognition systems Given: –Gene Ontology (~32,000 concepts) –In mice lacking ephrin-A5 function, cell proliferation and survival of newborn neurons… (PMID 20474079) Return: –GO:0008283 cell proliferation Performance tends to be low

3. Two paradigms of evaluation Traditional approach: use a corpus Expensive Time-consuming to produce Redundancy for some things… …underrepresentation of others (Oepen et al. 1998) Slow run-time (Cohen et al. 2008) Non-traditional approach: structured test suite Controls redundancy Ensures representation of all phenomena Easy to evaluate results and do error analysis Used successfully in grammar engineering

4. Goals Big picture: How to evaluate ontology concept recognition systems? Narrow goal of this work: Test hypothesis that we can use techniques from software testing and descriptive linguistics to build test suites that overcome the disadvantages of corpora and find performance gaps Broad goal of this work: Are there general principles for test suite design?

5. Methods Experiment 1: Build a structured test suite and apply it to an ontology concept recognition system Experiment 2: Compare to other test suite work (Cohen et al. 2004) to look for common principles

6. Structured testing defined Systematic exploration of paths and combinations of features in a program –Theoretical background: set theory –Devices: state machines, grammars 6

7. How to build a structured test suite Steps: list factors that might affect system performance and their variations Assemble individual test cases that instantiate these variations… …by using insights from linguistics and from how we know concept recognition systems work –Structural aspects: length –Content aspects: typography, orthography, lexical contents (function words)… …to build a structured set of test cases

8. Structured test suite Canonical GO:0000133Polarisome GO:0000108Repairosome GO:0000786Nucleosome GO:0001660Fever GO:0001726Ruffle GO:0005623Cell GO:0005694Chromosome GO:0005814Centriole GO:0005874Microtubule Non-canonical GO:0000133Polarisomes GO:0000108Repairosomes GO:0000786Nucleosomes GO:0001660Fevers GO:0001726Ruffles GO:0005623Cells GO:0005694Chromosomes GO:0005814Centrioles GO:0005874Microtubules

9. Structured test suite Features of terms Length Punctuation Presence of stopwords Ungrammatical terms Presence of numerals Official synonyms Ambiguous terms Types of changes Singular/plural variants Ordering and other syntactic variants Inserted text Coordination Verbal versus nominal constructions Adjectival versus nominal constructions Unofficial synonyms

10. Structured test suite Syntax –induction of apoptosis  apoptosis induction Part of speech –cell migration  cell migrated Inserted text –ensheathment of neurons  ensheathment of some neurons

11. Methods/Results I Gene Ontology, revision 9/24/2009 Canonical: 188 Non-canonical: 117 Observation: –5:1 “dirty” versus 5:1 “clean” is mark of “mature” testing Applied publicly available concept recognition system

12. Results No non-canonical terms were recognized 97.9% of canonical terms were recognized –All exceptions contain the word in What would it take to recognize the error pattern with canonical terms with a corpus- based approach??

13. Methods/Results II Compare dimensions of variability to those in Cohen et al. (2004) –Applied structured test suite to five named entity recognition systems –Found errors in all five 11 features in Cohen et al. (2004), 16 features in this work

14. Methods/Results II Shared features were: –Length –Numerals –Punctuation –Function/stopwords –Syntactic context –Canonical form in source Shared boundary conditions: length and punctuation ….so, linguistics is important

15. Conclusions Structured test suites are inexpensive and effective method for finding performance gaps in ontology concept recognition systems—strong alternative to corpus- based approach Structured test suites are easier to analyze than corpus results A core feature set may exist for named entity recognition test suites

16. Future work Extend the search for basic principles to a wider range of application types –Information extraction –Question-answering –Grammar engineering –Machine translation

17. Thank you Test suite available at bionlp-corpora.sourceforge.net