Cis-Regulatory/ Text Mining Interface Discussion

Questions (1) What does ORegAnno want from text mining? –Curation queue –Document mark-up –Mapping to database IDs (2) What does text mining need from ORegAnno? (3) What can text mining provide? –What level of performance is needed? (4) What is the right way to proceed? –Data sets for BioCreAtIvE? –Custom tools for individual “early adopters”?

Answers: (1) What does ORegAnno Want from Text Mining Management of curation queue –Ideally, user customized, so that user annotates those documents of immediate interest to her/him Document mark-up to highlight relevant passages –A workflow pipeline making either the html or pdf version of the document available, with the (potentially) relevant terms highlighted –Support for “cut and paste” transfer of relevant regions to the database comments fields Mapping to IDs, ontology codes –Gene, transcription factor (protein), organism, cell and tissue type, evidence types

Answers: (2) What does Text Mining Need From ORegAnno? Significant quantity of reliably annotated data to train text mining systems –Annotated at a level useful for natural language processing (e.g., marked for evidence at the phrase, sentence or passage level, depending on task) This requires that ORegAnno have: –A clear statement of the scope of the ORegAnno database and a stable set of annotation guidelines –Annotations with high inter-annotator agreement –Tracking of entries by annotator, including depth of annotation (different annotators will annotate to different levels of detail, depending on interests)

Answers: (3) What Can Text Mining Provide? Curation queue management: –Document classification approaches (from e.g., TREC Genomics or BioCreAtIvE) can be applied and evaluated, making use of new training data from pre-jamboree and jamboree annotation –We can experiment with “user defined” criteria, based on restrictions for gene, transcription factor, organism, tissue, etc. Document mark-up –Users could be provided with a list of genes/transcription factors in a paper, with hot links into the paper to find relevant passages –This would allow the annotator to drive the annotation process, selecting only those annotations that are correct and relevant. This in turn provides feedback using ORegAnno annotations to validate & train the text mining –Such a tool should make it easy for the annotator to provide the underlying text passages as evidence for the annotation, to provide more training data Mapping to unique identifiers/controlled vocabulary/ontology –For each entity type (gene, transcription factor, organism, tissue type...), a tool can provide a mapping to the correct identifier; where there is possible ambiguity, the tool could provide a ranked list for the annotator to choose from –A tool can also flag different evidence types, with suggested code(s)

Answers: (4) How to Proceed? Stabilize guidelines and redo the inter-annotator agreement expt (and write up) Prepare a Gold Standard data set of expert annotated data for training new annotators Collect sufficient amount of training data for the various tasks (queue management, document mark up, automated mapping) Develop end-to-end pipeline (in the style of the FlySlip project) to capture whole documents in machine-readable form for mark-up

Recommendations: Training Materials & Tools Case studies and gold-standard annotated articles On-line training –Perhaps with a way for new annotators to test themselves against a set of gold standard annotations –This will require automated comparison of annotations for certain fields Best tools links Tools: –Copy mechanism for largely duplicated record