Final Review 31 October WP2: Named Entity Recognition and Classification Claire Grover University of Edinburgh

Final Review 31 October NERC Multilingual IE Architecture Web pages ENERC FNERC HNERC INERC Fact Extraction Demarcator Database Domain Ontology

Final Review 31 October WP2: Objectives Specification of language neutral NERC architecture (month 6: D2.1) NERC v.1: adaptation and integration of the four existing NERC modules (month 12: D2.2) Specification of Corpus Collection Methodology NERC v.2: improvement of NERC v.1, incorporation of name matching (month 18: D2.3) NERC-based Demarcation NERC v.3: improvement of NERC v.2, incorporation of rapid adaptation mechanisms, porting to the 2 nd domain (month 26: D2.4)

Final Review 31 October Features Specific to CROSSMARC NERC Multilinguality. C urrently 4 languages but should be able to add new languages. Web pages as input. C onversion of HTML to XHTML and use of XML as common exchange format with a specific DTD per domain. Extensible to new domains. There is a need to rapidly add new domains.

Final Review 31 October Shared Features of the NERC Components XHTML input and output, shared DTD Shared domain ontology Each reuses existing NLP tools and linguistic resources Stepwise transformation of the XHTML to incrementally add mark-up, e.g. tokenisation, sentence identification, part-of-speech tagging, entity recognition.

Final Review 31 October NERC Version 2 Final version of NERC for the 1 st domain All four monolingual systems use hand-coded rule sets –HNERC uses the Ellogon Text Engineering Platform. –ENERC uses the LT TTT and LT XML tools and adds XML annotations incrementally. –INERC is implemented as a sequence of XSLT transformations of the XML document. –FNERC uses Lingway’s XTIRP Extraction Tool which applies a sequence of rule-based modules.

Final Review 31 October NERC Version 3 Reported in D2.4. Final version of NERC, dealing with 2 nd domain. Main focus is customisation methodology and experimentation to allow rapid adaptation to new domains. NERC architecture where the monolingual components are different from each other means that customisation methods are defined per component.

Final Review 31 October ENERC Customisation Methodology Retain XML pipeline architecture. Replace the named entity rule sets with a maximum entropy tagger. Experiments with the C&C Tagger and OpenNLP. Limited human intervention (selection of appropriate features).

Final Review 31 October FNERC Customisation Methodology Retain XTIRP-based architecture and modules. Use machine learning to assist in the acquisition of regular expression named entity rules. The machine-learning module produces a first version of human-readable rules plus lists of examples and counter-examples. The human expert modifies the rule set appropriately. This method reduces rule set development time to about a third.

Final Review 31 October HNERC Customisation Methodology ML-HNERC comprises: Token-based HNERC –operates over word tokens, treating NERC as a tagging problem. –word token classification performed by five independent taggers with the final tag chosen through a simple majority voter. Phrase-based HNERC –operates over phrases which have been identified using a grammar automatically induced from the training corpus –uses a C4.5 decision tree classifier to recognize phrases that describe entities.

Final Review 31 October INERC Customisation Methodology INERC is modular, with components which are general and reusable in new domains. Customization can be restricted to the lexical knowledge bases. Statistically driven process of generalizing from the annotated corpus material to derive more generalized lexical resources. Compute a frequency score to expand the lexical resources.

Final Review 31 October Evaluation Methodology For both domains we have a hand annotated corpus of 100 pages per language, split into training and testing material. Each monolingual NERC is evaluated against the testing corpus. Standard measures of precision, recall and f-measure are used.

Final Review 31 October Evaluation Summary Domain 1 F-score Domain 2 F-score ENERC FNERC HNERC INERC

Final Review 31 October Rule-based approach gives better results but it is knowledge intensive and requires significant resources for customisation to each new domain. The FNERC approach to rule induction is promising. In our experiments the machine learning approaches give lower results but: –they allow easy adaptation to new domains –there is scope to improve performance. –more training material would give better performance. Conclusions

Final Review 31 October Other WP2 Activities Collection and annotation of corpora for each language and domain. NERC-based Demarcation

Final Review 31 October Corpus Collection Methodology For each domain the process follows two steps: –identification of interesting characteristics of product descriptions and the collection of statistics relevant to these characteristics from at least 50 different sites for a language. –collection of pages and their separation into training and testing corpora.

Final Review 31 October Corpus collection principles Domain Independent principles: Training and testing corpora have the same number of pages Corpus size fixed for all languages. Corpora are representative of the statistics found per language in the site classification step. Domain Specific Principles: The maximum number of pages from one site allowed in a corpus must be decided depending on the domain. The testing corpus must contain X number of pages that come from sites not represented in the training corpus.

Final Review 31 October Annotation Annotation performed using NCSR’s annotation tool. Annotation guidelines drawn up per domain. Each corpus annotated by two separate annotators and inter-annotator agreement checked. Final corpus result of correction of cases of disagreement.

Final Review 31 October NERC-Based Demarcator Operates after NERC and before FE. Locates different product descriptions inside a web page. Current version is heuristics-based. Characteristic information: –1 st domain: manufacturer, model, price –2 nd domain: job_title, organization, education title Output: Product_No attribute on entities

Final Review 31 October Demarcator Evaluation GreekItalianEnglishFrench 1 st domain NE NUMEX TIMEX nd domain NE

Final Review 31 October Results Overview Successful multilingual NERC system which is an integral part of a resaerch platform for extracting information from web pages. An architecture that allows for new languages and swift adaptation to new domains. Four independent approaches each of which provide good results. Well motivated corpus collection methodology. Publicly distributed corpora for all languages and both domains

Final Review 31 October

Final Review 31 October

Final Review 31 October

Final Review 31 October

Final Review 31 October Shared DTDs Domain 1 NE: MANUF, MODEL, PROCESSOR, SOFT_OS TIMEX: TIME, DATE, DURATION NUMEX: LENGTH, WEIGHT, SPEED, CAPACITY, RESOLUTION, MONEY, PERCENT Domain 2 NE: MUNICIPALITY, REGION, COUNTRY, ORGANIZATION, JOB_TITLE, EDU_TITLE, LANGUAGE, S/W TIMEX: DATE, DURATION NUMEX: MONEY TERM: SCHEDULE, ORG_UNIT

Final Review 31 October st Domain Evaluation Results ENERCFNERCHNERCINERC NEMANUF MODEL SOFT_OS PROCESSOR NUMEXSPEED CAPACITY LENGTH RESOLUTION MONEY PERCENT WEIGHT TIMEXDATE DURATION TIME Overall(aprox)

Final Review 31 October nd Domain Evaluation Results ENERCFNERCHNERCINERC NEMUNICIPALITY REGION COUNTRY ORGANIZATION JOB_TITLE EDU_TITLE LANGUAGE S/W NUMEXMONEY TIMEXDATE DURATION TERMORG_UNIT SCHEDULE Overall

Final Review 31 October Further Evaluation: 2 nd Domain ENERC cross-validation New F-scorePrevious F-score NEMUNICIPALITY REGION COUNTRY ORGANIZATION JOB_TITLE EDU_TITLE0.36 LANGUAGE S/W NUMEXMONEY0.25 TIMEXDATE0.79 DURATION0.83 TERMORG_UNIT SCHEDULE0.00 Overall

Final Review 31 October Further Evaluation: 1st Domain ENERC cross-validation ME F-scoreRule-based F- score NEMANUF MODEL SOFT_OS PROCESSOR NUMEXSPEED CAPACITY LENGTH RESOLUTION0.96 MONEY PERCENT WEIGHT TIMEXDATE-0.45 DURATION TIME-0.47 Overall

Final Review 31 October Further Evaluation: 2 nd Domain ML-HNERC other languages Englis h ENERCFrenchFNERCItalianINERC NEMUNICIPALITY REGION COUNTRY ORGANIZATION JOB_TITLE EDU_TITLE LANGUAGE S/W NUMEXMONEY TIMEXDATE DURATION TERMORG_UNIT SCHEDULE Overall

Final Review 31 October Further Evaluation: 1st Domain ML-HNERC ML-HNERC F-score Rule-based HNERC F-score NEMANUF MODEL SOFT_OS PROCESSOR NUMEXSPEED CAPACITY LENGTH RESOLUTION MONEY PERCENT WEIGHT TIMEXDATE DURATION TIME-1.00 Overall

Final Review 31 October Further Evaluation: 3rd Domain ML-HNERC ML-HNERC F-score NEAGENCYs AREA0.34 CITY0.83 COUNTRY0.11 HOTEL_NAME0.00 PACK_TITLE0.00 SITE0.37 NUMEXMONEY0.73 TIMEXDATE0.22 DURATION0.67 TERMACCOM_TYPE0.63 Overall0.58

Final Review 31 October Statistics for offer description types in the 2 nd domain

Final Review 31 October nd Domain Characteristics

Final Review 31 October Summary statistics of the Italian Testing Corpus for the 2 nd Domain Pages50 Sites45 Job Offers156 Job Offers per Page3,12 NE, NUMEX TIMEX TERM Total1219 NE total1170 NUMEX Total0 TIMEX Total49 Mean names & expressions per description 7.81 Mean NEs per Job Offer7.5 Mean NUMEX per Job Offer0 Mean TIMEX per Job Offer0.31

Final Review 31 October Tag distribution in the Italian Job Offer Testing Corpus