The TERN Task EVALITA 2007 Valentina Bartalesi Lenzi & Rachele Sprugnoli

Outline Introduction to the Temporal Expression Recognition and Normalization Task Participants Evaluation and results: –dataset –metrics –systems’ results Conclusion EVALITA 2007 Workshop Rome, September 10, 2007

Introduction to the TERN Task (1) EVALITA 2007 Workshop Rome, September 10, 2007 Recognize and normalize Temporal Expressions (TEs) in Italian natural language texts Two subtasks: –Recognition only –Recognition and Normalization Recognition: detecting TEs occurring in the source data by identifying their extension Normalization: give a representation of TEs meaning by assigning values to a pre-defined sets of attributes

Introduction to the TERN Task (2) EVALITA 2007 Workshop Rome, September 10, 2007 Annotation specifications: TIMEX2 mark-up standard (see the Automatic Content Extraction program - ACE) with adaptation to Italian Markable TEs: –absolute expressions, e.g. 10 settembre 2007/September 10 th 2007 –relative expressions, e.g. ieri/yesterday –durations, e.g. due settimane/two weeks –set of times, e.g. ogni mese/every month –underspecified TEs, e.g. per lungo tempo/for a long time –culturally-determined expressions, e.g. anno scolastico/school year

2007 Participants We had 4 participants: –FBK-irst, Trento (FBKirst_Negri_TIME) –University of Alicante (UniAli_Puchol_TIME) –University of Alicante (UniAli_Saquete_TIME) –University of Perugia (UniPg_Faina_TIME) 3 teams partecipated to the Recognition + Normalization subtask (FBKirst_Negri_TIME - UniAli_Saquete_TIME - UniPg_Faina_TIME) 1 team partecipated to the Recognition only subtask (UniAli_Puchol_TIME) EVALITA 2007 Workshop Rome, September 10, 2007

I-CAB (Italian Content Annotation Bank) 525 news stories from the Italian local newspaper “L’Adige” 4 days 5 categories  7-8 September 2004  7-8 October 2004  News Stories  Cultural News  Economic News  Sports News  Local News EVALITA 2007 Workshop Rome, September 10, ,564 words

I-CAB (2) EVALITA 2007 Workshop Rome, September 10, sections: training (335 news stories) and test (190 news stories) # TEs = 4,603 # TEs – Training = 2,931 # TEs – Test = 1,672 Format:  SGML files containing the source text  APF (ACE Program Format) files containing the annotation

Evaluation Metrics (1) TERN scoring part of the ACE scorer with some adaptation concerning the attribute weights for the Recognition + Normalization subtask The final ranking is based on the TERN value score We also provided the following measures: –Precision –Recall –F-measure EVALITA 2007 Workshop Rome, September 10, 2007

Systems’ results (1) EVALITA 2007 Workshop Rome, September 10, 2007 Results for the Recognition only subtask

Systems’ results (2) EVALITA 2007 Workshop Rome, September 10, 2007 Results for the Recognition + Normalization subtask

Conclusion EVALITA 2007 Workshop Rome, September 10, 2007 First time I-CAB is released and used as a benchmark for an Information Extraction Task like TERN Expected attention in terms of participation for a new and relatively difficult Task We wish that the resources we developed and the results we obtained will encourage other teams to participate in future evaluation exercises We hope that this initiative will become a regular appointment as happens with similar international evaluation campaigns (e.g. ACE)