NRRC Summer ‘02 Workshop Proposal Temporal December 5, 2001 MITRE James Pustejovsky, Brandeis with input from: James Allan, UMASS Inderjeet Mani, MITRE.

Slides:



Advertisements
Similar presentations
ISDSI 2009 Francesco Guerra– Università di Modena e Reggio Emilia 1 DB unimo Searching for data and services F. Guerra 1, A. Maurino 2, M. Palmonari.
Advertisements

1 End-User Programming to Support Classroom Activities on Small Devices Craig Prince University of Washington VL/HCC 2008.
A Probabilistic Representation of Systemic Functional Grammar Robert Munro Department of Linguistics, SOAS, University of London.
1 Copyright © 2010, Elsevier Inc. All rights Reserved Fig 3.1 Chapter 3.
© Keith G Jeffery, Anne G S Asserson GL6: New York: December 2004: IP & Corporate Context Relating Intellectual Property Products to the Corporate.
HathiTrust and Print Storage Building around a digital core.
1 North Carolina Department of Public Instruction Fran Hoch & Hope Tesh Accountability Conference 2002 Limited English Proficiency Policy Changes North.
Cultural Heritage in REGional NETworks REGNET Project Meeting Content Group Part 2: Functional Testing.
Cultural Heritage in REGional NETworks REGNET. July 2002Project presentation REGNET 2 WP 7 - Management Task 7.1 – Project Management Task 7.2 – Quality.
2/12/01 Professor Richard Fikes Representing Time Computer Science Department Stanford University CS222 Winter 2001 Knowledge Systems Laboratory, Stanford.
1 Guy Duchossois, Work Plan Manager Report on 2006 Work Plan.
Ad Hoc Bulk Electric System Task Force Update RPIC February 19, 2009.
1 OOA-HR Workshop, 11 October 2006 Semantic Metadata Extraction using GATE Diana Maynard Natural Language Processing Group University of Sheffield, UK.
Multilinguality & Semantic Search Eelco Mossel (University of Hamburg) Review Meeting, January 2008, Zürich.
Language Specification using Metamodelling Joachim Fischer Humboldt University Berlin LAB Workshop Geneva
Copyright © 2010 American Institutes for Research All rights reserved. Washington 21st Century Community Learning Centers Program Evaluation – April Update.
1 Establishing Performance Indicators in Support of The Illinois Commitment Presented to the Illinois Board of Higher Education December 11, 2001.
Automatic Timeline Generation from News Articles Josh Taylor and Jessica Jenkins.
1 The OneGeology project IC GS Ian Jackson, February 2007.
Examples of Physical Query Plan Alternatives
1/(19) GATE Evaluation Tools GATE Training Course October 2006 Kalina Bontcheva.
1 Temporal Information Extraction Inderjeet Mani
Seminar for Teacher Assistants
Slide # 1 Report to ADC AR Status Doug Nebert, POC U.S. Geological Survey.
WP3: Fixing RES-H/C targets Lukas Kranzl, Andreas Müller RES-H Policy Kick-off, October 2008.
Chapter 10: The Traditional Approach to Design
Systems Analysis and Design in a Changing World, Fifth Edition
Mani Srivastava UCLA - EE Department Room: 6731-H Boelter Hall Tel: WWW: Copyright 2003.
Temporal Ordering of Events in the News Domain Preethi Raghavan.
Exploring the Effectiveness of Lexical Ontologies for Modeling Temporal Relations with Markov Logic Eun Y. Ha, Alok Baikadi, Carlyle Licata, Bradford Mott,
1 Functions and Applications
PROCEDURES TO USE TO ESTABLISH STANDARDS DEVELOPMENT COMMITTEES TO REPLACE COMMON CORE'S STANDARDS 1.
© NCSR, Paris, December 5-6, 2002 WP1: Plan for the remainder (1) Ontology Ontology  Enrich the lexicons for the 1 st domain based on partners remarks.
Restricted © Siemens AG All rights reserved Siemens Corporate Technology | Month 20XX Proposed topics for TDL phase 3.
Customer Relationship Management (CRM) w/ Registrar’s Office (RO) Colin Bell Director, Enterprise Architecture, IST January 9, 2014.
THE STANDARDS DEVELOPMENT PROCESS STEP 1 PUBLIC AND COMMITTEE PROPOSAL STAGE PUBLIC AND COMMITTEE PROPOSAL CLOSING DATE FIRST TECHNICAL COMMITTEE MEETING.
Jing-Shin Chang National Chi Nan University, IJCNLP-2013, Nagoya 2013/10/15 ACLCLP – Activities ( ) & Text Corpora.
GENERATING AUTOMATIC SEMANTIC ANNOTATIONS FOR RESEARCH DATASETS AYUSH SINGHAL AND JAIDEEP SRIVASTAVA CS DEPT., UNIVERSITY OF MINNESOTA, MN, USA.
W3C XML Query Language Working Group Mark Needleman Data Research Associates ZIG Current Awareness Session July 13, 2000.
Simulation Project Organization update & review of recommendations Gabriele Cosmo, CERN/PH-SFT Application Area Internal.
AQUAINT Kickoff Meeting – December 2001 Integrating Robust Semantics, Event Detection, Information Fusion, and Summarization for Multimedia Question Answering.
Erasmus University Rotterdam Introduction With the vast amount of information available on the Web, there is an increasing need to structure Web data in.
A Flexible and Extensible Architecture for Linguistic Annotation Steven Bird *, David Day †, John Garofolo ‡, John Henderson †, Christophe Laprun ‡ and.
Towards Drafting a Risk Ontology based on the IRIS Risk Glossary SUMMER ACADEMY Sep 1 st – Sep 4 th 2009 Nick Bassiliades, Dimitris Vrakas Logic Programming.
Incorporating Extra-linguistic Information into Reference Resolution in Collaborative Task Dialogue Ryu Iida Shumpei Kobayashi Takenobu Tokunaga Tokyo.
1/(13) Using Corpora and Evaluation Tools Diana Maynard Kalina Bontcheva
Assessment Workshop Title of the Project (date). Project Title Assessment Workshop October 25, 2015© Company Name All rights reserved2 Agenda Purpose.
Indirect Supervision Protocols for Learning in Natural Language Processing II. Learning by Inventing Binary Labels This work is supported by DARPA funding.
WP 9 training2 WP 9 TRAINING Kick-off Meeting – Valkenburg 9 December 2005 dr.Marlies Bitter-Rijpkema, Open Universiteit Nederland.
Numerical Libraries Project Microsoft Incubation Group Mary Beth Hribar Microsoft Corporation CSCAPES Workshop June 10, 2008 Copyright Microsoft Corporation,
ACE Automatic Content Extraction A program to develop technology to extract and characterize meaning from human language.
Splitting Complex Temporal Questions for Question Answering systems ACL 2004.
TimeML compliant text analysis for Temporal Reasoning Branimir Boguraev and Rie Kubota Ando.
Measuring How Good Your Search Engine Is. *. Information System Evaluation l Before 1993 evaluations were done using a few small, well-known corpora of.
Copyright © The OWASP Foundation Permission is granted to copy, distribute and/or modify this document under the terms of the OWASP License. The OWASP.
Comparing Document Segmentation for Passage Retrieval in Question Answering Jorg Tiedemann University of Groningen presented by: Moy’awiah Al-Shannaq
1 Open Ontology Repository initiative - Planning Meeting - Thu Co-conveners: PeterYim, LeoObrst & MikeDean ref.:
NRRC Summer ‘02 Workshop Proposal Re-use November 30, 2001 MITRE Marc Light and Abraham Ittycheriah.
NRRC Summer ‘02 Workshop Proposal: Habitability December 5, 2001 MITRE Christy Doran, MITRE Joe Marks, MERL
ARDA Massimo Lamanna / CERN Massimo Lamanna 2 TOC ARDA Workshop Post-workshop activities Milestones (already shown in December)
A Database of Narrative Schemas A 2010 paper by Nathaniel Chambers and Dan Jurafsky Presentation by Julia Kelly.
EU Update/CIS England WFD Stakeholder Forum 4 April 2008.
Presentation to Senior Management January 7, 2010
WG 2.B Integrated River Basin Management
Potential Milestones A timeline to finish in one year
مدل زنجیره ای در برنامه های سلامت
Assessment Workshop Title of the Project (date)
POD #30 1/29/19.
CS224N Section 3: Corpora, etc.
Sanjna Kashyap 18th February 2019
Presentation transcript:

NRRC Summer ‘02 Workshop Proposal Temporal December 5, 2001 MITRE James Pustejovsky, Brandeis with input from: James Allan, UMASS Inderjeet Mani, MITRE Janet Hitzeman, MITRE John Frank, MetaCarta Erik Rauch, MetaCarta

Page 2 Copyright © 2001 The MITRE Corporation. All rights reserved. Problem  Find most accurate time reference for events in news with the goal of answering temporally-based questions related to those events  Sample questions: -Is Gates currently CEO of Microsoft? -Were there any meetings between the terrorist hijackers and Iraq before the WTC event? -Did the Enron merger with Dynegy take place?

Page 3 Copyright © 2001 The MITRE Corporation. All rights reserved. Answers:  May be selections from Database entries  May be textual fragments from articles  May be abstracted and represented visually or graphically in a timeline or hyperbolic graph E2E2 E3E3 E1E1 Were there any meetings(e2,e1) between terrorist hijackers and Iraq before the WTC event (e3)?

Page 4 Copyright © 2001 The MITRE Corporation. All rights reserved. Workshop Goals:  TenseML: To Define and Design a Metadata Specification for Markup of events, their temporal anchoring, and how they are related to each other in News text.  Algorithm Design: Given the definition of TenseML, to design and develop algorithms for extracting such data from news texts.

Page 5 Copyright © 2001 The MITRE Corporation. All rights reserved. Preliminary Tasks:  Time stamping events (identifying an event and anchoring it in time);  Ordering events with respect to each other (relating more than one event in terms of precedence, overlap, and inclusion);  Temporal and event anaphora  Reasoning about the ramifications of an event (what is changed by virtue of an event);  Reasoning about the persistence of an event (how long does an event or the outcome of an event persist).

Page 6 Copyright © 2001 The MITRE Corporation. All rights reserved. TenseML:  DTD for event types (state, process, transition) - adopt a convention of how to tag events and aspect  DTD for temporal types - Intervals, points, scales  DTD for tense relations of precedence, overlap, inclusion, and identity  DTD for persistence and ramification consequences.  Basic Semantics for common English temporal expressions

Page 7 Copyright © 2001 The MITRE Corporation. All rights reserved. TIMEBANK Gold Standard  Create Gold standard -Human annotated table of temporally grounded events and temporal expressions: articles tagged according to the markup language defined by workshop participants  Annotation of events, temporal references, and relative event orderings: -Specification to be determined -Draft versions in Mani and Wilson (2000) -Ferro, Mani, Sundheim, and Wilson (2001) -Filatova and Hovy (2001) -Ferro (2001) -Setzer and Gaizauskas (2001)

Page 8 Copyright © 2001 The MITRE Corporation. All rights reserved. Domain and Data sets  Project is likely to use the news domain  Possible data sets: -Reuters Newswire source (in NewsML and CCS) -Factiva Corpus -TDT -TIDES -TIDES TDT-2 temporal corpus

Page 9 Copyright © 2001 The MITRE Corporation. All rights reserved. Algorithm Design  Evaluated and Develop algorithms for extracting event and time expressions  Evaluate and Develop algorithms for question answering for event and time expressions

Page 10 Copyright © 2001 The MITRE Corporation. All rights reserved. Evaluation Methods  Accuracy of event/time table generation -Against gold standard  Accuracy of answers for questions -NB: Assume question classification already known -Against gold standard (real answer) -Degradation from gold standard to system output  Evaluation will be continuous throughout workshop with true blind final exam at the end  Evaluations will be performed by the participants

Page 11 Copyright © 2001 The MITRE Corporation. All rights reserved. Work Plan and Schedule  Mid-January: Kickoff Meeting: 1 week -Begin define and design of specification: annotation  March: 3-day meeting -Assess annotation effort  April: 1 week meeting -Test and extend existing algorithms  May: 1 week meeting -Deploy algorithms  June: 1 week meeting -Integration  July: 1 week meeting (Close up and evaluation) -Evaluation and Wrap-up

Page 12 Copyright © 2001 The MITRE Corporation. All rights reserved. Proposed Team Membership  Core Members -Robert Ingria, LingoMotors -Dragomir Radev, U. Michigan -Janet Hitzeman, MITRE -John Moynes, Hapax -Jose Castano, Brandeis -James Allen, Rochester -Rob Gaizauskas, Sheffield  Leader: James Pustejovsky, Brandeis  Additional Participants -Lisa Ferro, MITRE -Inderjeet Mani, MITRE -Antonio Sanfilippo, SRA -Len Schubert, Rochester

Page 13 Copyright © 2001 The MITRE Corporation. All rights reserved. Impact  Better understanding of limits of current technology  Preliminary capabilities for answering questions using that output  Groundwork for event ordering capabilities  Specification for Temporal and Event Markup  Gold Standard Corpus that can be used by anyone in the community (TIMEBANK)  Add a new dimension to the kinds of Q&A possible

Page 14 Copyright © 2001 The MITRE Corporation. All rights reserved. Resources and Cost  Brought by the team -Expertise in Event Representation and Reasoning -Event Extraction Capabilities (LingoMotors) -TEMPEX time annotation (MITRE) -ACE specifications  Costs -Participants’ salary -Corpora, likely to be quite cheap -TenseML Annotation, from LDC or local annotators -Computers and storage -Coordination help -Travel, for the separate meetings