Part 2 Applications of ILP Formulations in Natural Language Processing

Slides:



Advertisements
Similar presentations
Latent Variables Naman Agarwal Michael Nute May 1, 2013.
Advertisements

Part VI NP-Hardness. Lecture 23 Whats NP? Hard Problems.
Coreference Based Event-Argument Relation Extraction on Biomedical Text Katsumasa Yoshikawa 1), Sebastian Riedel 2), Tsutomu Hirao 3), Masayuki Asahara.
CONSTRAINED CONDITIONAL MODELS TUTORIAL Jingyu Chen, Xiao Cheng.
Page 1 SRL via Generalized Inference Vasin Punyakanok, Dan Roth, Wen-tau Yih, Dav Zimak, Yuancheng Tu Department of Computer Science University of Illinois.
Structured SVM Chen-Tse Tsai and Siddharth Gupta.
A Linear Programming Formulation for Global Inference in Natural Language Tasks Dan RothWen-tau Yih Department of Computer Science University of Illinois.
1 Unsupervised Semantic Parsing Hoifung Poon and Pedro Domingos EMNLP 2009 Best Paper Award Speaker: Hao Xiong.
Global Learning of Type Entailment Rules Jonathan Berant, Ido Dagan, Jacob Goldberger June 21 st, 2011.
CS 6961: Structured Prediction Fall 2014 Introduction Lecture 1 What is structured prediction?
CSE332: Data Abstractions Lecture 27: A Few Words on NP Dan Grossman Spring 2010.
1 Module 2: Fundamental Concepts Problems Programs –Programming languages.
Page 1 Generalized Inference with Multiple Semantic Role Labeling Systems Peter Koomen, Vasin Punyakanok, Dan Roth, (Scott) Wen-tau Yih Department of Computer.
Chapter 11: Limitations of Algorithmic Power
1 Integrality constraints Integrality constraints are often crucial when modeling optimizayion problems as linear programs. We have seen that if our linear.
STRUCTURED PERCEPTRON Alice Lai and Shi Zhi. Presentation Outline Introduction to Structured Perceptron ILP-CRF Model Averaged Perceptron Latent Variable.
A Discriminative Latent Variable Model for Online Clustering Rajhans Samdani, Kai-Wei Chang, Dan Roth Department of Computer Science University of Illinois.
A Global Relaxation Labeling Approach to Coreference Resolution Coling 2010 Emili Sapena, Llu´ıs Padr´o and Jordi Turmo TALP Research Center Universitat.
Empirical Methods in Information Extraction Claire Cardie Appeared in AI Magazine, 18:4, Summarized by Seong-Bae Park.
Dual Coordinate Descent Algorithms for Efficient Large Margin Structured Prediction Ming-Wei Chang and Scott Wen-tau Yih Microsoft Research 1.
Illinois-Coref: The UI System in the CoNLL-2012 Shared Task Kai-Wei Chang, Rajhans Samdani, Alla Rozovskaya, Mark Sammons, and Dan Roth Supported by ARL,
Transformation of Timed Automata into Mixed Integer Linear Programs Sebastian Panek.
Unsupervised Constraint Driven Learning for Transliteration Discovery M. Chang, D. Goldwasser, D. Roth, and Y. Tu.
Coreference Resolution with Knowledge Haoruo Peng March 20,
June 2013 Inferning Workshop, ICML, Atlanta GA Amortized Integer Linear Programming Inference Dan Roth Department of Computer Science University of Illinois.
EMIS 8373: Integer Programming NP-Complete Problems updated 21 April 2009.
A Cross-Lingual ILP Solution to Zero Anaphora Resolution Ryu Iida & Massimo Poesio (ACL-HLT 2011)
Indirect Supervision Protocols for Learning in Natural Language Processing II. Learning by Inventing Binary Labels This work is supported by DARPA funding.
An Entity-Mention Model for Coreference Resolution with Inductive Logic Programming Xiaofeng Yang 1 Jian Su 1 Jun Lang 2 Chew Lim Tan 3 Ting Liu 2 Sheng.
Contingency-Constrained PMU Placement in Power Networks
NP-Complete Problems. Running Time v.s. Input Size Concern with problems whose complexity may be described by exponential functions. Tractable problems.
NP-COMPLETE PROBLEMS. Admin  Two more assignments…  No office hours on tomorrow.
NP-Complete problems.
Program Design. The design process How do you go about writing a program? –It’s like many other things in life Understand the problem to be solved Develop.
Inference Protocols for Coreference Resolution Kai-Wei Chang, Rajhans Samdani, Alla Rozovskaya, Nick Rizzolo, Mark Sammons, and Dan Roth This research.
Multi-core Structural SVM Training Kai-Wei Chang Department of Computer Science University of Illinois at Urbana-Champaign Joint Work With Vivek Srikumar.
DeepDive Model Dongfang Xu Ph.D student, School of Information, University of Arizona Dec 13, 2015.
Global Inference via Linear Programming Formulation Presenter: Natalia Prytkova Tutor: Maximilian Dylla
A Fast Finite-state Relaxation Method for Enforcing Global Constraints on Sequence Decoding Roy Tromble & Jason Eisner Johns Hopkins University.
Chapter 11 Introduction to Computational Complexity Copyright © 2011 The McGraw-Hill Companies, Inc. Permission required for reproduction or display. 1.
Department of Computer Science The University of Texas at Austin USA Joint Entity and Relation Extraction using Card-Pyramid Parsing Rohit J. Kate Raymond.
Page 1 July 2008 ICML Workshop on Prior Knowledge for Text and Language Constraints as Prior Knowledge Ming-Wei Chang, Lev Ratinov, Dan Roth Department.
Solving Hard Coreference Problems Haoruo Peng, Daniel Khashabi and Dan Roth Problem Description  Problems with Existing Coref Systems Rely heavily on.
Static model noOverlaps :: ArgumentCandidate[] candidates -> discrete[] types for (i : (0.. candidates.size() - 1)) for (j : (i candidates.size()
Dan Roth University of Illinois, Urbana-Champaign 7 Sequential Models Tutorial on Machine Learning in Natural.
P & NP.
Lecture 7: Constrained Conditional Models
WHAT IS A Process Map? Illustration of the sequence of activities, inputs and outputs to the activities, and relationship across multiple functions, or.
1.3 Finite State Machines.
Integer Linear Programming Formulations in Natural Language Processing
Lecture 2-2 NP Class.
Simone Paolo Ponzetto University of Heidelberg Massimo Poesio
Part VI NP-Hardness.
By Dan Roth and Wen-tau Yih PowerPoint by: Reno Kriz CIS
Kai-Wei Chang University of Virginia
Two Discourse Driven Language Models for Semantics
CIS 700 Advanced Machine Learning for NLP Inference Applications
Improving a Pipeline Architecture for Shallow Discourse Parsing
Margin-based Decomposed Amortized Inference
Clustering Algorithms for Noun Phrase Coreference Resolution
Relational Inference for Wikification
The Process of Photosynthesis
NP-Complete Problems.
Constraint satisfaction problems
6CO2 + 6H2O + Light + Chlorophyll Light energy also splits
Entity Linking Survey
The Winograd Schema Challenge Hector J. Levesque AAAI, 2011
Dan Roth Computer and Information Science University of Pennsylvania
Dan Roth Department of Computer Science
Constraint satisfaction problems
Presentation transcript:

Part 2 Applications of ILP Formulations in Natural Language Processing Roth & Srikumar: ILP formulations in Natural Language Processing

Outline Example: Co-reference resolution Example: Information Extraction What do constraints give us? Roth & Srikumar: ILP formulations in Natural Language Processing

Outline Example: Co-reference resolution Example: Information Extraction What do constraints give us? Roth & Srikumar: ILP formulations in Natural Language Processing

Co-reference Resolution Clinton told National Public Radio that his answers to questions about Lewinsky were constrained by Starr’s investigation. NPR reporter Mara Liasson asked Clinton “whether you had any conversations with her about her testimony, had any conversations at all.” [K-W Chang et al 2011, Denis & Baldridge, 2007] Roth & Srikumar: ILP formulations in Natural Language Processing

Applying ILP inference for Coreference Input: Mentions (i.e. spans of text) Output: Clusters of mentions that refer to the same entity Clinton told National Public Radio that his answers to questions about Lewinsky were constrained by Starr’s investigation. NPR reporter Mara Liasson asked Clinton “whether you had any conversations with her about her testimony, had any conversations at all.” Roth & Srikumar: ILP formulations in Natural Language Processing

Best-Link Inference For each mention, Best-Link considers the best mention on its left to connect to Then, it creates a link between them if the score is above some threshold (typically 0) Best-Link inference is simple and effective [Bengtson and Roth, 2008] 1.5 3.1 m* u 1.2 -1.5 0.2 Roth & Srikumar: ILP formulations in Natural Language Processing

All-Link Inference It scores a clustering of mentions by including all possible pairwise links in the score: [McCallum and Wellner, 2003; Finley and Joachims, 2005] 1.5 3.1 -0.5 Score: 1.5 + 3.1 - 0.5 + 1.5 = 5.6 Roth & Srikumar: ILP formulations in Natural Language Processing

Integer Linear Programming (ILP) Formulation for Co-Reference Best-Link All-Link See Chang et. al, CoNLL’12 and EMNLP’13 for more a general formulation that incorporates background knowledge. Pairwise mention score Binary variable: (for every edge between u and v) This is a totally uni-modular formulation Enforce the transitive closure of the clustering Say Bestlink is totally unimodular so the inference is easy but the all link has transitivity constraints, so it might be harder Transitivity constraints make the problem harder Roth & Srikumar: ILP formulations in Natural Language Processing

Outline Example: Co-reference resolution Example: Information Extraction What do constraints give us? Roth & Srikumar: ILP formulations in Natural Language Processing

Reading comprehension is hard! Water is split, providing a source of electrons and protons (hydrogen ions, H+) and giving off O2 as a by-product. Light absorbed by chlorophyll drives a transfer of the electrons and hydrogen ions from water to an acceptor called NADP+. What can the splitting of water lead to? A: Light absorption B: Transfer of ions [Berant et al, 2014] Roth & Srikumar: ILP formulations in Natural Language Processing

Reading comprehension is hard! Water is split, providing a source of electrons and protons (hydrogen ions, H+) and giving off O2 as a by-product. Light absorbed by chlorophyll drives a transfer of the electrons and hydrogen ions from water to an acceptor called NADP+. What can the splitting of water lead to? A: Light absorption B: Transfer of ions Roth & Srikumar: ILP formulations in Natural Language Processing

Reading comprehension is hard! Enable Water is split, providing a source of electrons and protons (hydrogen ions, H+) and giving off O2 as a by-product. Light absorbed by chlorophyll drives a transfer of the electrons and hydrogen ions from water to an acceptor called NADP+. What can the splitting of water lead to? A: Light absorption B: Transfer of ions Roth & Srikumar: ILP formulations in Natural Language Processing

Reading comprehension is hard! Enable Water is split, providing a source of electrons and protons (hydrogen ions, H+) and giving off O2 as a by-product. Light absorbed by chlorophyll drives a transfer of the electrons and hydrogen ions from water to an acceptor called NADP+. Cause What can the splitting of water lead to? A: Light absorption B: Transfer of ions Roth & Srikumar: ILP formulations in Natural Language Processing

Event-arguments and event-event relations Total event-argument score Total event-event relation score split absorb transfer water light ions Theme Enable Cause Roth & Srikumar: ILP formulations in Natural Language Processing

Event-arguments and event-event relations Total event-argument score Score that arg-candidate has label L Total event-event relation score Trigger t Argument candidate a Argument label L Agent Theme Source …?? Agent Theme Source …?? split absorb transfer water light ions Agent Theme Source …?? Roth & Srikumar: ILP formulations in Natural Language Processing

Event-arguments and event-event relations Total event-event relation score Score that arg-candidate has label L Score that trigger pair connected by relation R Trigger t Argument candidate a Argument label L Relation label R Pair of events (t1, t2) split absorb transfer water light ions Enable, Prevent, Same…? Enable, Prevent, Same…? Roth & Srikumar: ILP formulations in Natural Language Processing

Joint inference with constraints No overlapping arguments Maximum number of arguments per event Maximum number of events per entity Connectivity Events that share arguments must be related And a few other constraints Roth & Srikumar: ILP formulations in Natural Language Processing

Joint inference with constraints No overlapping arguments Maximum number of arguments per trigger Maximum number of triggers per entity Connectivity Events that share arguments must be related Event 1 Event 2 Not allowed! Result Agent Entity And a few other constraints Roth & Srikumar: ILP formulations in Natural Language Processing

Joint inference with constraints No overlapping arguments Maximum number of arguments per trigger Maximum number of triggers per entity Connectivity Events that share arguments must be related Event 1 Event 2 Enable Result Agent Entity And a few other constraints Roth & Srikumar: ILP formulations in Natural Language Processing

Outline Example: Co-reference resolution Example: Information Extraction What do constraints give us? Roth & Srikumar: ILP formulations in Natural Language Processing

What do constraints give us? Makes it easy to add additional knowledge Specify them as Boolean formulas Examples “If y1 is an A, then y2 or y3 should be a B or C” “No more than two A’s allowed in the output” Many inference problems have “standard” mappings to ILPs Sequences, parsing, dependency parsing At training time: Constraints act as surrogates for (many) training examples Roth & Srikumar: ILP formulations in Natural Language Processing

ILP for inference: Remarks Any combinatorial optimization problem can be written as an integer program Even the “easy”/polynomial ones Given an ILP, checking whether it admits a polynomial-time algorithm is intractable in general ILPs are a general language for thinking about combinatorial optimization The representation allows us to make general statements about inference Important: Framing/writing down the inference problem is separate from solving it Off-the-shelf solvers for ILPs are quite good Gurobi, CPLEX Use an off the shelf solver only if you can’t solve your inference problem otherwise Roth & Srikumar: ILP formulations in Natural Language Processing