Presentation is loading. Please wait.

Presentation is loading. Please wait.

A Confidence Model for Syntactically-Motivated Entailment Proofs Asher Stern & Ido Dagan ISCOL June 2011, Israel 1.

Similar presentations


Presentation on theme: "A Confidence Model for Syntactically-Motivated Entailment Proofs Asher Stern & Ido Dagan ISCOL June 2011, Israel 1."— Presentation transcript:

1 A Confidence Model for Syntactically-Motivated Entailment Proofs Asher Stern & Ido Dagan ISCOL June 2011, Israel 1

2 Recognizing Textual Entailment (RTE) Given a text, T, and a hypothesis, H Does T entail H 2 T: An explosion caused by gas took place at a Taba hotel H: A blast occurred at a hotel in Taba. Example

3 Proof Over Parse Trees 3 T = T 0 → T 1 → T 2 →... → T n = H

4 Bar Ilan Proof System - Entailment Rules 4 explosion blast Generic Syntactic Lexical Syntactic Lexical

5 Bar Ilan Proof System 5 H: A blast occurred at a hotel in Taba. LexicalLexical syntacticSyntactic An explosion caused by gas took place at a Taba hotel A blast caused by gas took place at a Taba hotel A blast took place at a Taba hotel A blast occurred at a Taba hotel A blast occurred at a hotel in Taba.

6 Tree-Edit-Distance 6 Insurgents attacked soldiers -> Soldiers were attacked by insurgents

7 Proof over parse trees Which steps? Tree-Edits – Regular or custom Entailment Rules How to classify? Decide “yes” if and only if a proof was found – Almost always “no” – Cannot handle knowledge inaccuracies Estimate a confidence to the proof correctness 7

8 Proof systems TED based Estimate the cost of a proof Complete proofs Arbitrary operations Limited knowledge Entailment Rules based Linguistically motivated Rich knowledge No estimation of proof correctness Incomplete proofs – Mixed system with ad-hoc approximate match criteria 8 Our System The benefits of both worlds, and more! – Linguistically motivated complete proofs – Confidence model

9 Our Method 1.Complete proofs – On the fly operations 2.Cost model 3.Learning model parameters 9

10 On the fly Operations “On the fly” operations – Insert node on the fly – Move node / move sub-tree on the fly – Flip part of speech – Etc. More syntactically motivated than Tree Edits Not justified, but: Their impact on the proof correctness can be estimated by the cost model. 10

11 Cost Model 11 The Idea: 1.Represent the proof as a feature-vector 2.Use the vector in a learning algorithm

12 Cost Model Represent a proof as F (P) = (F 1, F 2 … F D ) Define weight vector w=(w 1,w 2,…,w D ) Define proof cost Classify a proof – b is a threshold Learn the parameters (w,b) 12

13 Search Algorithm 13 Need to find the “best” proof “Best Proof” = proof with lowest cost ‒Assuming a weight vector is given Search space is exponential ‒pruning

14 Parameter Estimation Goal: find good weight vector and threshold (w,b) Use a standard machine learning algorithm (logistic regression or linear SVM) But: Training samples are not given as feature vectors – Learning algorithm requires training samples – Training samples construction requires weight vector – Learning weight vector done by learning algorithm Iterative learning 14

15 Parameter Estimation 15 Weight Vector Training Samples Learning Algorithm

16 Parameter Estimation 1.Start with w 0, a reasonable guess for weight vector 2.i=0 3.Repeat until convergence 1.Find the best proofs and construct vectors, using w i 2.Use a linear ML algorithm to find a new weight vector, w i+1 3.i = i+1 16

17 Results 17 SystemRTE-1RTE-2RTE-3RTE-5 Logical Resolution Refutation (Raina et al. 2005) 57.0 Probabilistic Calculus of Tree Transformations (Harmeling, 2009) 56.3957.88 Probabilistic Tree Edit model (Wang and Manning, 2010) 63.061.10 Deterministic Entailment Proofs (Bar-Haim et al., 2007) 61.1263.80 Our System 57.1361.6367.1363.50 OperationAvg. in positives Avg. in negatives Ratio Insert Named Entity0.0060.0162.67 Insert Content Word0.0380.0942.44 DIRT0.0130.0231.73 Change “subject” to “object” and vice versa0.0250.0401.60 Flip Part-of-speech0.0980.1011.03 Lin similarity0.0840.0720.86 WordNet0.0640.0520.81

18 Conclusions 1.Linguistically motivated proofs – Complete proofs 2.Cost model – Estimation of proof correctness 3.Search best proof 4.Learning parameters 5.Results – Reasonable behavior of learning scheme 18

19 Thank you Q & A 19


Download ppt "A Confidence Model for Syntactically-Motivated Entailment Proofs Asher Stern & Ido Dagan ISCOL June 2011, Israel 1."

Similar presentations


Ads by Google