1. ISCOL 2011 – Bar Ilan University /151 A Probabilistic Model for Lexical Entailment Eyal Shnarch, Jacob Goldberger, Ido Dagan Bar Ilan University

2. ISCOL 2011 – Bar Ilan University /152 Textual Entailment is a common task Obama gave a speech last night in the Israeli lobby conference... In his speech at the American Israel Public Affairs Committee yesterday, the president challenged … Barack Obama’s AIPAC address... AIPAC Israeli lobby American Israel Public Affairs Committee address speech Barack Obama the president Obama

3. ISCOL 2011 – Bar Ilan University /153 Textual Entailment AIPAC  Israeli lobbyspeech  address

4. ISCOL 2011 – Bar Ilan University /154 The president’s car got stuck in Ireland, surrounded by many people Obama’s Cadillac got stuck in Dublin in a large Irish crowd social group Modeling entailment at the lexical level

5. ISCOL 2011 – Bar Ilan University /155 rule 2 rule 1 The president’s car got stuck in Ireland, surrounded by many people Obama’s Cadillac got stuck in Dublin in a large Irish crowd social group Terminology rule lexical resource chain

6. ISCOL 2011 – Bar Ilan University /156 The president’s car got stuck in Ireland, surrounded by many people Obama’s Cadillac got stuck in Dublin in a large Irish crowd social group Goals p( ) Distinguish resources’ reliability levels Consider transitive chains length Consider multiple evidence

7. ISCOL 2011 – Bar Ilan University /157 Probabilistic model for Lexical Entailment t1t1 tmtm titi h1h1 hnhn hjhj t’ AND y OR chain … … …… validity probability of the resource which produces r (ACL 2011 short paper)

8. ISCOL 2011 – Bar Ilan University /158 Results on RTE are nice, but… F 1 % Model RTE 6RTE 5 33.830.5Avg. of all systems 38.536.2Base Prob. 47.644.4Best lexical system 48.045.6Best full system

9. ISCOL 2011 – Bar Ilan University /159 Extension 1: relaxing with noisy-AND

10. ISCOL 2011 – Bar Ilan University /1510 Better results on RTE with extension 1 F 1 % Model RTE 6RTE 5 33.830.5Avg. of all systems 38.536.2Base Prob. 43.144.6Base Prob. + noisy-AND 47.644.4Best lexical system 48.045.6Best full system

11. ISCOL 2011 – Bar Ilan University /1511 Extension 2: considering coverage

12. ISCOL 2011 – Bar Ilan University /1512 Same (better) results on RTE with extension 2 F 1 % Model RTE 6RTE 5 33.830.5Avg. of all systems 38.536.2Base Prob. 43.144.6Base Prob. + noisy-AND 44.742.8Base Prob. + coverage normalization 47.644.4Best lexical system 48.045.6Best full system

13. ISCOL 2011 – Bar Ilan University /1513 Putting it all together is best F 1 % Model RTE 6RTE 5 33.830.5Avg. of all systems 38.536.2Base Prob. 43.144.6Base Prob. + noisy-AND 44.742.8Base Prob. + coverage normalization 45.648.3Full Prob. model (noisy-AND + coverage norm) 47.644.4Best lexical system 48.045.6Best full system Negative result: F1 usually decreases when allowing chains

14. ISCOL 2011 – Bar Ilan University /1514 Future work Better model for transitivity noisy-AND for chains too Verify rule application in a specific context Test with other application data sets passage retrieval for QA Integrate into a full entailment system

15. ISCOL 2011 – Bar Ilan University /1515 Summary Learn for each lexical resource an individual reliability value Consider multiple evidence and chain length Probabilistic method to relax the strict AND demand Taking into account the number of covered terms when modeling entailment probability A first probabilistic model: noisy-