An Extended Model of Natural Logic Bill MacCartney and Christopher D. Manning NLP Group Stanford University 8 January 2009

2 Natural language inference (NLI) Aka recognizing textual entailment (RTE) Does premise P justify an inference to hypothesis H? An informal, intuitive notion of inference: not strict logic Emphasis on variability of linguistic expression Necessary to goal of natural language understanding (NLU) Can also enable semantic search, question answering, … P Every firm polled saw costs grow more than expected, even after adjusting for inflation. H Every big company in the poll reported cost increases. yes Some no Introduction Semantic Relations Joins Lexical Relations Projectivity Implicatives Inference Evaluation Conclusion

3 NLI: a spectrum of approaches lexical/ semantic overlap Jijkoun & de Rijke 2005 patterned relation extraction Romano et al semantic graph matching MacCartney et al Hickl et al FOL & theorem proving Bos & Markert 2006 robust, but shallow deep, but brittle natural logic (this work) Problem: imprecise  easily confounded by negation, quantifiers, conditionals, factive & implicative verbs, etc. Problem: hard to translate NL to FOL idioms, anaphora, ellipsis, intensionality, tense, aspect, vagueness, modals, indexicals, reciprocals, propositional attitudes, scope ambiguities, anaphoric adjectives, non- intersective adjectives, temporal & causal relations, unselective quantifiers, adverbs of quantification, donkey sentences, generic determiners, comparatives, phrasal verbs, … Solution? Introduction Semantic Relations Joins Lexical Relations Projectivity Implicatives Inference Evaluation Conclusion

4 What is natural logic? (  natural deduction) Characterizes valid patterns of inference via surface forms precise, yet sidesteps difficulties of translating to FOL A long history traditional logic: Aristotle’s syllogisms, scholastics, Leibniz, … modern natural logic begins with Lakoff (1970) van Benthem & Sánchez Valencia ( ): monotonicity calculus Nairn et al. (2006): an account of implicatives & factives We introduce a new theory of natural logic extends monotonicity calculus to account for negation & exclusion incorporates elements of Nairn et al.’s model of implicatives Introduction Semantic Relations Joins Lexical Relations Projectivity Implicatives Inference Evaluation Conclusion

5 16 elementary set relations ?? ?? yy xx x y Assign sets  x, y  to one of 16 relations, depending on emptiness or non- emptiness of each of four partitions Introduction Semantic Relations Joins Lexical Relations Projectivity Implicatives Inference Evaluation Conclusion empty non-empty

6 16 elementary set relations x ^ y x  yx  y x  yx  y x ⊐ yx ⊐ y x ⊏ yx ⊏ y x | yx # y But 9 of 16 are degenerate: either x or y is either empty or universal. I.e., they correspond to semantically vacuous expressions, which are rare outside logic textbooks. We therefore focus on the remaining seven relations. Introduction Semantic Relations Joins Lexical Relations Projectivity Implicatives Inference Evaluation Conclusion

7 The set of 7 basic semantic relations Vennsymbo l name example x  yx  y equivalence couch  sofa x ⊏ yx ⊏ y forward entailment (strict) crow ⊏ bird x ⊐ yx ⊐ y reverse entailment (strict) European ⊐ French x ^ y negation (exhaustive exclusion) human ^ nonhuman x | y alternation (non-exhaustive exclusion) cat | dog x  y cover (exhaustive non-exclusion) animal  nonhuman x # y independence hungry # hippo Relations are defined for all semantic types: tiny ⊏ small, hover ⊏ fly, kick ⊏ strike, this morning ⊏ today, in Beijing ⊏ in China, everyone ⊏ someone, all ⊏ most ⊏ some Introduction Semantic Relations Joins Lexical Relations Projectivity Implicatives Inference Evaluation Conclusion

8 | x R y Joining semantic relations fishhumannonhuman ^ yz S??  ⋈  ⊏ ⋈ ⊏  ⊏ ⊐ ⋈ ⊐  ⊐ ^ ⋈ ^  R ⋈  R  ⋈ R  R ⊏ Introduction Semantic Relations Joins Lexical Relations Projectivity Implicatives Inference Evaluation Conclusion

9 Some joins yield unions of relations! x | yy | zx ? z couch | table | sofacouch  sofa pistol | knife | gunpistol ⊏ gun dog | cat | terrierdog ⊐ terrier rose | orchid | daisyrose | daisy woman | frog | Eskimowoman # Eskimo What is | | ? ⋈ | |   { , ⊏, ⊐, |, #} ⋈ Introduction Semantic Relations Joins Lexical Relations Projectivity Implicatives Inference Evaluation Conclusion

10 Of 49 join pairs, 32 yield relations in ; 17 yield unions Larger unions convey less information — limits power of inference In practice, any union which contains # can be approximated by # The complete join table Introduction Semantic Relations Joins Lexical Relations Projectivity Implicatives Inference Evaluation Conclusion

11  will depend on: 1.the lexical semantic relation generated by e:  (e) 2.other properties of the context x in which e is applied  (, ) Lexical semantic relations xe(x)e(x) compound expression atomic edit: DEL, INS, SUB semantic relation Example: suppose x is red car If e is SUB ( car, convertible ), then  (e) is ⊐ If e is DEL ( red ), then  (e) is ⊏ Crucially,  (e) depends solely on lexical items in e, independent of context x But how are lexical semantic relations determined? Introduction Semantic Relations Joins Lexical Relations Projectivity Implicatives Inference Evaluation Conclusion

12 Lexical semantic relations: SUBs  ( SUB (x, y)) =  (x, y) For open-class terms, use lexical resource (e.g. WordNet)  for synonyms: sofa  couch, forbid  prohibit ⊏ for hypo-/hypernyms: crow ⊏ bird, frigid ⊏ cold, soar ⊏ rise |for antonyms and coordinate terms: hot | cold, cat | dog  or | for proper nouns: USA  United States, JFK | FDR # for most other pairs: hungry # hippo Closed-class terms may require special handling Quantifiers: all ⊏ some, some ^ no, no | all, at least 4  at most 6 See paper for discussion of pronouns, prepositions, … Introduction Semantic Relations Joins Lexical Relations Projectivity Implicatives Inference Evaluation Conclusion

13 Lexical semantic relations: DELs & INSs Generic (default) case:  ( DEL ()) = ⊏,  ( INS ()) = ⊐ Examples: red car ⊏ car, sing ⊐ sing off-key Even quite long phrases: car parked outside since last week ⊏ car Applies to intersective modifiers, conjuncts, independent clauses, … This heuristic underlies most approaches to RTE! Does P subsume H? Deletions OK; insertions penalized. Special cases Negation: didn’t sleep ^ did sleep Implicatives & factives (e.g. refuse to, admit that ): discussed later Non-intersective adjectives: former spy | spy, alleged spy # spy Auxiliaries etc.: is sleeping  sleeps, did sleep  slept Introduction Semantic Relations Joins Lexical Relations Projectivity Implicatives Inference Evaluation Conclusion

14 The impact of semantic composition How are semantic relations affected by semantic composition? x y  ? The monotonicity calculus provides a partial answer UP  ⊏  ⊏ ⊐  ⊐ #  # DOWN  ⊏  ⊐ ⊐  ⊏ #  # NON  ⊏  # ⊐  # #  # If f has monotonicity… How is  (x, y) projected by f? But how are other relations (|, ^,  ) projected? Introduction Semantic Relations Joins Lexical Relations Projectivity Implicatives Inference Evaluation means fn application 

15 A typology of projectivity Projectivity signatures: a generalization of monotonicity classes negatio n  ⊏  ⊐ ⊐  ⊏ ^  ^ |   | #  # not French  not German not more than 4 | not less than 6 not human ^ not nonhuman didn’t kiss ⊐ didn’t touch not ill ⊏ not seasick In principle, 7 7 possible signatures, but few actually realized ↦ Each projectivity signature is a map not happy  not glad isn’t swimming # isn’t hungry Introduction Semantic Relations Joins Lexical Relations Projectivity Implicatives Inference Evaluation Conclusion

16 A typology of projectivity Projectivity signatures: a generalization of monotonicity classes Each projectivity signature is a map In principle, 7 7 possible signatures, but few actually realized ↦ negatio n  ⊏  ⊐ ⊐  ⊏ ^  ^ |   | #  # metallic pipe # nonferrous pipe intersective modification  ⊏  ⊏ ⊐  ⊐ ^  | |  |  # #  # live human | live nonhuman French wine | Spanish wine See paper for projectivity of various quantifiers, verbs Introduction Semantic Relations Joins Lexical Relations Projectivity Implicatives Inference Evaluation Conclusion

17 Projecting through multiple levels ⊏ ⊏ ⊐ ⊐ ⊐ @ Propagate semantic relation between atoms upward, according to projectivity class of each node on path to root nobody can enter with a shirt ⊏ nobody can enter with clothes Introduction Semantic Relations Joins Lexical Relations Projectivity Implicatives Inference Evaluation Conclusion

18 Implicatives & factives [Nairn et al. 06] signatur e example implicative s + / – he managed to escape + / o he was forced to sell o / – he was permitted to live implicative s – / + he forgot to pay – / o he refused to fight o / + he hesitated to ask factives+ / + he admitted that he knew – / – he pretended he was sick o / o he wanted to fly 9 signatures, per implications (+, –, or o) in positive and negative contexts Introduction Semantic Relations Joins Lexical Relations Projectivity Implicatives Inference Evaluation Conclusion

19 Implicatives & factives signatur e example  ( DEL )  ( INS ) implicative s + / – he managed to escape  he escaped  + / o he was forced to sell ⊏ he sold ⊏⊐ o / – he was permitted to live ⊐ he lived ⊐⊏ implicative s – / + he forgot to pay ^ he paid ^^ – / o he refused to fight | he fought || o / + he hesitated to ask  he asked  factives+ / + he admitted that he knew ⊏ he knew ⊏⊐ – / – he pretended he was sick | he was sick || o / o he wanted to fly # he flew ## We can specify relation generated by DEL or INS of each signature Introduction Semantic Relations Joins Lexical Relations Projectivity Implicatives Inference Evaluation Conclusion Room for variation w.r.t. infinitives, complementizers, passivation, etc. Some more intuitive when negated: he didn’t hesitate to ask | he didn’t ask Factives not fully explained: he didn’t admit that he knew | he didn’t know

20 Putting it all together 1.Find a sequence of edits  e 1, …, e n  which transforms p into h. Define x 0 = p, x n = h, and x i = e i (x i–1 ) for i  [1, n]. 2.For each atomic edit e i : a.Determine the lexical semantic relation  (e i ). b.Project  (e i ) upward through the semantic composition tree of expression x i–1 to find the atomic semantic relation  (x i–1, x i ) 3.Join atomic semantic relations across the sequence of edits:  (p, h) =  (x 0, x n ) =  (x 0, x 1 ) ⋈ … ⋈  (x i–1, x i ) ⋈ … ⋈  (x n–1, x n ) Limitations: need to find appropriate edit sequence connecting p and h; tendency of ⋈ operation toward less-informative semantic relations; lack of general mechanism for combining multiple premises Less deductive power than FOL. Can’t handle e.g. de Morgan’s Laws. Introduction Semantic Relations Joins Lexical Relations Projectivity Implicatives Inference Evaluation Conclusion

21 An example P The doctor didn’t hesitate to recommend Prozac. H The doctor recommended medication. yes ieiei xixi lexatomjoin The doctor didn’t hesitate to recommend Prozac. 1DEL( hesitate to ) The doctor didn’t recommend Prozac. 2DEL( didn’t ) The doctor recommended Prozac. 3SUB( Prozac, medication ) The doctor recommended medication. Introduction Semantic Relations Joins Lexical Relations Projectivity Implicatives Inference Evaluation Conclusion  || ^^ ⊏ ⊏⊏⊏ yes

22 Different edit orders? ieiei lexatomjoin 1DEL( hesitate to )  || 2DEL( didn’t )^^ ⊏ 3SUB( Prozac, medication ) ⊏⊏⊏ ieiei lexatomjoin 1DEL( didn’t )^^^ 2DEL( hesitate to )  ⊏ 3SUB( Prozac, medication ) ⊏⊏⊏ ieiei lexatomjoin 1SUB( Prozac, medication ) ⊏⊏⊏ 2DEL( hesitate to )  || 3DEL( didn’t )^^ ⊏ ieiei lexatomjoin 1DEL( hesitate to )  || 2SUB( Prozac, medication ) ⊏⊐ | 3DEL( didn’t )^^ ⊏ ieiei lexatomjoin 1DEL( didn’t )^^^ 2SUB( Prozac, medication ) ⊏⊐ | 3DEL( hesitate to )  ⊏ ieiei lexatomjoin 1SUB( Prozac, medication ) ⊏⊏⊏ 2DEL( didn’t )^^| 3DEL( hesitate to )  ⊏ Intermediate steps may vary; final result is typically (though not necessarily) the same Introduction Semantic Relations Joins Lexical Relations Projectivity Implicatives Inference Evaluation Conclusion

23 Implementation & evaluation The NatLog system: an implementation of this model in code For implementation details, see [MacCartney & Manning 2008] Evaluation on FraCaS test suite 183 NLI problems, nine sections, three-way classification Accuracy 70% overall; 87% on “relevant” sections (60% coverage) Precision 89% overall: rarely predicts entailment wrongly Evaluation on RTE3 test suite Longer, more natural premises; greater diversity of inference types NatLog alone has mediocre accuracy (59%) but good precision Hybridization with broad-coverage RTE system yields gains of 4% Introduction Semantic Relations Joins Lexical Relations Projectivity Implicatives Inference Evaluation Conclusion

24 Natural logic is not a universal solution for NLI Many types of inference not amenable to natural logic approach Our inference method faces many limitations on deductive power More work to be done in fleshing out our account Establishing projectivity signatures for more quantifiers, verbs, etc. Better incorporating presuppositions But, our model of natural logic fills an important niche Precise reasoning on negation, antonymy, quantifiers, implicatives, … Sidesteps the myriad difficulties of full semantic interpretation Practical value demonstrated on FraCaS and RTE3 test suites Conclusion Natural logic is not a universal solution for NLI Many types of inference not amenable to natural logic approach Our inference method faces many limitations on deductive power More work to be done in fleshing out our account Establishing projectivity signatures for more quantifiers, verbs, etc. Better incorporating presuppositions But, our model of natural logic fills an important niche Precise reasoning on negation, antonymy, quantifiers, implicatives, … Sidesteps the myriad difficulties of full semantic interpretation Practical value demonstrated on FraCaS and RTE3 test suites Introduction Semantic Relations Joins Lexical Relations Projectivity Implicatives Inference Evaluation Conclusion :-) Thanks! Questions?