Machine Learning Group Department of Computer Sciences University of Texas at Austin Learning Language Semantics from Ambiguous Supervision Rohit J. Kate Raymond J. Mooney

2 Semantic Parsing Involves learning language semantics to transform natural language (NL) sentences into computer executable complete meaning representations (MRs) for some application Geoquery: An example database query application Which rivers run through the states bordering Texas? Query answer(traverse(next_to(stateid(‘texas’)))) Semantic Parsing Arkansas, Canadian, Cimarron, Gila, Mississippi, Rio Grande … Answer

3 Learning for Semantic Parsing Learning for semantic parsing consists of inducing a semantic parser from training data which can map novel sentences into their meaning representations Many accurate learning systems for semantic parsing have been recently developed: [ Ge & Mooney, 2005], [Zettlemoyer & Collins, 2005], [Wong & Mooney, 2006], [Kate & Mooney, 2006], [Nguyen, Shimazu & Phan, 2006]

4 Unambiguous Supervision for Learning Semantic Parsers The training data for semantic parsing consists of hundreds of natural language sentences unambiguously paired with their meaning representations

5 Unambiguous Supervision for Learning Semantic Parsers The training data for semantic parsing consists of hundreds of natural language sentences unambiguously paired with their meaning representations Which rivers run through the states bordering Texas? answer(traverse(next_to(stateid(‘texas’)))) What is the lowest point of the state with the largest area? answer(lowest(place(loc(largest_one(area(state(all))))))) What is the largest city in states that border California? answer(largest(city(loc(next_to(stateid( 'california')))))) ……

6 Shortcomings of Unambiguous Supervision It requires considerable human effort to annotate each sentence with its correct meaning representation Does not model the type of supervision children receive when they are learning a language –Children are not taught meanings of individual sentences –They learn to identify the correct meaning of a sentence from several meanings possible in their perceptual context

7 ??? “Mary is on the phone”

8 Ambiguous Supervision for Learning Semantic Parsers A computer system simultaneously exposed to perceptual contexts and natural language utterances should be able to learn the underlying language semantics We consider ambiguous training data of sentences associated with multiple potential meaning representations –Siskind (1996) uses this type “referentially uncertain” training data to learn meanings of words Capturing meaning representations from perceptual contexts is a difficult unsolved problem –Our system directly works with symbolic meaning representations

9 “Mary is on the phone” ???

10 “Mary is on the phone” ???

11 Ironing(Mommy, Shirt) “Mary is on the phone” ???

12 Ironing(Mommy, Shirt) Working(Sister, Computer) “Mary is on the phone” ???

13 Ironing(Mommy, Shirt) Working(Sister, Computer) Carrying(Daddy, Bag) “Mary is on the phone” ???

14 Ironing(Mommy, Shirt) Working(Sister, Computer) Carrying(Daddy, Bag) Talking(Mary, Phone) Sitting(Mary, Chair) “Mary is on the phone” Ambiguous Training Example ???

15 Ironing(Mommy, Shirt) Working(Sister, Computer) Talking(Mary, Phone) Sitting(Mary, Chair) “Mommy is ironing shirt” Next Ambiguous Training Example ???

16 Ambiguous Supervision for Learning Semantic Parsers contd. Our model of ambiguous supervision corresponds to the type of data that will be gathered from a temporal sequence of perceptual contexts with occasional language commentary We assume each sentence has exactly one meaning in a perceptual context Each meaning is associated with at most one sentence in a perceptual context

17 Sample Ambiguous Corpus Daisy gave the clock to the mouse. Mommy saw that Mary gave the hammer to the dog. The dog broke the box. John gave the bag to the mouse. The dog threw the ball. ate(mouse, orange) gave(daisy, clock, mouse) ate(dog, apple) saw(mother, gave(mary, dog, hammer)) broke(dog, box) gave(woman, toy, mouse) gave(john, bag, mouse) threw(dog, ball) runs(dog) saw(john, walks(man, dog)) Forms a bipartite graph

18 Rest of the Talk Brief background on KRISP, the semantic parsing learning system for unambiguous supervision KRISPER: Extended system to handle ambiguous supervision Corpus construction Experiments

19 KRISP: Semantic Parser Learner for Unambiguous Supervision KRISP: Kernel-based Robust Interpretation for Semantic Parsing [Kate & Mooney 2006] Takes NL sentences unambiguously paired with their MRs as training data Treats the formal MR language grammar’s productions as semantic concepts Trains an SVM classifier for each production with string subsequence kernel [Lodhi et al. 2002]

20 MR: answer(traverse(next_to(stateid(‘texas’)))) Parse tree of MR: Productions: ANSWER  answer(RIVER) RIVER  TRAVERSE(STATE) STATE  NEXT_TO(STATE) TRAVERSE  traverse NEXT_TO  next_to STATEID  ‘texas’ ANSWER answer STATE RIVER STATE NEXT_TO TRAVERSE STATEID stateid ‘ texas ’ next_to traverse Meaning Representation Language ANSWER  answer(RIVER) RIVER  TRAVERSE(STATE) TRAVERSE  traverse STATE  NEXT_TO(STATE) NEXT_TO  next_to STATE  STATEID STATEID  ‘ texas ’

21 Semantic Parsing by KRISP SVM classifier for each production gives the probability that a substring represents the semantic concept of the production Which rivers run through the states bordering Texas? NEXT_TO  next_to NEXT_TO  next_to 0.95

22 Semantic Parsing by KRISP SVM classifier for each production gives the probability that a substring represents the semantic concept of the production Which rivers run through the states bordering Texas? TRAVERSE  traverse

23 Semantic Parsing by KRISP Semantic parsing is done by finding the most probable derivation of the sentence [Kate & Mooney 2006] Which rivers run through the states bordering Texas? ANSWER  answer(RIVER) RIVER  TRAVERSE(STATE) TRAVERSE  traverse STATE  NEXT_TO(STATE) NEXT_TO  next_to STATE  STATEID STATEID  ‘ texas ’ Probability of the derivation is the product of the probabilities at the nodes.

24 Semantic Parsing by KRISP Given a sentence and a meaning representation, KRISP can also find the probability that it is the correct meaning representation for the sentence

25 K RISPER: K RISP with E M-like R etraining Extension of K RISP that learns from ambiguous supervision Uses an iterative EM-like method to gradually converge on a correct meaning for each sentence

26 KRISPER’s Training Algorithm Daisy gave the clock to the mouse. Mommy saw that Mary gave the hammer to the dog. The dog broke the box. John gave the bag to the mouse. The dog threw the ball. ate(mouse, orange) gave(daisy, clock, mouse) ate(dog, apple) saw(mother, gave(mary, dog, hammer)) broke(dog, box) gave(woman, toy, mouse) gave(john, bag, mouse) threw(dog, ball) runs(dog) saw(john, walks(man, dog)) 1. Assume every possible meaning for a sentence is correct

27 KRISPER’s Training Algorithm Daisy gave the clock to the mouse. Mommy saw that Mary gave the hammer to the dog. The dog broke the box. John gave the bag to the mouse. The dog threw the ball. ate(mouse, orange) gave(daisy, clock, mouse) ate(dog, apple) saw(mother, gave(mary, dog, hammer)) broke(dog, box) gave(woman, toy, mouse) gave(john, bag, mouse) threw(dog, ball) runs(dog) saw(john, walks(man, dog)) 1. Assume every possible meaning for a sentence is correct

28 KRISPER’s Training Algorithm contd. Daisy gave the clock to the mouse. Mommy saw that Mary gave the hammer to the dog. The dog broke the box. John gave the bag to the mouse. The dog threw the ball. ate(mouse, orange) gave(daisy, clock, mouse) ate(dog, apple) saw(mother, gave(mary, dog, hammer)) broke(dog, box) gave(woman, toy, mouse) gave(john, bag, mouse) threw(dog, ball) runs(dog) saw(john, walks(man, dog)) 2. Resulting NL-MR pairs are weighted and given to K RISP 1/2 1/4 1/5 1/3

29 KRISPER’s Training Algorithm contd. Daisy gave the clock to the mouse. Mommy saw that Mary gave the hammer to the dog. The dog broke the box. John gave the bag to the mouse. The dog threw the ball. ate(mouse, orange) gave(daisy, clock, mouse) ate(dog, apple) saw(mother, gave(mary, dog, hammer)) broke(dog, box) gave(woman, toy, mouse) gave(john, bag, mouse) threw(dog, ball) runs(dog) saw(john, walks(man, dog)) 3. Estimate the confidence of each NL-MR pair using the resulting parser

30 KRISPER’s Training Algorithm contd. Daisy gave the clock to the mouse. Mommy saw that Mary gave the hammer to the dog. The dog broke the box. John gave the bag to the mouse. The dog threw the ball. ate(mouse, orange) gave(daisy, clock, mouse) ate(dog, apple) saw(mother, gave(mary, dog, hammer)) broke(dog, box) gave(woman, toy, mouse) gave(john, bag, mouse) threw(dog, ball) runs(dog) saw(john, walks(man, dog)) 3. Estimate the confidence of each NL-MR pair using the resulting parser

31 KRISPER’s Training Algorithm contd. Daisy gave the clock to the mouse. Mommy saw that Mary gave the hammer to the dog. The dog broke the box. John gave the bag to the mouse. The dog threw the ball. ate(mouse, orange) gave(daisy, clock, mouse) ate(dog, apple) saw(mother, gave(mary, dog, hammer)) broke(dog, box) gave(woman, toy, mouse) gave(john, bag, mouse) threw(dog, ball) runs(dog) saw(john, walks(man, dog)) 4. Use maximum weighted matching on a bipartite graph to find the best NL-MR pairs [Munkres, 1957]

32 KRISPER’s Training Algorithm contd. Daisy gave the clock to the mouse. Mommy saw that Mary gave the hammer to the dog. The dog broke the box. John gave the bag to the mouse. The dog threw the ball. ate(mouse, orange) gave(daisy, clock, mouse) ate(dog, apple) saw(mother, gave(mary, dog, hammer)) broke(dog, box) gave(woman, toy, mouse) gave(john, bag, mouse) threw(dog, ball) runs(dog) saw(john, walks(man, dog)) 4. Use maximum weighted matching on a bipartite graph to find the best NL-MR pairs [Munkres, 1957]

33 KRISPER’s Training Algorithm contd. Daisy gave the clock to the mouse. Mommy saw that Mary gave the hammer to the dog. The dog broke the box. John gave the bag to the mouse. The dog threw the ball. ate(mouse, orange) gave(daisy, clock, mouse) ate(dog, apple) saw(mother, gave(mary, dog, hammer)) broke(dog, box) gave(woman, toy, mouse) gave(john, bag, mouse) threw(dog, ball) runs(dog) saw(john, walks(man, dog)) 5. Give the best pairs to K RISP in the next iteration, continue till converges

34 Corpus Construction There is no real-world ambiguous corpus yet available for semantic parsing to our knowledge We artificially obfuscated the real-world unambiguous corpus by adding extra distracter MRs to each training pair (Ambig-Geoquery) We also created an artificial ambiguous corpus (Ambig-ChildWorld) which more accurately models real-world ambiguities in which potential candidate MRs are often related

35 Ambig-Geoquery Corpus NL MR Start with the unambiguous Geoquery corpus

36 Ambig-Geoquery Corpus NL MR Insert 0 to  random MRs from the corpus between each pair MR

37 Ambig-Geoquery Corpus NL MR Form a window of width from 0 to  in either direction for each NL sentence MR

38 Ambig-Geoquery Corpus NL MR Form the ambiguous corpus MR

39 Ambig-ChildWorld Corpus Although Ambig-Geoquery corpus uses real- world NL-MR pairs, it does not model relatedness between potential MRs for each sentence, common in perceptual contexts Constructed a synchronous grammar [Aho & Ullman, 1972] to simultaneously generate artificial NL-MR pairs Uses 15 verbs and 37 nouns (people, animals, things), MRs are in predicate logic without quantifiers

40 Ambig-ChildWorld Corpus contd. Different perceptual contexts were modeled by choosing subsets of productions of the synchronous grammar This leads to subsets of verbs and nouns (e.g. only Mommy, Daddy, Mary) causing more relatedness among potential MRs For each such perceptual context, data was generated in a way similar to Ambig-Geoquery corpus

41 Ambiguity in Corpora Three levels of ambiguity were created by varying parameters  and  MRs per NL Level 125%50%25% Level 211%22%34%22%11% Level 36%13%19%26%18%12%6%

42 Methodology Performed 10-fold cross validation Metrics: Measured best F-measure across the precision-recall curve obtained using output confidence thresholds

43 Results on Ambig-Geoquery Corpus

44 Results on Ambig-ChildWorld Corpus

45 Future Work Construct a real-world ambiguous corpus and test this approach Combine this system with a vision-based system that extracts MRs from perceptual contexts

46 Conclusions We presented the problem of learning language semantics from ambiguous supervision This form of supervision is more representative of natural training environment for a language learning system We presented an approach that learns from ambiguous supervision by iteratively re-training a system for unambiguous supervision Experimental results on two artificial corpora showed that this approach is able to cope with ambiguities to learn accurate semantic parsers

47 Thank you! Questions??