Reading Report Semantic Parsing: Sempre (自始至终) 瞿裕忠 南京大学计算机系

Articles Jonathan Berant, Andrew Chou, Roy Frostig, Percy Liang: Semantic Parsing on Freebase from Question-Answer Pairs. EMNLP 2013: 1533-1544 Jonathan Berant, Percy Liang: Semantic Parsing via Paraphrasing. ACL (1) 2014: 1415-1425

Background Traditional semantic parsers have two limitations Require annotated logical forms as supervision, Operate in limited domains with a small number of logical predicates. Recent work Reducing the amount of supervision, OR Increasing the number of logical predicates The goal of this paper is to do both learn a semantic parser without annotated logical forms that scales to the large number of predicates on Freebase.

Background: Problem Statement Input KB (RDF graph, Freebase) A training set of question-answer pairs Output: a semantic parser Maps new questions to answers via latent logical forms (on Freebase).

Background (this work) Map questions to answers via latent logical forms.

Background (related work) Mapping phrases (attend) to predicates (education) Learn the lexicon from per-example supervision (In limited-domain) Use a combination of manual rules, distant supervision, and schema matching (On Freebase) This work Coarse alignment based on Freebase and a text corpus A bridging operation that generates predicates compatible with neighboring predicates. light verbs, e.g., “go”, and prepositions

Background (related work) At the compositional level Manually specify combination rules Induce rules from annotated logical forms This work Define a few simple composition rules which over-generate and then use model features to simulate soft rules and categories. In particular, use POS tag features and features on the denotations of the predicted logical forms.

Framework Composition: Derivations are constructed recursively A lexicon mapping natural language phrases to predicates A small set of composition rules

Approach (Alignment) r1 is a phrase r2 is a predicate with Freebase name s2

Approach (Bridging) What government does Chile have? What is the cover price of X-men? Who did Tom Cruise marry in 2006?

Approach (composition) Each derivation d is the result of applying some number of intersection, join, and bridging operations. To control this number, we define indicator features on each of these counts. POS tag features Denotation features

Experiment: Setup Datasets FREE917 [Cai and Yates (2013)]: question with logic form WebQuestions (new dataset): question-answer pairs 17 hand-written rules Map question words (how many) to logic form (Count). POS tagging and named-entity recognition Entity: named entity, proper nouns or a sequence of at least two tokens. Unary: a sequence of nouns Binaries: a content word, a verb followed by either a noun phrase or a particle.

Experiment: Dataset WebQuestions Used the Google Suggest API to obtain questions begin with a wh-word and contain exactly one entity. Started with the question “Where was Barack Obama born?” Performed a breadth-first search over questions (nodes) Queried the question excluding the entity, the phrase before the entity, or the phrase after it (1M questions) 100K were submitted to Amazon Mechanical Turk (AMT) Workers answer the question using only the Freebase page The answer: entity, values, or list of entities on the page Allowed the user to filter the list by typing 6,642 were annotated identically by at least two AMT workers.

Experiment: Dataset

Experiment: Results WebQuestions 35% for the final test, 65% for development (80% for training, 20% for validation) To map entities, built a Lucene index over the 41M Freebase entities As a baseline, omit bridging, remove denotation and alignment features. The accuracy on the test set of this system is 26.9%, whereas the full system obtains 31.4%.

Experiment: Results FREE917 917 questions involving 635 Freebase relations, annotated with lambda calculus forms. [Cai and Yates (2013)] Converted questions into simple-DCS, executed them on Freebase and used the resulting answers to train and evaluate. 30% for the final test, 70% for development (80% for training, 20% for validation) To map phrases to Freebase entities we used the manually-created entity lexicon used by Cai and Yates (2013), which contains 1,100 entries. Accuracy: 62% > 59%

Experiment: Analysis Generation of binary predicates In which comic book issue did Kitty Pryde first appear? ComicBookFirstAppearance

Experiment: Analysis Feature variations What number is Kevin Youkilis on the Boston Red Sox How many people were at the 2006 FIFA world cup final? PeopleInvolved, SoccerMatchAttendance

Experiment: Error analysis Disambiguating entities in WebQestions is much harder (41M entities) Where did the battle of New Orleans start? Bridging often fail when the question’s entity is compatible with many binaries. What did Charles Babbage make? A wrong binary compatible with the type Person. The system sometimes incorrectly draws verbs from subordinate clauses. Where did Walt Disney live before he died? the place of death of Walt Disney Many possible derivations What kind of system of government does the United States have?

The Software Publicly released datasets the source code for SEMPRE, the semantic parser http://nlp.stanford.edu/software/sempre/ ParaSempre http://www-nlp.stanford.edu/software/sempre/

Semantic Parsing via Paraphrasing (ParaSempre) Challenge in semantic parsing What does X do for a living? , What is X’s profession? What is the location of ACL 2014? (no entity in KB) Out of 500,000 relations extracted by the ReVerb Open IE system (Fader et al., 2011), only about 10,000 can be aligned to Freebase (Berant et al., 2013). An approach for semantic parsing based on paraphrasing exploit large amounts of text not covered by the KB factoid questions with a modest amount of compositionality

ParaSempre – the framework Construct a manageable set of candidate logical forms Generate canonical utterances for each logical form Choose the one that best paraphrases the input utterance

Construct candidate logical forms Use a set of templates Find an entity in x and grow the logical form from that entity. Resulting in 645 formulas per utterance On WebQestions

Generate canonical utterances Use rules to generate utterances from the template p.e and R[p].e Generate 1,423 utterances per input utterance On WebQuestions

Paraphrasing Given pairs (c, z) of canonical utterances and logical forms To score pairs (c, z) based on a paraphrase model. Two paraphrase models that emphasize simplicity and efficiency. Since for each question-answer pair, we consider thousands of canonical utterances as potential paraphrases. Association model Vector space model

Paraphrasing Association model For each pair of utterances (x, c), we go through all spans of x and c and identify a set of pairs of potential paraphrases which we call associations. Define features on each association; the weighted combination of these features yields a score.

Paraphrasing Association model: candidate associations 1.3 million phrase pairs constructed using the PARALEX corpus Also those ones which contains token pairs sharing the same lemma, the same POS tag, or being linked through a derivation link on WordNet

Paraphrasing Association model Unlike standard paraphrase detection and RTE systems, we use lexicalized features, firing approximately 400,000 features on WEBQUESTIONS. Obtain soft syntactic rules, e.g. the feature “JJ N ^ N” indicates that omitting adjectives before nouns is possible. deleting pronouns is acceptable, while deleting nouns is not. The model learns which associations are characteristic of paraphrases and which are not.

Paraphrasing Vector space model Good example Construct vector representations of words. (50 dimensions) Construct a vector for each utterance by simply averaging the vectors of all content words (nouns, verbs, and adjectives). Estimate a paraphrase score for two utterances x and c via a weighted combination of the components of the vector representations Good example Where is made Kia car? (from WEBQUESTIONS) What city is Kia motors a headquarters of?

Empirical evaluation

Empirical evaluation

Error analysis (ParaSempre) Exact match of word, e.g. “work”. What company did Henry Ford work for? What written work novel by Henry Ford? The employer of Henry Ford Entity recognition Where was the gallipoli campaign waged? GalipoliCampaign. Temporal information Where did Harriet Tubman live after the civil war?

Related papers Q. Cai and A. Yates. Large-scale semantic parsing via schema matching and lexicon extension. ACL 2013. T. Kwiatkowski, E. Choi, Y. Artzi, and L. Zettlemoyer. Scaling semantic parsers with on-the-fly ontology matching. EMNLP 2013. Yushi Wang, Jonathan Berant, Percy Liang. Building a Semantic Parser Overnight. Association for Computational Linguistics (ACL), 2015.

致谢 欢迎提问！