1. Reading Report Semantic Parsing: Sempre (自始至终)
瞿裕忠
南京大学计算机系

2. Articles
Jonathan Berant, Andrew Chou, Roy Frostig, Percy Liang: Semantic Parsing on Freebase from Question-Answer Pairs. EMNLP 2013:
Jonathan Berant, Percy Liang: Semantic Parsing via Paraphrasing. ACL (1) 2014:

3. Background Traditional semantic parsers have two limitations
Require annotated logical forms as supervision,
Operate in limited domains with a small number of logical predicates.
Recent work
Reducing the amount of supervision, OR
Increasing the number of logical predicates
The goal of this paper is to do both
learn a semantic parser without annotated logical forms that scales to the large number of predicates on Freebase.

4. Background: Problem Statement
Input
KB (RDF graph, Freebase)
A training set of question-answer pairs
Output: a semantic parser
Maps new questions to answers via latent logical forms (on Freebase).

5. Background (this work)
Map questions to answers via latent logical forms.

6. Background (related work)
Mapping phrases (attend) to predicates (education)
Learn the lexicon from per-example supervision (In limited-domain)
Use a combination of manual rules, distant supervision, and schema matching (On Freebase)
This work
Coarse alignment based on Freebase and a text corpus
A bridging operation that generates predicates compatible with neighboring predicates.
light verbs, e.g., “go”, and prepositions

7. Background (related work)
At the compositional level
Manually specify combination rules
Induce rules from annotated logical forms
This work
Define a few simple composition rules which over-generate and then use model features to simulate soft rules and categories. In particular, use POS tag features and features on the denotations of the predicted logical forms.

8. Framework Composition: Derivations are constructed recursively
A lexicon mapping natural language phrases to predicates
A small set of composition rules

9. Approach (Alignment)
r1 is a phrase r2 is a predicate with Freebase name s2

10. Approach (Bridging) What government does Chile have?
What is the cover price of X-men?
Who did Tom Cruise marry in 2006?

11. Approach (composition)
Each derivation d is the result of applying some number of intersection, join, and bridging operations.
To control this number, we define indicator features on each of these counts.
POS tag features
Denotation features

12. Experiment: Setup Datasets
FREE917 [Cai and Yates (2013)]: question with logic form
WebQuestions (new dataset): question-answer pairs
17 hand-written rules
Map question words (how many) to logic form (Count).
POS tagging and named-entity recognition
Entity: named entity, proper nouns or a sequence of at least two tokens.
Unary: a sequence of nouns
Binaries: a content word, a verb followed by either a noun phrase or a particle.

13. Experiment: Dataset WebQuestions
Used the Google Suggest API to obtain questions
begin with a wh-word and contain exactly one entity.
Started with the question “Where was Barack Obama born?”
Performed a breadth-first search over questions (nodes)
Queried the question excluding the entity, the phrase before the entity, or the phrase after it (1M questions)
100K were submitted to Amazon Mechanical Turk (AMT)
Workers answer the question using only the Freebase page
The answer: entity, values, or list of entities on the page
Allowed the user to filter the list by typing
6,642 were annotated identically by at least two AMT workers.

14. Experiment: Dataset

15. Experiment: Results WebQuestions
35% for the final test, 65% for development (80% for training, 20% for validation)
To map entities, built a Lucene index over the 41M Freebase entities
As a baseline, omit bridging, remove denotation and alignment features.
The accuracy on the test set of this system is 26.9%, whereas the full system obtains 31.4%.

16. Experiment: Results FREE917
917 questions involving 635 Freebase relations, annotated with lambda calculus forms. [Cai and Yates (2013)]
Converted questions into simple-DCS, executed them on Freebase and used the resulting answers to train and evaluate.
30% for the final test, 70% for development (80% for training, 20% for validation)
To map phrases to Freebase entities we used the manually-created entity lexicon used by Cai and Yates (2013), which contains 1,100 entries.
Accuracy: 62% > 59%

17. Experiment: Analysis Generation of binary predicates
In which comic book issue did Kitty Pryde first appear?
ComicBookFirstAppearance

18. Experiment: Analysis Feature variations
What number is Kevin Youkilis on the Boston Red Sox
How many people were at the 2006 FIFA world cup final?
PeopleInvolved, SoccerMatchAttendance

19. Experiment: Error analysis
Disambiguating entities in WebQestions is much harder (41M entities)
Where did the battle of New Orleans start?
Bridging often fail when the question’s entity is compatible with many binaries.
What did Charles Babbage make?
A wrong binary compatible with the type Person.
The system sometimes incorrectly draws verbs from subordinate clauses.
Where did Walt Disney live before he died?
the place of death of Walt Disney
Many possible derivations
What kind of system of government does the United States have?

20. The Software Publicly released datasets
the source code for SEMPRE, the semantic parser
ParaSempre

21. Semantic Parsing via Paraphrasing (ParaSempre)
Challenge in semantic parsing
What does X do for a living? , What is X’s profession?
What is the location of ACL 2014? (no entity in KB)
Out of 500,000 relations extracted by the ReVerb Open IE system (Fader et al., 2011), only about 10,000 can be aligned to Freebase (Berant et al., 2013).
An approach for semantic parsing based on paraphrasing
exploit large amounts of text not covered by the KB
factoid questions with a modest amount of compositionality

22. ParaSempre – the framework
Construct a manageable set of candidate logical forms
Generate canonical utterances for each logical form
Choose the one that best paraphrases the input utterance

23. Construct candidate logical forms
Use a set of templates
Find an entity in x and grow the logical form from that entity.
Resulting in 645 formulas per utterance On WebQestions

24. Generate canonical utterances
Use rules to generate utterances from the template p.e and R[p].e
Generate 1,423 utterances per input utterance On WebQuestions

25. Paraphrasing
Given pairs (c, z) of canonical utterances and logical forms
To score pairs (c, z) based on a paraphrase model.
Two paraphrase models that emphasize simplicity and efficiency.
Since for each question-answer pair, we consider thousands of canonical utterances as potential paraphrases.
Association model
Vector space model

26. Paraphrasing Association model
For each pair of utterances (x, c), we go through all spans of x and c and identify a set of pairs of potential paraphrases which we call associations.
Define features on each association; the weighted combination of these features yields a score.

27. Paraphrasing Association model: candidate associations
1.3 million phrase pairs constructed using the PARALEX corpus
Also those ones which contains token pairs sharing the same lemma, the same POS tag, or being linked through a derivation link on WordNet

28. Paraphrasing Association model
Unlike standard paraphrase detection and RTE systems, we use lexicalized features, firing approximately 400,000 features on WEBQUESTIONS.
Obtain soft syntactic rules, e.g.
the feature “JJ N ^ N” indicates that omitting adjectives before nouns is possible.
deleting pronouns is acceptable, while deleting nouns is not.
The model learns which associations are characteristic of paraphrases and which are not.

29. Paraphrasing Vector space model Good example
Construct vector representations of words. (50 dimensions)
Construct a vector for each utterance by simply averaging the vectors of all content words (nouns, verbs, and adjectives).
Estimate a paraphrase score for two utterances x and c via a weighted combination of the components of the vector representations
Good example
Where is made Kia car? (from WEBQUESTIONS)
What city is Kia motors a headquarters of?

30. Empirical evaluation

31. Empirical evaluation

32. Error analysis (ParaSempre)
Exact match of word, e.g. “work”.
What company did Henry Ford work for?
What written work novel by Henry Ford?
The employer of Henry Ford
Entity recognition
Where was the gallipoli campaign waged?
GalipoliCampaign.
Temporal information
Where did Harriet Tubman live after the civil war?

33. Related papers
Q. Cai and A. Yates. Large-scale semantic parsing via schema matching and lexicon extension. ACL 2013.
T. Kwiatkowski, E. Choi, Y. Artzi, and L. Zettlemoyer. Scaling semantic parsers with on-the-fly ontology matching. EMNLP 2013.
Yushi Wang, Jonathan Berant, Percy Liang. Building a Semantic Parser Overnight. Association for Computational Linguistics (ACL), 2015.

34. 致谢
欢迎提问！