1. Generating Natural Answers by Incorporating Copying and Retrieving Mechanisms in Sequence-to-Sequence Learning
Shizhu He, Cao liu, Kang Liu and Jun Zhao
National Laboratory of Pattern Recognition
Institute of Automation, Chinese Academy of Sciences
000

2. Outline
Task
Introduction
Our Proposed Approach
Experiments
000

3. Outline
Task
Introduction
Our Proposed Approach
Experiments
000

4. Natural Question Answering
Question answering (QA)
devote to providing exact answers, often in the form of phrases and entities
focus on analyzing questions, retrieving facts from text snippets or KBs, and predicting the answers through ranking and reasoning
Real-world environments
people prefer the correct answer replied with a more natural way
Siri will reply a natural answer “Jet Li is 1.64m in height.” for the question “How tall is Jet Li?”, rather than only answering one entity “1.64m”.

5. Natural Question Answering

6. For the question “Where was Tagore born?”
Natura-QA vs. Chatbot
Chatbots are able to generate natural replies
Chatbots could only generate fluent responses for ‘Chatting-type’messages
Chatbots are often unable to answer factual questions which need to interact with external KBs.
For the question “Where was Tagore born?”

7. Natural Question Answering
The system first needs to recognize the topic entity
Then extract multiple related facts
Finally generate the natural answer

8. Outline
Task
Introduction
Our Proposed Approach
Experiments
000

9. Generating natural answer engineering methods
Natural Language Processing (NLP) tools
POS tagging, parsing , etc.
Pattern engineering
“where was %entity from?” “%entity was born in %birthplace, %pronoun is %nationality citizen.”
Weaknesses:
Suffers from high costs of manual annotations for training data and patterns.
Low coverage that cannot flexibly deal with variable linguistic phenomena in different domains.

10. Generating natural answers end-to-end manner
This paradigm tries to consider question answering in an end-to-end framework.
The complicated QA process, including analyzing question, retrieving relevant facts from KB, and generating correct, coherent, natural answers, could be resolved jointly.
Challenges:
the words in a natural answer may be generated by different ways.
we even need to deal with some morphological variants(e.g. “Singapore” in KB but “Singaporean” in answer) .
Knowledge-driven & Data-driven
KBs are symbol systems
End-to-end models are numerical systems

11. Generating natural answers end-to-end manner
Words in a natural answer may be generated by different ways
The common words usually are predicted using a (conditional) language model
the major entities/phrases are selected from the source question
the answering entities/phrases are retrieved from the corresponding KB

12. Related Work GenQA (Yin et al., 2016)
is able to retrieve facts from KBs with neural models
cannot copy SUs from the question in generating answers
could not deal with complex questions which need to utilize multiple facts

13. Related Work CopyNet (Gu et al., 2016)
is able to copy words from the original source in generating the target
cannot retrieve SUs from external memory (e.g. KBs, Texts, etc.)

14. seq2seq
Vocabulary (e.g. 30,000 words)
an encoding RNN
an decoding RNN

15. Outline
Task
Introduction
Our Proposed Approach
Experiments
000

16. COREQA Incorporating Copying and Retrieving Mechanisms in QA
To generate natural answers for information inquired questions
recognize key topics in the question
extract related facts from KB
generate natural answer with question and the retrieved facts
generate a coherent reply
fusion those instance-level knowledge with some global-level “smooth” and“glue” words

17. COREQA Incorporating Copying and Retrieving Mechanisms in QA

18. Encoder Question Encoding : Bi-RNN
{ 𝒉 1 → ,…, 𝒉 𝐿 𝑋 → } { 𝒉 𝐿 𝑋 ← ,…, 𝒉 1 ← }
𝐌 𝑄 = 𝐡 𝑡 , 𝐡 𝑡 =[ 𝒉 𝑡 → , 𝒉 𝐿 𝑋 −𝑡+1 ← ]
𝐪=[ 𝒉 𝐿 𝑋 → , 𝒉 1 ← ]
Knowledge Base Encoding : Memory Network
Fact: (s, p, o)
𝐟=[𝐬,𝐩,𝐨]
𝐌 𝐾𝐵 = 𝒇 𝑖 ={ 𝒇 1 ,…, 𝒇 𝐿 𝐹 }

19. Decoder Answer words prediction
Predicts SUs based on a mixed probabilistic model of three modes, namely the predict-mode, the copy-mode and the retrieve-mode
State update
st, yt-1, ct  st
yt-1  [e(yt-1); rqt-1; rkbt-1]
Reading short-Memory MQ and MKB
attentive models
accumulated attentive vectors

20. Answer words prediction
three correlative output layer: shortlist prediction layer, question location copying layer and candidate-facts location retrieving layer
we have adopted the instance-specific vocabulary ： V∪ 𝑋 𝑄 ∪ 𝑋 𝐾𝐵
the probabilistic function for generating any target SU yt is a “mixture” model
p 𝑦 𝑡 𝒔 𝑡 , 𝑦 𝑡−1 , 𝑴 𝑄 , 𝑴 𝐾𝐵 =
𝑝 𝑝𝑟 𝑦 𝑡 𝒔 𝑡 , 𝑦 𝑡−1 , 𝒄 𝑡 ∙ 𝑝 𝑚 𝑝𝑟| 𝒔 𝑡 , 𝑦 𝑡−1 +
𝑝 𝑐𝑜 𝑦 𝑡 𝒔 𝑡 , 𝑦 𝑡−1 , 𝑴 𝑄 ∙ 𝑝 𝑚 𝑐𝑜| 𝒔 𝑡 , 𝑦 𝑡−1 +
𝑝 𝑟𝑒 𝑦 𝑡 𝒔 𝑡 , 𝑦 𝑡−1 , 𝑴 𝐾𝐵 ∙ 𝑝 𝑚 𝑟𝑒| 𝒔 𝑡 , 𝑦 𝑡−1

21. Answer words prediction
Predict-mode
𝑝 𝑝𝑟 𝑦 𝑡 ∙ = 1 𝑍 𝑒 𝜑 𝑝𝑟 ( 𝑦 𝑡 )
Copy-mode
𝑝 𝑐𝑜 𝑦 𝑡 ∙ = 1 𝑍 𝑗: 𝑄 𝑗 = 𝑦 𝑡 𝑒 𝜑 𝑐𝑜 ( 𝑦 𝑡 )
Retrieve-mode 𝑝 𝑟𝑒 𝑦 𝑡 ∙ = 1 𝑍 𝑗: 𝑓 𝑗 = 𝑦 𝑡 𝑒 𝜑 𝑟𝑒 ( 𝑦 𝑡 )

22. Outline
Task
Introduction
Our Proposed Approach
Experiments
000

23. Natural QA in Restricted Domain
The QA systems need to answer questions involving 4 concrete properties of birthdate (including year, month and day) and gender).
There are plenty of QA patterns which focus on different aspects of birthdate, for example, “What year were you born?” touches on “year”, but “When is your birthday?” touches on “month and day”.

24. Experimental Results

25. For unseen entities
We re-construct 2,000 new person entities and their corresponding facts about four known properties
Obtain Q-A pairs through matching the sampling patterns

26. Natural QA in Open Domain
Dataset ： CQA Question-Answering Pairs
Grounding Q-A pairs with ILP
(Yin et al., 2016) Q A T1 T2 T3
敢死队主演有谁？
李连杰 史泰龙 阿诺施瓦辛格
敢死队 编剧 主演
史泰龙
李连杰
梅西在什么级联赛
西班牙足球甲级联赛 梅西 职业 国籍 足球
西班牙
段誉是谁的儿子
段延庆和刀白凤 段誉 生父 父母
段延庆
刀白凤

27. Experimental Results
Automatic evaluation (AE)
Manual evaluation (ME)

28. Case Study

29. Conclusion
We propose an end-to-end system to generate natural answers through incorporating copying and retrieving mechanisms in sequence-to-sequence learning.
The sequences of SUs in the generated answer may be predicted from the vocabulary, copied from the given question and retrieved from the corresponding KB.

30. Email: shizhu.he@nlpr.ia.ac.cn
Thanks