1. Discovering Logical Knowledge for Deep Question Answering
Reporter: Xipeng Qiu
School of Computer Science
Fudan University
12/27/2018

2. QA4MR
Machine reading is automatic, unsupervised understanding of texts, which builds a bridge between natural languages and knowledge understandable by machines. Question answering for machine reading (QA4MR) is a task to answer questions by reading single documents with the background corpus.

3. A real example
Now, I just would like to finish o to tell you the little story about Avelile. This is Avelile. She goes with me wherever I go. I tell her story to everyone, because she represents one of millions of HIV/AIDS orphans. Avelile's mother had HIV virus. She died from AIDS-related illness. Avelile had the virus. She was born with the virus. And here she is at seven years old, weighing no more than a one year-old baby. At this point in her life, she's suffering with full-blown AIDS and had pneumonia. We met her in a hospital in the Eastern Cape and spent a whole afternoon with her - an adorable child.

4. A real example
Qestion:Why is Avelile suering from AIDS? Answers: A. Because she contracted the virus in hospital B. Because she was underweight C. Because she had pneumonia D. Because her mother transmitted the virus to her E. Because she is an orphan

5. Motivation
Most open-domain question answering systems achieve better performances with large corpora, such as Web, by taking advantage of information redundancy.
However, explicit answers are not always mentioned in the corpus, many answers are implicitly contained and can only be deducted by inference.
Some systems use limited man-made rules which can only cover very little knowledge of domains.
Other systems use lexical chain or eXtended WordNet (XWN) to construct the inference chain.
However, sometimes lexical information is far from enough to reason out the correct answers.

6. Our Approach
We propose an approach to discover logical knowledge for deep question answering, which automatically extracts knowledge in an unsupervised, domain-independent manner from background texts and reasons out implicit answers for the questions.
Firstly, we use semantic role labeling to transform natural language expressions to predicates in first-order logic.
Then we use association analysis to uncover the implicit relations among these predicates and build propositions for inference.
Since our knowledge is drawn from different sources, we use Markov logic to merge multiple knowledge bases without resolving their inconsistencies.

7. The architecture of deep question answering with logical knowledge

8. Example of Logic Transformation for Question and Answer
For the question “Why is Avelile suffering from AIDS?” , the logical form in an ideal situation is
Suffer(Avelile, AIDS)
One of the candidate answers “because her mother transmitted the virus to her” is transferred to
Transmit(Avelile’s Mother, AIDS, Avelile)
As this is the right answer, we wish that the fact
Transmit(Avelile’s Mother, AIDS, Avelile) → Suffer(Avelile, AIDS)
can be strongly supported by the knowledge base and the inference engine.

9. some rules that were discovered from AIDS
Bear(Child, Mother) −→ Pass(HIV, Somebody) Pass(Intercourse, Infection) −→ Transmit(Somebody, STD) Transmit(Mother, STD, Baby) −→ Have(STD) Transfer(Antibody, Child, Mother) −→ Attack(Antibody, SomeBody) Transmit(Blood, Infection) −→ Transmit(Mother, Something, Baby) Pass(Mother, something, Child) −→ Spread(Something) Share(Needle, Somebody) −→ Infect(Somebody, HIV) Have(Somebody, Vote) −→ Pass(Somebody, Option)

10. Another Example For the question and answer,
Q:“In which European cities has Annie Lennox performed?”
A:“Edinburgh and Oslo”
We wish to transfer them into
Perform(Annie Lennox) ∧ InCity(Edinburgh) ∧ InEuropean(Edinburgh)
Perform(Annie Lennox) ∧ InCity(Oslo) ∧ InEuropean(Oslo)
These two facts are supported by the knowledge base.

11. Discovering Deep Logical Knowledge from Background Corpus
Our system first extracts predicates from the background corpus.
Here, a predicate is composed of a verb and its corresponding arguments, which is the same to the predicate defined in first-order logic.
For example, ``\textit{Transmit}(sender, sent, sent-to)'' and ``\textit{Have}(somebody, something)'' are both predicates.
Then association analysis in data mining and statistical relevance in inductive logic programming are adopted to learn inference rules which construct the knowledge base.
After building this implicit knowledge base we transfer questions into logical forms. Then supported by explicit knowledge from the background corpus, WordNet and rules discovered, we will validate which is the best answer to the questions.
First of all, we focus on how to get useful implicit knowledge from the background corpus.

12. Finding Word Classes
We automatically acquire the hyponymy lexical relation from unrestricted texts. For all named entities, we regard them as instances of variables ``Somebody'' or ``Somewhere''.

13. Discovering Relations
Although there are structured resources such as WordNet, yet many more text resources are not structured including Web resources.
We use semantic role labeling (SRL) to label each part of a sentence with semantic role tags and predicates are generated after filling each blank of this predicate.

14. Discovering Relations
For example, given a sentence Her mother transmitted the HIV virus to her, that SRL identifies different arguments of the verb predicate and produces the following output:
[sender A0 Her mother][V:transmit transmitted][sent A1 the HIV virus][sent-to A2 to her].
We can format it into the predicate, which says that the mother of one person may transmit virus to him.
Transmit(MOTHER, VIRUS, PERSON)

15. Learning Inference Rules
However, due to the noises and lack of negative train examples, there may be some wrong or not exactly right inference rules.
So we use a probabilistic logical inference engine to bridge the gap between the questions and answers.
Markov logic networks (MLNs) [RD06] are one of the appropriate models.

16. Markov Logic Networks Syntax: First-order logic + Weights
Semantics: Templates for Markov nets
Inference: MCMC etc.al
Learning: ILP + Pseudo-likelihood

17. Markov Logic Networks
A logical KB is a set of hard constraints on the set of possible worlds
Let’s make them soft constraints: When a world violates a formula, It becomes less probable, not impossible
Give each formula a weight (Higher weight  Stronger constraint)

18. Definition A Markov Logic Network (MLN) is a set of pairs (F, w) where
F is a formula in first-order logic
w is a real number
Together with a set of constants, it defines a Markov network with
One node for each grounding of each predicate in the MLN
One feature for each grounding of each formula F in the MLN, with the corresponding weight w

19. inference
We use Tuffy [NRDS11] as the inference engine of MLNs in our system.
Tuffy is an open-source Markov Logic Network inference engine

20. Weight Learning
Due to the highly skewed distribution of predicates, some common predicates appear much more frequently than others and they are always an important prerequisite in propositions.
The appearance of highly frequent predicates leads to a very low level of mutual information between the pre- condition and the result.
So we use the following formula to calculate the weight of one rule discovered.

21. Experiments
We use the English corpus of QA4MRE task at CLEF2011 [PHF+11] to evaluate our system.
Top 3 approaches among all participants on QA4MRE 2011 task.
Baseline method of QA4MRE (Random Baseline): A method that randomly select an answer for each question.
Lexical Chain: Our previous system, which came 4th [CQH11] in QA4MRE 2011 task.

22. Evaluation Measure
is used in our experiment, which takes into account the option of not answering certain questions. nR: number of questions correctly answered nU: number of questions unanswered. n: total number of questions

23. Results
value of each question answering system is shown in the following table.
Due to different preprocess, the scores of QA4MRE2011 systems are just used as references and can not be directly compared.

24. Conclusions
In this paper, we propose an approach to discover logical knowledge for deep question answering.
To automatically extract knowledge in an unsupervised, domain-independent manner from background texts
To reason the implicit answers for the questions.
By adding explicit background knowledge and deep logical knowledge, our system substantially improve the performance of QA4MRE.

25. Question?
Thanks!