1. Machine Reading as a Process of Partial Question-Answering Peter Clark and Phil Harrison Boeing Research & Technology June 2010

2. Overview  Machine Reading and Question-Answering  Approach  Algorithm  Preliminary Results  Summary

3. Machine Reading  Machine Reading =  A “holy grail” of AI  Constructing an inference-supporting representation from text  Connecting what is read with what is already known  Reader already knows something  Text is elaborating/deepening that knowledge Do I already know this? Can I interpret this as something that I know? Can I interpret some of this as something I know? Machine Reading

4. Do I already know this? Can I interpret this as something that I know? Can I interpret some of this as something I know? Do I already know this? Can I interpret this as something that I know? Can I interpret some of this as something I know? Question-Answering Machine Reading Any remainder = new knowledge Any remainder = failed query

5. Machine Reading Question-Answering Machine Reading Main insight: These are similar processes Can apply question-answering techniques to machine reading. Why is that important? Question-answering is precisely a technology for linking what is said (asked) with what is known. i.e., To read text T Ask: Is it true that T?

6. Overview  Machine Reading and Question-Answering  Approach  Algorithm  Preliminary Results  Summary

7. General Approach “The mitotic spindle consists of hollow microtubules.” “Does the mitotic spindle consist of hollow microtubules?” “Mitotic spindle has parts [hollow] microtubules” “Those microtubules are hollow” Text: Question: Partial Answer: New Knowledge: Knowledge has guided interpretation

8. General Approach “The mitotic spindle consists of hollow microtubules.” “Does the mitotic spindle consist of hollow microtubules?” “The mitotic spindle has parts [hollow] microtubules” “Those microtubules are hollow” Text: Question: Partial Answer: New Knowledge:..and identified the “anchor points” in the KB for new knowledge

9. General Approach “The mitotic spindle consists of hollow microtubules.” “Does the mitotic spindle consists of hollow microtubules?” “The mitotic spindle has parts [hollow] microtubules” “Those microtubules are hollow” Text: Question: Partial Answer: New Knowledge:

10. Pipelined (KB independent) NLP Word-Sense Disambiguation Semantic Role Labeling ? Topic in the KB During prophase, the cell… Parse, logical form

11. Interleaved Interpretation and Answering Topic in the KB During prophase, the cell… Logical Form

12. Interleaved Interpretation and Answering Topic in the KB During prophase, the cell… Logical Form

13. Interleaved Interpretation and Answering Topic in the KB Existing Knowledge During prophase, the cell… Logical Form

14. Interleaved Interpretation and Answering Topic in the KB During prophase, the cell… Logical Form

15. Interleaved Interpretation and Answering Topic in the KB Existing Knowledge During prophase, the cell… Logical Form

16. Interleaved Interpretation and Answering Topic in the KB During prophase, the cell… Logical Form

17. Interleaved Interpretation and Answering Topic in the KB Existing Knowledge During prophase, the cell… Logical Form

18. Interleaved Interpretation and Answering Topic in the KB Suppose this is the best we can do, interpreting text as existing knowledge During prophase, the cell… Logical Form

19. Interleaved Interpretation and Answering Topic in the KB Traditional NLP During prophase, the cell… Logical Form

20. Interleaved Interpretation and Answering Topic in the KB New Knowledge During prophase, the cell… Logical Form

21. Interleaved Interpretation and Answering Topic in the KB Extended KB During prophase, the cell… Logical Form

22. Interleaved Interpretation and Answering Topic in the KB Extended KB Word sense choices Semantic role choices Paraphrase rewrites During prophase, the cell… Logical Form

23. Some Possible Semantic Role Labels… “DNA synthesized by the polymerase” agent?location? means? KB

24. Some Possible Paraphrases (DIRT)… “spindle consists of microtubules” “microtubules are part of the spindle” “spindle is staffed by microtubules” “microtubules participate in the spindle” … KB

25. Overview  Machine Reading and Question-Answering  Approach  Algorithm  Preliminary Results  Summary

26. Knowledge Representation  Ontology:  ~400 biology concepts, ~400 general concepts  Axioms:  Mainly “Forall…exists…” axioms, e.g.,  “All eukaryotic cells contain a nucleus”  “Subevents of mitosis are prophase, metaphase, …”  Inference:  Reason about an instance of a concept  Conclusions apply to all instances of the concept (via UG)

27. Topics  Topic = the concept that a text describes  We assume a text is about a single topic  Topic could be identified using ML (we do it by hand)  Given topic, can find (some) expected “participants” from KB The centrosomes are pushed apart to opposite ends of the cell nucleus by the action of molecular motors acting on the microtubules. The nuclear envelope breaks downm allowing…. Topic: Prophase

28. Topics  Topic = the concept that a text describes  Participants = Individuals implied to exist given the topic  Can infer (some) participants using the KB Topic: Prophase KB Prophase The centrosomes are pushed apart to opposite ends of the cell nucleus by the action of molecular motors acting on the microtubules. The nuclear envelope breaks downm allowing…. → centrosome moves to the pole of a eukaryotic cell → nucleus, cytoplasm → nuclear membrane, etc. etc.

29. Topics  Topic = the concept that a text describes  Participants = Individuals implied to exist given the topic  Can infer (some) participants using the KB Topic: Prophase KB Prophase  Text provides information about participants The centrosomes are pushed apart to opposite ends of the cell nucleus by the action of molecular motors acting on the microtubules. The nuclear envelope breaks downm allowing…. → centrosome moves to the pole of a eukaryotic cell → nucleus, cytoplasm → nuclear membrane, etc. etc.

30. Algorithm  Identify the topic of the text  Parse and create initial “logical form” “The mitotic spindle consists of hollow microtubules.” "mitotic-spindle"(s), "consist"(c), "hollow"(h), "microtubule"(m), subject(c,s), "of"(c,m), modifier(m,h). 1. Setup Create representation of topic + (known) participants in KB 2. Search: repeat: interpret + (try to) prove parts of the LF until: as much proved as possible Interpret remainder (normal NLP) and add to KB Topic: Prophase

31. Y4:Mitotic-Spindle X0:Prophase Y0:Move Y1:Centrosome Y7:Microtubule Y3:Elongate Y2:Eukaryotic-Cell Y5:Pole subevent has-part has-region object has-part Y6:Create … … destination Create a representation of the topic in the KB “The mitotic spindle consists of hollow microtubules.”

32. Y4:Mitotic-Spindle X0:Prophase Y0:Move Y1:Centrosome Y7:Microtubule Y3:Elongate Y2:Eukaryotic-Cell Y5:Pole subevent has-part has-region object has-part Y6:Create … … destination "mitotic-spindle"(s), "consist"(c), "hollow"(h), "microtubule"(m), subject(c,s), "of"(c,m), mod(m,h). LF interpretation: “The mitotic spindle consists of hollow microtubules.” Generate Logical Form

33. Y4:Mitotic-Spindle X0:Prophase Y0:Move Y1:Centrosome Y7:Microtubule Y3:Elongate Y2:Eukaryotic-Cell Y5:Pole subevent has-part has-region object has-part Y6:Create … … destination "mitotic-spindle"(s), "consist"(c), "hollow"(h), "microtubule"(m), subject(c,s), "of"(c,m), mod(m,h). LF interpretation: “The mitotic spindle consists of hollow microtubules.” Interpret and (try) prove some part of the LF

34. "mitotic-spindle"(s), "consist"(c), "hollow"(h), "microtubule"(m), subject(c,s), "of"(c,m), mod(m,h). LF interpretation: X0:Prophase Y0:Move Y1:Centrosome Y7:Microtubule Y3:Elongate Y2:Eukaryotic-Cell Y5:Pole subevent has-part has-region object has-part Y6:Create … … destination Y4:Mitotic-Spindle isa(Y4,MSpindle), "consist"(c), "hollow"(h), "microtubule"(m), subject(c,Y4),"of"(c,m),mod(m,h) Bind a LF variable “The mitotic spindle consists of hollow microtubules.” Interpret and (try) prove some part of the LF

35. "mitotic-spindle"(s), "consist"(c), "hollow"(h), "microtubule"(m), subject(c,s), "of"(c,m), mod(m,h). LF interpretation: X0:Prophase Y0:Move Y1:Centrosome Y7:Microtubule Y3:Elongate Y2:Eukaryotic-Cell Y5:Pole subevent has-part has-region object has-part Y6:Create … … destination Y4:Mitotic-Spindle isa(Y4,MSpindle), "consist"(c), "hollow"(h), "microtubule"(m), subject(c,Y4),"of"(c,m),mod(m,h) “The mitotic spindle consists of hollow microtubules.” isa(Y4,MSpindle), "hollow"(h), "microtubule"(m), material(Y4,m), mod(m,h). ?  Interpret and (try) prove some part of the LF

36. LF interpretation: X0:Prophase Y0:Move Y1:Centrosome Y7:Microtubule Y3:Elongate Y2:Eukaryotic-Cell Y5:Pole subevent has-part has-region object has-part Y6:Create … … "mitotic-spindle"(s), "consist"(c), "hollow"(h), "microtubule"(m), subject(c,s), "of"(c,m), mod(m,h). isa(Y4,MSpindle), "consist"(c), "hollow"(h), "microtubule"(m), subject(c,Y4),"of"(c,m),mod(m,h) isa(Y4,MSpindle), "hollow"(h), "microtubule"(m), has-part(Y4,m), mod(m,h). destination Y4:Mitotic-Spindle “The mitotic spindle consists of hollow microtubules.” Interpret and (try) prove some part of the LF ?

37. LF interpretation: Y4:Mitotic-Spindle X0:Prophase Y0:Move Y1:Centrosome Y7:Microtubule Y3:Elongate Y2:Eukaryotic-Cell Y5:Pole subevent has-part has-region object has-part Y6:Create … … "mitotic-spindle"(s), "consist"(c), "hollow"(h), "microtubule"(m), subject(c,s), "of"(c,m), mod(m,h). isa(Y4,MSpindle), "consist"(c), "hollow"(h), "microtubule"(m), subject(c,Y4),"of"(c,m),mod(m,h) isa(Y4,MSpindle), "hollow"(h), "microtubule"(m), has-part(Y4,m), mod(m,h). Recognized Old Knowledge destination “The mitotic spindle consists of hollow microtubules.” Interpret and (try) prove some part of the LF

38. LF interpretation: Y4:Mitotic-Spindle X0:Prophase Y0:Move Y1:Centrosome Y7:Microtubule Y3:Elongate Y2:Eukaryotic-Cell Y5:Pole subevent has-part has-region object has-part Y6:Create … … "mitotic-spindle"(s), "consist"(c), "hollow"(h), "microtubule"(m), subject(c,s), "of"(c,m), mod(m,h). isa(Y4,MSpindle), "consist"(c), "hollow"(h), "microtubule"(m), subject(c,Y4),"of"(c,m),mod(m,h) isa(Y4,MSpindle), "hollow"(h), "microtubule"(m), has-part(Y4,m), mod(m,h). Recognized Old Knowledge destination isa(Y4,MSpindle), "hollow"(h), isa(Y7,Microtubule), has-part(Y4,Y7), modifier(Y7,h). “The mitotic spindle consists of hollow microtubules.” Interpret and (try) prove some part of the LF !

39. LF interpretation: X0:Prophase Y0:Move Y1:Centrosome Y3:Elongate Y2:Eukaryotic-Cell Y5:Pole subevent has-part has-region object Y6:Create … … "mitotic-spindle"(s), "consist"(c), "hollow"(h), "microtubule"(m), subject(c,s), "of"(c,m), mod(m,h). isa(Y4,MSpindle), "consist"(c), "hollow"(h), "microtubule"(m), subject(c,Y4),"of"(c,m),mod(m,h) isa(Y4,MSpindle), "hollow"(h), "microtubule"(m), has-part(Y4,m), mod(m,h). isa(Y4,MSpindle), "hollow"(h), isa(Y7,Microtubule), has-part(Y4,Y7), modifier(Y7,h). destination Y7:Microtubule has-part Y4:Mitotic-Spindle “The mitotic spindle consists of hollow microtubules.”

40. LF interpretation: X0:Prophase Y0:Move Y1:Centrosome Y3:Elongate Y2:Eukaryotic-Cell Y5:Pole subevent has-part has-region object Y6:Create … … "mitotic-spindle"(s), "consist"(c), "hollow"(h), "microtubule"(m), subject(c,s), "of"(c,m), mod(m,h). isa(Y4,MSpindle), "consist"(c), "hollow"(h), "microtubule"(m), subject(c,Y4),"of"(c,m),mod(m,h) isa(Y4,MSpindle), "hollow"(h), "microtubule"(m), has-part(Y4,m), mod(m,h). isa(Y4,MSpindle), "hollow"(h), isa(Y7,Microtubule), has-part(Y4,Y7), modifier(Y7,h). isa(Y4,MSpindle), isa(Y8,Hollow), isa(Y7,Microtubule), has-part(Y4,Y7), shape(Y7,Y8). Y4:Mitotic-Spindle has-part destination Y7:Microtubule Traditional NLP for the rest… “The mitotic spindle consists of hollow microtubules.”

41. LF interpretation: X0:Prophase Y0:Move Y1:Centrosome Y3:Elongate Y2:Eukaryotic-Cell Y5:Pole subevent has-part has-region object Y6:Create … … "mitotic-spindle"(s), "consist"(c), "hollow"(h), "microtubule"(m), subject(c,s), "of"(c,m), mod(m,h). isa(Y4,MSpindle), "consist"(c), "hollow"(h), "microtubule"(m), subject(c,Y4),"of"(c,m),mod(m,h) isa(Y4,MSpindle), "hollow"(h), "microtubule"(m), has-part(Y4,m), mod(m,h). isa(Y4,MSpindle), "hollow"(h), isa(Y7,Microtubule), has-part(Y4,Y7), modifier(Y7,h). isa(Y4,MSpindle), isa(Y8,Hollow), isa(Y7,Microtubule), has-part(Y4,Y7), shape(Y7,Y8). Y4:Mitotic-Spindle has-part Y8:Hollow shape New Knowledge destination Y7:Microtubule Add to the KB “The mitotic spindle consists of hollow microtubules.”

42. X0:Prophase Y0:Move Y1:Centrosome Y3:Elongate Y2:Eukaryotic-Cell Y5:Pole subevent has-part has-region object Y6:Create … … Y4:Mitotic-Spindle has-part Y8:Hollow shape New Knowledge destination Y7:Microtubule “The mitotic spindle consists of hollow microtubules.”

43. Overview  Machine Reading and Question-Answering  Approach  Algorithm  Illustration and Preliminary Results  Summary

44. Illustration “During prophase, chromosomes become visible, the nucleolus disappears, the mitotic spindle forms, and the nuclear envelope disappears. Chromosomes become more coiled and can be viewed under a light microscope. Each duplicated chromosome is seen as a pair of sister chromatids joined by the duplicated but unseparated centromere. The nucleolus disappears during prophase. In the cytoplasm, the mitotic spindle, consisting of microtubules and other proteins, forms between the two pairs of centrioles as they migrate to opposite poles of the cell. The nuclear envelope disappears at the end of prophase. This signals the beginning of the substage called prometaphase.” In all prophase events: The chromosome moves. The chromatids are attached by the centromere. The nucleolus disappears during the prophase. The mitotic spindle has parts the microtubule and the protein. The mitotic spindle is created between the centrioles in the cytoplasm. The centrioles move to the poles. The nuclear envelope disappears at the end. Something signals. Input Text + Topic (here, Prophase): Output Axioms (expressed in English):

45. Illustration “During prophase, chromosomes become visible, the nucleolus disappears, the mitotic spindle forms, and the nuclear envelope disappears. Chromosomes become more coiled and can be viewed under a light microscope. Each duplicated chromosome is seen as a pair of sister chromatids joined by the duplicated but unseparated centromere. The nucleolus disappears during prophase. In the cytoplasm, the mitotic spindle, consisting of microtubules and other proteins, forms between the two pairs of centrioles as they migrate to opposite poles of the cell. The nuclear envelope disappears at the end of prophase. This signals the beginning of the substage called prometaphase.” In all prophase events: The chromosome moves. The chromatids are attached by the centromere. The nucleolus disappears during the prophase. The mitotic spindle has parts the microtubule and the protein. The mitotic spindle is created between the centrioles in the cytoplasm. The centrioles move to the poles. The nuclear envelope disappears at the end. Something signals. Input Text: Output Axioms (expressed in English): Good interpretation using paraphrases

46. Illustration “During prophase, chromosomes become visible, the nucleolus disappears, the mitotic spindle forms, and the nuclear envelope disappears. Chromosomes become more coiled and can be viewed under a light microscope. Each duplicated chromosome is seen as a pair of sister chromatids joined by the duplicated but unseparated centromere. The nucleolus disappears during prophase. In the cytoplasm, the mitotic spindle, consisting of microtubules and other proteins, forms between the two pairs of centrioles as they migrate to opposite poles of the cell. The nuclear envelope disappears at the end of prophase. This signals the beginning of the substage called prometaphase.” In all prophase events: The chromosome moves. The chromatids are attached by the centromere. The nucleolus disappears during the prophase. The mitotic spindle has parts the microtubule and the protein. The mitotic spindle is created between the centrioles in the cytoplasm. The centrioles move to the poles. The nuclear envelope disappears at the end. Something signals. Input Text: Output Axioms (expressed in English): Useful New Knowledge

47. Illustration “During prophase, chromosomes become visible, the nucleolus disappears, the mitotic spindle forms, and the nuclear envelope disappears. Chromosomes become more coiled and can be viewed under a light microscope. Each duplicated chromosome is seen as a pair of sister chromatids joined by the duplicated but unseparated centromere. The nucleolus disappears during prophase. In the cytoplasm, the mitotic spindle, consisting of microtubules and other proteins, forms between the two pairs of centrioles as they migrate to opposite poles of the cell. The nuclear envelope disappears at the end of prophase. This signals the beginning of the substage called prometaphase.” In all prophase events: The chromosome moves. The chromatids are attached by the centromere. The nucleolus disappears during the prophase. The mitotic spindle has parts the microtubule and the protein. The mitotic spindle is created between the centrioles in the cytoplasm. The centrioles move to the poles. The nuclear envelope disappears at the end. Something signals. Input Text: Output Axioms (expressed in English): Good interpretation

48. Illustration “During prophase, chromosomes become visible, the nucleolus disappears, the mitotic spindle forms, and the nuclear envelope disappears. Chromosomes become more coiled and can be viewed under a light microscope. Each duplicated chromosome is seen as a pair of sister chromatids joined by the duplicated but unseparated centromere. The nucleolus disappears during prophase. In the cytoplasm, the mitotic spindle, consisting of microtubules and other proteins, forms between the two pairs of centrioles as they migrate to opposite poles of the cell. The nuclear envelope disappears at the end of prophase. This signals the beginning of the substage called prometaphase.” In all prophase events: The chromosome moves. The chromatids are attached by the centromere. The nucleolus disappears during the prophase. The mitotic spindle has parts the microtubule and the protein. The mitotic spindle is created between the centrioles in the cytoplasm. The centrioles move to the poles. The nuclear envelope disappears at the end. Something signals. Input Text: Output Axioms (expressed in English):  Not very useful

49. Illustration “During prophase, chromosomes become visible, the nucleolus disappears, the mitotic spindle forms, and the nuclear envelope disappears. Chromosomes become more coiled and can be viewed under a light microscope. Each duplicated chromosome is seen as a pair of sister chromatids joined by the duplicated but unseparated centromere. The nucleolus disappears during prophase. In the cytoplasm, the mitotic spindle, consisting of microtubules and other proteins, forms between the two pairs of centrioles as they migrate to opposite poles of the cell. The nuclear envelope disappears at the end of prophase. This signals the beginning of the substage called prometaphase.” In all prophase events: The chromosome moves. The chromatids are attached by the centromere. The nucleolus disappears during the prophase. The mitotic spindle has parts the microtubule and the protein. The mitotic spindle is created between the centrioles in the cytoplasm. The centrioles move to the poles. The nuclear envelope disappears at the end. Something signals. Input Text: Output Axioms (expressed in English):  Bad interpretation

50. A Preliminary Experiment  10 paragraphs (110 sentences) about prophase, from Web   114 logic statements created  23 (20%) fully known to the KB  27 (24%) partially new knowledge  64 (56%) completely new knowledge  Biologist ranked the statements (expressed in English) as:  c = correct; useful knowledge for the KB  q = questionable; not useful (meaningless, vague)  i = incorrect  

51. A Preliminary Experiment 100Incorrect 3881Questionable 251922Correct Fully new Mixture of known & new Fully known Statements that are:   “The membrane break down” Questionable due to poor rendering in English, not the original logic

52. A Preliminary Experiment 100Incorrect 3881Questionable 251922Correct Fully new Mixture of known & new Fully known Statements that are:   70% judged correct

53. A Preliminary Experiment 100Incorrect 3881Questionable 251922Correct Fully new Mixture of known & new Fully known Statements that are:   39% judged correct

54. A Preliminary Experiment  Is extracting and integrating some useful knowledge  Potentially useful as interactive tool 100Incorrect 3881Questionable 251922Correct Fully new Mixture of known & new Fully known Statements that are:  

55. Summary  Clearly only a first step  Simple KR, single parse, contradictions, noisy, …  But:  Interpretation guided by knowledge  Identifies the “hooks” for new knowledge  Is a “real” context for machine reading To read T, ask “Is it true that T?”