QuASI: Question Answering using Statistics, Semantics, and Inference Marti Hearst, Jerry Feldman, Chris Manning, Srini Narayanan Univ. of California-Berkeley / ICSI / Stanford Univ

Outline Project Overview Three topics: Assigning semantic relations via lexical hierarchies From sentences to meanings via syntax From text analysis to inference using conceptual schemas

Main Goals Support Question-Answering and NLP in general by: Deepening our understanding of concepts that underlie all languages Creating empirical approaches to identifying semantic relations from free text Developing probabilistic inferencing algorithms

Two Main Thrusts Text-based: Use empirical corpus-based techniques to extract simple semantic relations Combine these relations to perform simple inferences “statistical semantic grammar” Concept-based: Determine language-universal conceptual principles Determine how inferences are made among these

Assigning Semantic Relations Using a Lexical Hierarchy

Noun Compounds (NCs) Any sequence of nouns that itself functions as a noun asthma hospitalizations asthma hospitalization rates health care personnel hand wash Technical text is rich with NCs Open-labeled long-term study of the subcutaneous sumatriptan efficacy and tolerability in acute migraine treatment.

NCs: 3 computational tasks (Lauer & Dras ’94) Identification Syntactic analysis (attachments) [Baseline [headache frequency]] [[Tension headache] patient] Semantic analysis Headache treatment treatment for headache Corticosteroid treatment treatment that uses corticosteroid

The lexical Hierarchy: MeSH Tree Structures 1. Anatomy [A] 2. Organisms [B] 3. Diseases [C] 4. Chemicals and Drugs [D] 5. Analytical, Diagnostic and Therapeutic Techniques and Equipment [E] 6. Psychiatry and Psychology [F] 7. Biological Sciences [G] 8. Physical Sciences [H] 9. Anthropology, Education, Sociology and Social Phenomena [I] 10. Technology and Food and Beverages [J] 11. Humanities [K] 12. Information Science [L] 13. Persons [M] 14. Health Care [N] 15. Geographic Locations [Z]

MeSH Tree Structures 1. Anatomy [A] Body Regions [A01] + Musculoskeletal System [A02] Digestive System [A03] + Respiratory System [A04] + Urogenital System [A05] + Endocrine System [A06] + Cardiovascular System [A07] + Nervous System [A08] + Sense Organs [A09] + Tissues [A10] + Cells [A11] + Fluids and Secretions [A12] + Animal Structures [A13] + Stomatognathic System [A14] (…..) Body Regions [A01] Abdomen [A01.047] Groin [A ] Inguinal Canal [A ] Peritoneum [A ] + Umbilicus [A ] Axilla [A01.133] Back [A01.176] + Breast [A01.236] + Buttocks [A01.258] Extremities [A01.378] + Head [A01.456] + Neck [A01.598] (….)

Mapping Nouns to MeSH Concepts headache recurrence C C headache pain C G breast cancer cells A C04 A11

Descent of Hierarchy Idea: Use the top levels of the lexical hierarchy to identify semantic relations Hypothesis: A particular semantic relations holds between all 2-word NCs that can be categorized by a category pair.

Linguistic Motivation Can cast NC into head-modifier relation, and assume head noun has an argument and qualia structure. (used-in): kitchen knife (made-of): steel knife (instrument-for): carving knife (used-on): putty knife (used-by): butcther’s knife

Distribution of Category Pairs

Classification decisions A01 N02 A01 N A01 N A01 N A01 N03 (W) J01 A01 (W) A02 F01 C04 H01 C04 H C04 H C04 H01.770

Levels of the classification decision Anatomy: 250 CPs 187 (75%) remain first level 56 (22%) descend one level 7 (3%) descend two levels Natural Science (H01): 21 CPs 1 (4%) remain first level 8 (39%) descend one level 12 (57%) descend two level Neoplasm (C04) 3 CPs: 3 (100%) descend one level

Evaluation Test decisions on “testing” set Count how many NCs that fall in the groups defined in the classification “rules” are similar with each other Accuracy: Anatomy: 91% accurate Natural Science: 79% Diseases: 100% Total: 89.6% via intra-category averaging 90.8% via extra-category averaging

Future work Analyze full spectrum of hierarchy NCs with > 2 terms [[growth hormone] deficiency] (purpose + defect) Other syntactic structures Non-biomedical words Other ontologies (e.g.,WordNet)?

From sentences to meanings via syntax A* Parsing & Stochastic HPSG

1. A* Parsing Goal: develop parsers that are Accurate – produce good parses Exact – find the models’ best parses Fast – seconds to parse long sentences Exhaustive Parsing – Slow but Exact e.g., chart Parsing, [Earley 70, Kay 80] Approximate Parsing – Fast but Inexact Beam Parsing, [Collins 97, Charniak 01] Best-First Parsing [Charniak et al. 98, etc.] Technology exists to get any two, but not all three of these goals

A* Search for Parsing Problem with uniform- cost parse search Even unlikely small edges have high score. We end up processing every small edge! Solution: A* Estimates Small edges have to fit into a full parse. The smaller the edge, the more the full parse will cost! Score =  Score =  +    

The Estimate Trade- off The more we specify, the better estimate of  we get… Fix outside size: Score = Add left tag: Score = Add right tag: Score = Entire context gives the exact best parse. Score = -18.1

A* Savings: Penn Treebank (SX-F filters more than Caraballo and Charniak 1998 while guaranteeing optimality, but less than Charniak et al. 1998)

2. Stochastic HPSG / Redwoods Treebank The Redwoods treebank is being built at Stanford as a resource for for deep NLP Provides full HPSG (Head-driven Phrase Structure Grammar) analyses, including semantic logical forms Current corpus is spoken dialog data (Verbmobil) parsed by robust broad coverage HPSG grammar Information at different levels of detail can be extracted from the treebank Precise deep grammatical analyses can be combined with probabilistic models Procedures are being developed for automatically updating the treebank

Basic Representation Levels Derivation tree of lexical items and constructions Phrase Structure Tree (S (NP “I”)(VP (V “am”) (ADJ_P “sorry”))) Underspecified MRS meaning representation <e1:BOOL:INDICATIVE*:PRESENT*:STRICT_NONPRF, {h2: pron_rel(x3:-*:STD_1SG:1SG:GENDER), h4: def_rel(x3,h5,h6,v7:BOOL),h8:_sorry_rel(e1,x3,v9:BOOL,v10:BOOL),h11: prpstn_rel(h12)}, {h5 QEQ h2, h12 QEQ h8}> Full HPSG signs for sentences are available IHCOMP “am” “I” BE_C_AM “sorry” SORRY_A1 SUBJH

Initial exploration: PCFG Results Accuracy Complete match (parse selection) accuracy Models with increasing parent node annotation PC FG Log lin ear Generative PCFG vs. loglinear modeling

In progress work: Using semantic forms Example: “I need to meet with you again”. I need to [[meet with you] again] (preferred) (ii) I [need to [meet with you] again] People use semantic information to disambiguate Building random field models over relations _need2_rel _meet_v_rel pron_rel with_rel again_rel pron_rel _need2_rel pron_rel with_rel again_relpron_rel _meet_v_rel

Concept-based Analysis From text analysis to inference using conceptual schemas

Inference and Conceptual Schemas Hypothesis: Linguistic input is converted into a mental simulation based on bodily-grounded structures. Components: Semantic schemas image schemas and executing schemas are abstractions over neurally grounded perceptual and motor representations Linguistic units lexical and phrasal construction representations invoke schemas, in part through metaphor Inference links these structures and provides parameters for a simulation engine

Conceptual Schemas Much is known about conceptual schemas, particularly image schemas However, this understanding has not yet been formalized We will develop such a formalism They have also not been checked extensively against other languages We will examine Chinese, Russian, and other languages in addition to English

Schema Formalism SCHEMA SUBCASE OF EVOKES AS ROLES : CONSTRAINTS :: :: |

A Simple Example SCHEMA hypotenuse SUBCASE OF line-segment EVOKES right-triangle AS rt ROLES Comment inherited from line-segment CONSTRAINTS SELF rt.long-side

Source-Path-Goal SCHEMA: spg ROLES: source: Place path: Directed Curve goal: Place trajector: Entity

Translational Motion SCHEMA translational motion SUBCASE OF motion EVOKES spg AS s ROLES mover s.trajector source s.source goal s.goal CONSTRAINTS before:: mover.location source after:: mover.location goal

Extending Inferential Capabilities Given the formalization of the conceptual schemas How to use them for inferencing? Earlier pilot systems Used metaphor and Bayesian belief networks Successfully construed certain inferences But don’t scale New approach Probabilistic relational models Support an open ontology

A Common Representation Representation should support Uncertainty, probability Conflicts, contradictions Current plan Probabilistic Relational Models (Koller et al.) DAML + OIL

An Open Ontology for Conceptual Relations Build a formal markup language for conceptual schemas We propose to use DAML+OIL as the base. Advantages of the approach Common framework for extending and reuse Closer ties to other efforts within AQUAINT as well as the larger research community on the Semantic Web. Some Issues Expressiveness of DAML+OIL Representing Probabilistic Information Extension to MetaNet, capture abstract concepts

DAML-I: An Image Schema Markup Language A basic type of schema <daml:subPropertyOf rdf:resource="&conc-rel;#role"/

Putting it all Together We have proposed two different types of semantics Universal conceptual schemas Semantic relations In Phase I they will remain separate However, we are exploring using PRMs as a common representational format In later Phases they will be combined