MAGIC Seen from the Perspective of RAGS Kathleen R. McKeown Department of Computer Science Columbia University.

Slides:



Advertisements
Similar presentations
Database Planning, Design, and Administration
Advertisements

Computational language: week 10 Lexical Knowledge Representation concluded Syntax-based computational language Sentence structure: syntax Context free.
Quranic Arabic Corpus Data Mining & Text Analytics By Ismail Teladia & Abdullah Alazwari.
 Plan, develop, and distribute course calendars, rules, and materials  Document enrollment, participation, and communications  Inform learners of progress.
Psycholinguistic what is psycholinguistic? 1-pyscholinguistic is the study of the cognitive process of language acquisition and use. 2-The scope of psycholinguistic.
RRL: A Rich Representation Language for the Description of Agent Behaviour in NECA Paul Piwek, ITRI, Brighton Brigitte Krenn, OFAI, Vienna Marc Schröder,
Automatic Prosodic Event Detection Using Acoustic, Lexical, and Syntactic Evidence Sankaranarayanan Ananthakrishnan, Shrikanth S. Narayanan IEEE 2007 Min-Hsuan.
Chapter 20: Natural Language Generation Presented by: Anastasia Gorbunova LING538: Computational Linguistics, Fall 2006 Speech and Language Processing.
Natural Language Generation Research Presentation Presenter Shamima Mithun.
S © Siemens Corporate Research, Inc. S I E M E N S C O R P O R A T E R E S E A R C H A Corpus-based Analysis for the Ordering of Clause Aggregation Operators.
1 Words and the Lexicon September 10th 2009 Lecture #3.
A Flexible Workbench for Document Analysis and Text Mining NLDB’2004, Salford, June Gulla, Brasethvik and Kaada A Flexible Workbench for Document.
Automatic Prosody Labeling Final Presentation Andrew Rosenberg ELEN Speech and Audio Processing and Recognition 4/27/05.
Dianne Bradley & Eva Fern á ndez Graduate Center & Queens College CUNY Eliciting and Documenting Default Prosody ABRALIN23-FEB-05.
1 THE USER INTERFACE Basic Principles. 2 Requirements System Design Detailed Design Implementation Installation & Testing Maintenance User Interface Model.
Information Retrieval and Extraction 資訊檢索與擷取 Chia-Hui Chang National Central University
CSE 730 Information Retrieval of Biomedical Data The use of medical lexicon in biomedical IR.
IS 247 Information Visualization and Presentation 10 May 2002 James Reffell Moryma Aydelott Jean-Anne Fitzpatrick The NewsHound Project Presents:
Lecture Nine Database Planning, Design, and Administration
The Software Product Life Cycle. Views of the Software Product Life Cycle  Management  Software engineering  Engineering design  Architectural design.
Enhance legal retrieval applications with an automatically induced knowledge base Ka Kan Lo.
Creation of a Russian-English Translation Program Karen Shiells.
Software Architecture premaster course 1.  Israa Mosatafa Islam  Neveen Adel Mohamed  Omnia Ibrahim Ahmed  Dr Hany Ammar 2.
Statistical Natural Language Processing. What is NLP?  Natural Language Processing (NLP), or Computational Linguistics, is concerned with theoretical.
These courseware materials are to be used in conjunction with Software Engineering: A Practitioner’s Approach, 6/e and are provided with permission by.
Lecture 1, 7/21/2005Natural Language Processing1 CS60057 Speech &Natural Language Processing Autumn 2005 Lecture 1 21 July 2005.
AQUAINT Kickoff Meeting – December 2001 Integrating Robust Semantics, Event Detection, Information Fusion, and Summarization for Multimedia Question Answering.
English Language Development Assessment (ELDA) Background to ELDA for Test Coordinator and Administrator Training Mike Fast, AIR CCSSO/LEP-SCASS March.
Modeling Language Acquisition with Neural Networks A preliminary research plan Steve R. Howell.
Chapter 10 Architectural Design
Introduction to Natural Language Generation
Chapter 9 Database Planning, Design, and Administration Sungchul Hong.
Database System Development Lifecycle © Pearson Education Limited 1995, 2005.
Overview of the Database Development Process
1 Chapter 14 Architectural Design. 2 Why Architecture? The architecture is not the operational software. Rather, it is a representation that enables a.
Some Advances in Transformation-Based Part of Speech Tagging
PropBank, VerbNet & SemLink Edward Loper. PropBank 1M words of WSJ annotated with predicate- argument structures for verbs. –The location & type of each.
Some Thoughts on HPC in Natural Language Engineering Steven Bird University of Melbourne & University of Pennsylvania.
ITEC224 Database Programming
1 Chapter 14 Architectural Design 2 Why Architecture? The architecture is not the operational software. Rather, it is a representation that enables a.
Mathematical Modeling and Formal Specification Languages CIS 376 Bruce R. Maxim UM-Dearborn.
A Cognitive Substrate for Natural Language Understanding Nick Cassimatis Arthi Murugesan Magdalena Bugajska.
1 SYS366 Lecture Visual Modeling and Business Use Case Diagrams.
Organizing Your Information
What you have learned and how you can use it : Grammars and Lexicons Parts I-III.
Introduction to Computational Linguistics
Introduction to Computational Linguistics Jay Munson (special thanks to Misty Azara) May 30, 2003.
Personalized Interaction With Semantic Information Portals Eric Schwarzkopf DFKI
Predicting Student Emotions in Computer-Human Tutoring Dialogues Diane J. Litman&Kate Forbes-Riley University of Pittsburgh Department of Computer Science.
Tool for Ontology Paraphrasing, Querying and Visualization on the Semantic Web Project By Senthil Kumar K III MCA (SS)‏
Flowsheet Analysis Design Concepts Introduction Acknowledgements Proposed Design By studying PICU flowsheets in the context of their actual use, we have.
NLG STEC Workshop April 20-21, 2007 Arlington, VA Nancy Green Univ. of North Carolina Greensboro, USA.
Collocations and Terminology Vasileios Hatzivassiloglou University of Texas at Dallas.
Performance Comparison of Speaker and Emotion Recognition
Hybrid Method for Tagging Arabic Text Written By: Yamina Tlili-Guiassa University Badji Mokhtar Annaba, Algeria Presented By: Ahmed Bukhamsin.
Oman College of Management and Technology Course – MM Topic 7 Production and Distribution of Multimedia Titles CS/MIS Department.
Modeling Security-Relevant Data Semantics Xue Ying Chen Department of Computer Science.
Pastra and Saggion, EACL 2003 Colouring Summaries BLEU Katerina Pastra and Horacio Saggion Department of Computer Science, Natural Language Processing.
Chapter 13 설계 개념 Architectural Design 임현승 강원대학교 Revised from the slides by Roger S. Pressman and Bruce R. Maxim for the book “Software Engineering: A Practitioner’s.
1 SYS366 Week 2 - Lecture 2 Visual Modeling & UML.
Software Architecture for Multimodal Interactive Systems : Voice-enabled Graphical Notebook.
AQUAINT Mid-Year PI Meeting – June 2002 Integrating Robust Semantics, Event Detection, Information Fusion, and Summarization for Multimedia Question Answering.
Design Evaluation Overview Introduction Model for Interface Design Evaluation Types of Evaluation –Conceptual Design –Usability –Learning Outcome.
1 These courseware materials are to be used in conjunction with Software Engineering: A Practitioner’s Approach, 5/e and are provided with permission by.
Chapter 9 Architectural Design. Why Architecture? The architecture is not the operational software. Rather, it is a representation that enables a software.
Columbia University Department of Computer Science Henning Schulzrinne 2001 NSF CISE/EIA RI PI’s Workshop.
Columbia University Department of Computer Science
Information Structure and Prosody
Chapter 9 Architectural Design.
Presentation transcript:

MAGIC Seen from the Perspective of RAGS Kathleen R. McKeown Department of Computer Science Columbia University

MAGIC Multimedia Abstract Generation of Intensive Care data Collaborators: Steven Feiner, Desmond Jordan Shimei Pan, James Shaw, Michelle Zhou Kris Concepcion, Liz Chen, Jeanne Fromer

Scenario Goal: provide post-operative information on bypass patients (CABG)  Prior to completion of surgery and before transport to Cardiac Intensive Care Unit (ICU)  Status needed for ICU nurse, cardiologist  Time critical

Issues for Language Generation  Conciseness: Coordinated speech and text that is brief but unambiguous  Coordination with other media: Modify wording and speech to coordinate references with graphical highlighting  Media specific tailoring:  Produce wording appropriate for spoken language  Use information from language generation to improve quality of synthesized speech

Status  Implemented prototype showing coordination between media for limited input  Text output for large numbers of input cases  Undergoing evaluation *now* in ICU  Runs on live data on a daily basis  5-10% error rate  Continuing research on effects of LG information on prosody, partial results

Principles  Early processes produce media independent representations  Representations use partial orderings in order to make early commitments where possible and retain flexibility  Both the speech and graphics content planner may add content and ordering constraints  Constraints on later decisions may be added early on (e.g., lexical choice)

Data Server and Filter (conceptual)  Input  18:25 DripsNorepinephrine  18:27 DripsNorepinephrine  18:29 Misc.Magnesium Sulfate  18:29 CardiacDefibrillated by surgeon  18:33:11100 (BP)51 (HR)  18:34:  Output  C-inanimate entity -> C-drug -> C-operating-room-medication ->C-Drip -> C-Norepinephrine  Top-level categories  C-state, C-event, C-entity (abstract, physical, organization, math)  Inferences  Hypotension: time, duration, drugs given

General Content Planner - SOAP (Rhetorical, semantic, conceptual)  Overview  Demographics  Name, Age, MRN, Gender, Doctor, Operation  Medical history  Lines  Therapy  Devices  Detail View  Drips (on leaving)  Induction info  Devices  Lab report  Timeline  Inferences  End values  Conclusions

Speech Content Planner - Satisfying Conciseness  Speech content planner groups information into sentences  Ms. Jones is an 80 year old, hypertensive diabetic female patient of Dr. Smith undergoing CABG.  Ms. Jones is an 80 year old, female patient of Dr. Smith undergoing CABG. She has a history of diabetes and hypertension.  To satisfy communicative goal to be concise, selects adjectives, prepositional phrases when possible.

Input to speech content planner - semantic propositions  X is-a patient  X has-property last name = Jones  X has-property age = 80 years  X has-property history = hypertension  X has-property history = diabetes  X has-property gender = female  X has-property surgery = CABG  X has-property doctor = Y  Y has-property last name = Smith

Forming Sentence Structure (Rhetorical, semantic, lexical, syntactic)  ((relation is-a) (arg1 ((item ((class name) (last-name “Jones”))))) (arg2 ((item ((class patient))))))  ((relation is-a) (arg1 ((item ((class name) (last-name “Jones”))))) (arg2 ((item ((class patient)) (premod ((history hypertension))))))

3 Types of Aggregation  Hypotactic aggregation: Given a set of propositions, can one be realized as a modifier?  Semantic aggregation: if a patient is on multiple drips and all devices, a patient has received massive cardiotonic therapy  Paratactic aggregation: Combine related propositions using conjunction and apposition

Coordination across media  Temporal media  Coordinate spoken references with highlighting of graphical references  Requires negotiation of ordering and duration of media actions

Negotiating Ordering  Spoken language generator has grammatical constraints on linear ordering  Graphics generator has spatial constraints on layout  Individual accounts of these constraints may result in an incoherent presentation

Ms. Jones is an 80 year old, diabetic, hypertensive female patient of Dr. Smith undergoing CABG.

Problems for Language Generation: Ordering  When to provide an ordering over references?  produce a partial ordering after word choice  How to select an ordering compatible with graphics?  produce several possibilities ordered by preference  How to communicate orderings with graphics?  maintain a mapping between strings and semantic objects

Media Negotiation (Conceptual, Semantic, Document) ä Speech components produce candidate partial orders 1.(< name age (* diabetes hypertension) gender surgeon operation) (< name age gender surgeon operation (* diabetes hypertension) 5 3. (< name age gender (* diabetes hypertension) surgeon operation) 4

Media Negotiation ä Graphics components produce candidate partial orders 1. (di (highlight demographics) ((<m) (subhighlight (mrn age gender))(subhighlight (medhistory))(subhighlight (surgeon operation)))10 2. (di (highlight demographics)(* (subhighlight (mrn age gender))(subhighlight (medhistory))(subhighlight (surgeon operation))) 7

CTS Architecture Prosody model Speech Corpus NLG System Prosody Realizer TTSTTS Machine Learning Input Other Source Text + Text + Structure Prosodic Annotated Text Sound Rules Rules

Focus of Research (Rhetorical, Semantic, Syntactic, Prosodic)  Build a prosody model for CTS using prosodic features (based on ToBI):  pitch accent, phrase accent, boundary tone, break index.  Features produced by LG  Syntactic structure, POS tags, Semantic boundaries, Concept  Informativeness, predictability (statistical models)  Abnormality, unexpectedness, sequential rhetorical relation

Mapping to RAGS  Data filter - conceptual  General Content Planner - rhetorical, semantic, conceptual  Speech Content Planner - rhetorical, semantic plus constraints on lexicalization, syntax  Lexical Chooser - semantic, lexical, syntactic  Media Coordination - semantic, conceptual, document  Syntactic Realization - semantic, syntactic  Prosody Realization - rhetorical, semantic, syntactic, prosodic

Acknowledgments This work was funded in part by  DARPA  NSF  ONR  New York State Center for Advanced Technology  NLM