SAMI: Situational Awareness from Multi-modal Input Naveen Ashish

Talk Organization Why are we at RESCUE interested ? Situational Awareness (SA) Introduction Information processing requirements Envisioned system Technical challenges Expected outcomes and artifacts Extraction System Demonstration Team: Kemal Altintas, Stella Chen, Ram Hariharan, Yiming Ma, Dawit Seid Naveen Ashish, Sharad Mehrotra, Nalini Venkatasubramanian Amnon Meyers

Information from Various Sources News, video, audio footage Pushing “Human-as-sensor” Emergency responders People/Victims at disaster GIS, satellite imagery, maps

SA and Decision Making SAMI What areas should we start evacuating first ? Have all medical supplies reached ? Where are the fire personnel ? SAMI

Situational Awareness Wide variety of fields Beginning in mid-80s, accelerating thru 90s Fighter aircraft, ATM, Power plants, Manufacturing Definitions "the perception of elements in the environment along with a comprehension of their meaning and along with a projection of their status in the near future" "the combining of new information with existing knowledge in working memory and the development of a composite picture of the situation along with projections of future status and subsequent decisions as to appropriate courses of action to take" Situational awareness and decision making Areas Cognitive science Information processing Human factors Knowing what is going on

Abstraction of Information Awareness Events Multimodal Input: Text, Audio, Video

First-cut Architecture EVENT BASE Querying and Analysis Graph View VISUALIZATION and USER INTERFACES Spatial Indexing PDF Histogram KNOWLEDGE: ONTOLOGIES Text Audio Video Internet RAW DATA EVENT EXTRACTION Topic Modeling Ontologies, NLP REFINEMENT Disambiguation Location

Research Areas Event Modeling Event Extraction Disambiguation GIS Querying Location Uncertainty Graph Analysis

Event Modeling What is an event ? Event Representation RELIABILITY PEOPLE EVACUATION LOCATION TIME REPORT TYPE NAME AGENCY FROM TO OPERATION NUMBER

Domain Knowledge Captured as Ontologies THAILAND EVACUATION IS-A ……. SOUTHERN REGION IS-A ROAD EVACUATION AIR EVACUATION PHUKET, CHANGWAT PHUKET Captured as Ontologies

Event Extraction Long history of information extraction IR (MUC efforts) Web data extraction DARPA ACE Entities, Relations, Events Events in 2004 Event extraction accuracy is still low SA Domain Stream of information Duplicated, ambiguous Reliability Modalities Text

Semantics Driven Approach Challenges Framework Ontologies What semantics required for event extraction ? Application With NLP, ML techniques Performance SA specific Duplicates, reconciliation, temporal, …..

Disambiguation

Disambiguation

Uncertainty is a Challenge Report 1: “... a massive accident involving a hazmat truck on I5-N between Sand Canyon and Alton Pkwy ...” Report 2: “... a strange chemical smell on Rt133 between I405 and Irvine Blvd ...” Report 2 point-location in terms of landmarks uncertain, not (x,y) reasoning on such data support all types of queries Report 1 There are many challenges when dealing with text. One of them is uncertainty. Often people describe locations in terms of landmarks, they do not give you the exact (x,y) coordinates of the events. Those descriptions are uncertain. For example, the first report says that the accident is somewhere on this road and the second report tells that there is a strange smell somewhere on this road. In reality, these two report might refer to the same accident that happened say here. So our goal is to be able to reason on top of such data.

Implications of Uncertainty in Text How to model uncertainty? probabilistic model P(location | report) e.g. report says “near building A” Queries cannot be answered exactly... use probabilistic queries all events: P(location  R | report) > 0 SA requirements triaging capabilities fast response top-k threshold: P(location  R | report) >  -RQ, k-RQ, k -RQ How to map text to probabilities? use spatial ontologies A B R There are several implications that arise from uncertainty in text. First, we need to decide how to model spatial uncertainty. We use a probabilistic model for that. Specifically, we are interested in this conditional probability: given a report about an event, what the location of that event is. A report can describe an event location, for instance, as “near building A”. Finally, observe that queries cannot be answered exactly (...) The solution is to use probabilistic queries. For example a probabilistic version of a range query is to retrieve all the objects that have non zero probability to be inside a given range. In addition, SA applications for crisis response have specific requirement for queries. Namely, triaging capability should be provided for filtering out only important information. Also the solution should scale to large dataset and be very efficient in terms of query response time. To support the desired functionality we will consider two enhancements of probabilistic queries: top-k and threshold. The top-k enhancement of a spatial query returns only at most k elements from the result set that have the highest probability to be in the result set. The threshold enhancement returns onl ... OK, probabilistic model is fine, but how do we map text into probabilities in the first place. Our solution is to use spatial ontologies for that.

Graph Analysis GAAL Inherent spatio-temporal properties Graphs are powerful for querying and analysis

GIS Search Current FGDC Search

GIS Search Progressive Refinement of Data

Deliverables, Outcomes, Artifacts “Vertical” thrusts Event extraction system (TEXT) Disambiguation system GIS search system Overall system demonstration ? “By-products” Ontologies Computer science research areas Databases Semantic-Web Information Retrieval Intelligent Agents (AI)

Thank you !