1. SAMI: Situational Awareness from Multi-modal Input
Naveen Ashish

2. Talk Organization Why are we at RESCUE interested ?
Situational Awareness (SA)
Introduction
Information processing requirements
Envisioned system
Technical challenges
Expected outcomes and artifacts
Extraction System Demonstration
Team: Kemal Altintas, Stella Chen, Ram Hariharan, Yiming Ma, Dawit Seid
Naveen Ashish, Sharad Mehrotra, Nalini Venkatasubramanian
Amnon Meyers

3. Information from Various Sources
News, video, audio footage
Pushing “Human-as-sensor”
Emergency responders
People/Victims at disaster
GIS, satellite imagery, maps

4. SA and Decision Making SAMI
What areas should we start evacuating first ?
Have all medical supplies reached ?
Where are the fire personnel ?
SAMI

5. Situational Awareness
Wide variety of fields
Beginning in mid-80s, accelerating thru 90s
Fighter aircraft, ATM, Power plants, Manufacturing
Definitions
"the perception of elements in the environment along with a comprehension of their meaning and along with a projection of their status in the near future"
"the combining of new information with existing knowledge in working memory and the development of a composite picture of the situation along with projections of future status and subsequent decisions as to appropriate courses of action to take"
Situational awareness and decision making
Areas
Cognitive science
Information processing
Human factors
Knowing what is going on

6. Abstraction of Information
Awareness
Events
Multimodal Input:
Text, Audio, Video

7. First-cut Architecture
EVENT BASE
Querying and Analysis
Graph View
VISUALIZATION and USER INTERFACES
Spatial
Indexing
PDF Histogram
KNOWLEDGE: ONTOLOGIES
Text
Audio
Video
Internet
RAW DATA
EVENT
EXTRACTION
Topic Modeling
Ontologies, NLP
REFINEMENT
Disambiguation
Location

8. Research Areas Event Modeling Event Extraction Disambiguation
GIS Querying
Location Uncertainty
Graph Analysis

9. Event Modeling What is an event ? Event Representation RELIABILITY
PEOPLE EVACUATION
LOCATION
TIME
REPORT
TYPE
NAME
AGENCY
FROM TO
OPERATION
NUMBER

10. Domain Knowledge Captured as Ontologies THAILAND EVACUATION IS-A …….
SOUTHERN
REGION
IS-A
ROAD
EVACUATION
AIR
EVACUATION
PHUKET,
CHANGWAT
PHUKET
Captured as Ontologies

11. Event Extraction Long history of information extraction
IR (MUC efforts)
Web data extraction
DARPA ACE
Entities, Relations, Events
Events in 2004
Event extraction accuracy is still low
SA Domain
Stream of information
Duplicated, ambiguous
Reliability
Modalities
Text

12. Semantics Driven Approach
Challenges
Framework
Ontologies
What semantics required for event extraction ?
Application
With NLP, ML techniques
Performance
SA specific
Duplicates, reconciliation, temporal, …..

13. Disambiguation

14. Disambiguation

15. Uncertainty is a Challenge
Report 1: “... a massive accident involving a hazmat truck on
I5-N between Sand Canyon and Alton Pkwy ...”
Report 2: “... a strange chemical smell on Rt133 between I405
and Irvine Blvd ...”
Report 2
point-location
in terms of landmarks
uncertain, not (x,y)
reasoning on such data
support all types of queries
Report 1
There are many challenges when dealing with text.
One of them is uncertainty.
Often people describe locations in terms of landmarks, they do not give
you the exact (x,y) coordinates of the events. Those descriptions are uncertain.
For example, the first report says that the accident is somewhere on this
road and the second report tells that there is a strange smell somewhere on this road.
In reality, these two report might refer to the same accident that happened
say here.
So our goal is to be able to reason on top of such data.

16. Implications of Uncertainty in Text
How to model uncertainty?
probabilistic model
P(location | report)
e.g. report says “near building A”
Queries
cannot be answered exactly...
use probabilistic queries
all events: P(location  R | report) > 0
SA requirements
triaging capabilities
fast response
top-k
threshold: P(location  R | report) > 
-RQ, k-RQ, k -RQ
How to map text to probabilities?
use spatial ontologies A B R
There are several implications that arise from uncertainty in text.
First, we need to decide how to model spatial uncertainty. We use
a probabilistic model for that. Specifically, we are interested in
this conditional probability: given a report about an event, what
the location of that event is.
A report can describe an event location, for instance, as “near building A”.
Finally, observe that queries cannot be answered exactly (...)
The solution is to use probabilistic queries. For example a
probabilistic version of a range query is to retrieve all the objects
that have non zero probability to be inside a given range.
In addition, SA applications for crisis response have
specific requirement for queries. Namely, triaging
capability should be provided for filtering out only
important information. Also the solution should scale
to large dataset and be very efficient in terms of query
response time.
To support the desired functionality we will consider
two enhancements of probabilistic queries: top-k and
threshold. The top-k enhancement of a spatial query
returns only at most k elements from the result set
that have the highest probability to be in the result set.
The threshold enhancement returns onl ...
OK, probabilistic model is fine, but how do we map text into probabilities
in the first place. Our solution is to use spatial ontologies for that.

17. Graph Analysis GAAL Inherent spatio-temporal properties
Graphs are powerful for querying and analysis

18. GIS Search
Current FGDC Search

19. GIS Search
Progressive Refinement of Data

20. Deliverables, Outcomes, Artifacts
“Vertical” thrusts
Event extraction system (TEXT)
Disambiguation system
GIS search system
Overall system demonstration ?
“By-products”
Ontologies
Computer science research areas
Databases
Semantic-Web
Information Retrieval
Intelligent Agents (AI)

21. Thank you !