SAMI: Situational Awareness from Multimodal Input

1
SAMI Situational Awareness from Multi-modal Input

• Naveen Ashish

2
Talk Organization

• Why are we at RESCUE interested ?
• Situational Awareness (SA)
• Introduction
• Information processing requirements
• Envisioned system
• Technical challenges
• Expected outcomes and artifacts
• Extraction System Demonstration

Team Kemal Altintas, Stella Chen, Ram Hariharan,
Yiming Ma, Dawit Seid Naveen Ashish, Sharad
Mehrotra, Nalini Venkatasubramanian Amnon Meyers
3
Information from Various Sources
News, video, audio footage
Pushing Human-as-sensor
Emergency responders
People/Victims at disaster
GIS, satellite imagery, maps
4
SA and Decision Making
Where are the fire personnel ?
Have all medical supplies reached ?
What areas should we start evacuating first ?
SAMI
5
Situational Awareness

• Wide variety of fields
• Beginning in mid-80s, accelerating thru 90s
• Fighter aircraft, ATM, Power plants,
  Manufacturing
• Definitions
• "the perception of elements in the environment
  along with a comprehension of their meaning and
  along with a projection of their status in the
  near future"
• "the combining of new information with existing
  knowledge in working memory and the development
  of a composite picture of the situation along
  with projections of future status and subsequent
  decisions as to appropriate courses of action to
  take"
• Situational awareness and decision making
• Areas
• Cognitive science
• Information processing
• Human factors

Knowing what is going on
6
Abstraction of Information
7
First-cut Architecture
8
Research Areas
Event Modeling
Event Extraction
Disambiguation
GIS Querying
Location Uncertainty
Graph Analysis
9
Event Modeling

• What is an event ?
• Event Representation

10
Domain Knowledge
EVACUATION
IS-A
IS-A
AIR EVACUATION

• Captured as Ontologies

11
Event Extraction

• Long history of information extraction
• IR (MUC efforts)
• Web data extraction
• DARPA ACE
• Entities, Relations, Events
• Events in 2004
• Event extraction accuracy is still low
• SA Domain
• Stream of information
• Duplicated, ambiguous
• Reliability
• Modalities
• Text

12
Semantics Driven Approach

• Semantics Driven
• Challenges
• Framework
• Ontologies
• What semantics required for event extraction ?
• Application
• With NLP, ML techniques
• Performance
• SA specific
• Duplicates, reconciliation, temporal, ..

13
Disambiguation
14
Disambiguation
15
Uncertainty is a Challenge
Report 1 ... a massive accident involving a
hazmat truck on I5-N between
Sand Canyon and Alton Pkwy ... Report 2 ... a
strange chemical smell on Rt133 between I405
and Irvine Blvd ...
Report 2

• point-location
• in terms of landmarks
• uncertain, not (x,y)
• reasoning on such data
• support all types of queries

Report 1
16
Implications of Uncertainty in Text

• How to model uncertainty?
• probabilistic model
• P(location report)
• e.g. report says near building A
• Queries
• cannot be answered exactly...
• use probabilistic queries
• all events P(location ? R report) gt 0
• SA requirements
• triaging capabilities
• fast response
• top-k
• threshold P(location ? R report) gt ?
• ?-RQ, k-RQ, k? -RQ
• How to map text to probabilities?
• use spatial ontologies

A
B
R
17
Graph Analysis

• GAAL
• Inherent spatio-temporal properties
• Graphs are powerful for querying and analysis

18
GIS Search
Current FGDC Search
19
GIS Search
Progressive Refinement of Data
20
Deliverables, Outcomes, Artifacts

• Vertical thrusts
• Event extraction system (TEXT)
• Disambiguation system
• GIS search system
• Overall system demonstration ?
• By-products
• Ontologies
• Computer science research areas

Databases
Semantic-Web
Information Retrieval
Intelligent Agents (AI)
21
Thank you !