SAMI: Situational Awareness from Multimodal Input

1
SAMI Situational Awareness from Multi-modal Input

• Naveen Ashish

2
Talk Organization

• Why are we at RESCUE interested ?
• Situational Awareness (SA)
• Introduction
• System architecture
• Research challenges
• Expected outcomes and artifacts
• Extraction system demonstration

3
Team
Bhaskar Rao Mohan Trivedi Rajesh Hegde Sangho
Park Shankar Shivappa
Naveen Ashish Sharad Mehrotra Nalini
Venkatasubramanian Utz Westermann Dmitry
Kalashnikov Stella Chen Vibhav Gogate Priya
Govindarajan Ram Hariharan John Hutchinson Yiming
Ma Dawit Seid Jay Lickfett Chris Davision Quent
Cassen
Ron Eguchi Mike Mio
Jacob Green
4
Information from Various Sources
5
More Data ? More Information
6
Situational Awareness

• Wide variety of fields
• Beginning in mid-80s, accelerating thru 90s
• Fighter aircraft, ATM, Power plants,
  Manufacturing
• Definitions
• "the perception of elements in the environment
  along with a comprehension of their meaning and
  along with a projection of their status in the
  near future"
• "the combining of new information with existing
  knowledge in working memory and the development
  of a composite picture of the situation along
  with projections of future status and subsequent
  decisions as to appropriate courses of action to
  take"
• Situational awareness and decision making
• Areas
• Cognitive science
• Information processing
• Human factors

Knowing what is going on
7
Abstraction of Information
8
First-cut Architecture
Centered around EVENTS as fundamental abstractions
9
Research Areas
Event Modeling
Event Extraction
Disambiguation
GIS Querying
Location Uncertainty
Graph Analysis
10
Event Modeling

• What is an event ?
• Event Representation

11
Domain Knowledge
EVACUATION
IS-A
IS-A
AIR EVACUATION

• Captured as Ontologies

12
Event Extraction

• Long history of information extraction
• IR (MUC efforts)
• Web data extraction
• DARPA ACE
• Entities, Relations, Events
• Events in 2004
• Event extraction accuracy is still low
• SA Domain
• Stream of information
• Duplicated, ambiguous
• Reliability
• Conversations
• Modalities
• Text

13
Semantics Driven Approach

• Semantics Driven
• Challenges
• Framework
• Ontologies
• What semantics required for event extraction ?
• Application
• With NLP, ML techniques
• Performance
• SA specific
• Duplicates, reconciliation, temporal,
  conversations ..

14
Disambiguation
15
Disambiguation
16
Uncertainty is a Challenge
Report 1 ... a massive accident involving a
hazmat truck on I5-N between
Sand Canyon and Alton Pkwy ... Report 2 ... a
strange chemical smell on Rt133 between I405
and Irvine Blvd ...
Report 2

• point-location
• in terms of landmarks
• uncertain, not (x,y)
• reasoning on such data
• support all types of queries

Report 1
17
Implications of Uncertainty in Text

• How to model uncertainty?
• probabilistic model
• P(location report)
• e.g. report says near building A
• Queries
• cannot be answered exactly...
• use probabilistic queries
• all events P(location ? R report) gt 0
• SA requirements
• triaging capabilities
• fast response
• top-k
• threshold P(location ? R report) gt ?
• ?-RQ, k-RQ, k? -RQ
• How to map text to probabilities?
• use spatial ontologies

A
B
R
18
Graph Analysis

• GAAL
• Inherent spatio-temporal properties
• Graphs are powerful for querying and analysis

19
GIS Search
Current FGDC Search
20
GIS Search
Progressive Refinement of Data
21
Deliverables, Outcomes, Artifacts

• Vertical thrusts
• Event extraction system (TEXT)
• Disambiguation system
• GIS search system
• Overall system demonstration ?
• By-products
• Ontologies
• Computer science research areas

Databases
Semantic-Web
Information Retrieval
Intelligent Agents (AI)
22
Thank you !
http//sami.ics.uci.edu