SAMI: Situational Awareness from Multimodal Input

1
SAMI Situational Awareness from Multi-modal Input

• Naveen Ashish

2
Talk Organization

• Why are we at RESCUE interested ?
• Situational Awareness (SA)
• Introduction
• System architecture
• Research challenges
• Expected outcomes and artifacts
• Extraction system demonstration

3
Team
Bhaskar Rao Mohan Trivedi Rajesh Hegde Sangho
Park Shankar Shivappa
Naveen Ashish Sharad Mehrotra Nalini
Venkatasubramanian Utz Westermann Dmitry
Kalashnikov Stella Chen Vibhav Gogate Priya
Govindarajan Ram Hariharan John Hutchinson Yiming
Ma Dawit Seid Jay Lickfett Chris Davision Quent
Cassen
Ron Eguchi Mike Mio
Jacob Green
4
Information from Various Sources
5
More Data ? More Information
6
Situational Awareness

• Wide variety of fields
• Beginning in mid-80s, accelerating thru 90s
• Fighter aircraft, ATM, Power plants,
  Manufacturing
• Definitions
• "the perception of elements in the environment
  along with a comprehension of their meaning and
  along with a projection of their status in the
  near future"
• "the combining of new information with existing
  knowledge in working memory and the development
  of a composite picture of the situation along
  with projections of future status and subsequent
  decisions as to appropriate courses of action to
  take"
• Situational awareness and decision making
• Areas
• Cognitive science
• Information processing
• Human factors

Knowing what is going on
7
Abstraction of Information
8
First-cut Architecture
Centered around EVENTS as fundamental abstractions
9
Research Areas
Event Modeling
Event Extraction
Disambiguation
GIS Querying
Location Uncertainty
Graph Analysis
10
Event Modeling

• What is an event ?
• Event Representation

11
Domain Knowledge
EVACUATION
IS-A
IS-A
AIR EVACUATION

• Captured as Ontologies

12
Event Extraction

• Long history of information extraction
• IR (MUC efforts)
• Web data extraction
• DARPA ACE
• Entities, Relations, Events
• Events in 2004
• Event extraction accuracy is still low
• SA Domain
• Stream of information
• Duplicated, ambiguous
• Reliability
• Conversations
• Modalities
• Text

13
Focus in RESCUE

• Specific types of input sources
• Regular unstructured text
• Such as online news stories
• Transcribed conversations and communications
• Particular structures and segments that can be
  exploited
• Approach
• Event extraction
• Embellished information extraction
• Determining and exploiting linkages
• Eg between locations and events

14
Disambiguation
15
Disambiguation
16
Uncertainty is a Challenge
Report 1 ... a massive accident involving a
hazmat truck on I5-N between
Sand Canyon and Alton Pkwy ... Report 2 ... a
strange chemical smell on Rt133 between I405
and Irvine Blvd ...
Report 2

• point-location
• in terms of landmarks
• uncertain, not (x,y)
• reasoning on such data
• support all types of queries

Report 1
17
Implications of Uncertainty in Text

• How to model uncertainty?
• probabilistic model
• P(location report)
• e.g. report says near building A
• Queries
• cannot be answered exactly...
• use probabilistic queries
• all events P(location ? R report) gt 0
• SA requirements
• triaging capabilities
• fast response
• top-k
• threshold P(location ? R report) gt ?
• ?-RQ, k-RQ, k? -RQ
• How to map text to probabilities?
• use spatial ontologies

A
B
R
18
Graph Analysis

• GAAL
• Inherent spatio-temporal properties
• Graphs are powerful for querying and analysis

19
GIS Search
Current FGDC Search
20
GIS Search

• Areas in RESCUE Focus
• GIS search and integration
• GIS deep web data
• Contextual GIS search
• Probabilistic search framework

21
Predictive Modeling and Inference

• Anomalous event detection from sensor data
• Highway traffic sensor data
• Model based estimation
• Eg estimate of number of people in particular
  area at particular time

22
Artifacts

• Integrated Information Dashboard
• Manifestation in City of Ontario Emergency
  Information Portal
• Web-based portal for citizens
• Integrated information, map-based, alerts,
  messages, shelter status, ..
• Multimodal reconnaissance system
• ImageCat UCI
• Real-time multimodal information capture and
  ingestion system

23
Thank you !
http//sami.ics.uci.edu