Situation Awareness Telcordias E2A Architecture and Three Case Studies

1
Situation AwarenessTelcordias E2A Architecture
and Three Case Studies

• Dimitrios Georgakopoulosdimitris_at_research.telcord
  ia.com
• EPS, SF, November, 2006

2
Awareness

• Awareness is information packages (complex event
  objects, their pedigree, and related evidence)
  that are highly relevant to the situated needs of
  a user/event consumer
• Contextual relevancy
• Events must be cast in terms of concepts (e.g.,
  space, time, objects) familiar to the user
• Situational relevancy
• Delivered events must help each user perform the
  specific activities he/she is working on or is
  responsible for
• Temporal relevancy
• Events must be delivered in timely fashion to
  permit effective response

3
Events to Awareness Concept of Operations
Event Processing System
Capture context info
Author awarenessspecifications
4
Telcordias Events to Awareness Architecture
(E2A)
Awareness Specifications
Event Ontology
Routing Task Specifications
Content Routing and Coordination
Awareness Computation (AC)
Event Contextualization (EC)
Awareness
PrimitiveEvents
Contextualized Events
Axtionable Events (i.e. alerts task requests)

Users
Event Extraction Analysis (EA)
Proactive Event analysis Tasking
Event repositiry

• Continuous stream processing of events for real
  time event detection ?
• Event Subscriptions and tasks

5
E2A Component responsibilities

• Event contextualization
• Injects primitive events
• Contextualizes and fuses events
• Awareness Computation
• Utilizes user-specified awareness specifications
  to compute complex events continuously and
  incrementally
• Proactively seek missing events
• Coordination
• Manages alert and tasking interactions with
  end-users
• Manages tasking of event sources
• Application context(s), event ontology, awareness
  specifications, and task specifications
• Permits application-specific customization

6
Situation Awareness Case Studies

• Complex event sensing
• Surveillance
• Critical Infrastructure protection
• Reconnaissance
• Broadcast news analysis
• UAVs/UASs
• Coordination and adaptation
• Intelligence gathering involving collaboration of
  large multi-organizational teams
• Disaster/crisis mitigation
• at a large scale
• Blue Force Tracking (DoDs Net-Centric Data
  Strategy)

7
The Surveillance Problem
8
Providing Situation Awareness in Video
Surveillance

• Provide situation awareness by automatically
  delivering alerts and related evidence to the
  appropriate users
• Situation understanding involves determining the
  causes of an alert
• Supports situation understanding via event drill
  down
• Users can view constituent events and evidence

9
Surveillance Case Study

• Event sources
• Video cameras, IR, radar, acoustic, images
• RFID readers, badge scanners, biometric
• People
• Surveillance case study characteristics
• Video, sound, and images must be analyzed to
  extract events
• Event extraction and analysis by far the
  costliest operation and this makes resource
  optimization hard
• Events emerge over time and space
• Out of order events are typical due to analysis
  overhead
• To provide situation awareness complex events
  must be mapped into the context of the specific
  facility/retain under surveillance (i.e., must be
  re-contextualized form the context of the
  specific sensors to the context understood by the
  users)
• Windows do not make much sense
• Events are often uncertain due to the complexity
  of the activity they report on (e.g., human
  behavior)
• Events must be detected in human real-time to
  enable responce to security threats Situational
  relevancy
• ..

10
The Intelligence Gathering Problem

• Event-driven collaboration of large,
  multi-organizational teams using CT analysis
  tools and operating in dynamically changing
  situations
• Reduce information overload, and improve
  decision-making
• Real time enterprise adaptation as the situation
  evolves

11
Intelligence Gathering Case Study

• Event sources
• Information/knowledge sources (e.g., open sources
  in the web), people
• Policies, processes, resources,
• Analysis algorithms (e.g., text analysis,
  evidential reasoning)
• Intelligence Gathering case study characteristic
• Events are typically heterogeneous
• Events must be mapped and evaluated into many
  different contexts reflecting jurisdictions,
  organizations, teams, and activities
• To determine compliance with a policy defined in
  another context
• To determine whether to start or adapt a process
  defined in a different context
• Out of order events due to analysis overhead and
  human decision making
• Events are often uncertain due to the complexity
  of the activities monitored (e.g., human
  behavior) and due to gaps in available
  information
• Events must be detected in human real-time to
  be able to respond to threats
• Event-driven process adaptation is common

12
A Context Network for Intelligence Gathering
Federal
Relations
Policy resource flow
Event flow
DHS
1
n
Policies 1 - Federal Search Warrant 2 -
FBI Affidavit 3 - NJ Search Warrant 4 - DHS
Notification 5 Information sharing
FBI
4
2
Texas
2
3
2
NJ
5
6
k
Activities and processes 1 - CBP Admission 2 -
DHS Notification 3 - Search Warrant 4 - Database
Search 5 - Investigation 6 - Event subscription
3
Task force
4
Austin
CBP
5
Events/Resources 1 - Person enters the US 2 -
Group active in the US 3 - Person belongs to
group 4 Person belongs to active group
in the US
m
1
1
Mary
Bob
Carol
Yanni
John
Alice
Xavier
3
4
13
Providing Situation Awareness in Intelligence
Gathering

• Situation awareness
• Teamwork awareness
• Ongoing policy compliance
• Dynamic adaptation to reflect changes in the
  events
• Process adaptation
• Context net adaptation

14
Enabling Net-Centricity ? Data Strategy
The Department of Defense Strategy To move from
privately owned and stored data in disparate
networks and within legacy systems/applications
to an enterprise information environment where
authorized known and authorized unanticipated
users can access any information and can post
their contributions for enterprise-wide access.
Producer and Developer
Consumer
Consumer
Producer
Ubiquitous Global Network
System 1 Data
Security Services (e.g., PKI, SAML)
Metadata Catalogs
System 2 Data
Shared Data Space
Enterprise Community Services
. . .
Application Services (e.g., Web)
Metadata Registries
System N Data
Developer

• From Producer-centric
• Multiple calls to find data
• Private data only supports planned consumers
• Data translation needed for understanding when
  pulled from multiple sources

• To Consumer-centric
• Data is visible, accessible and understandable
• Shared data supports planned and unplanned
  consumers
• Shared meaning of the data enables understanding

15
B A R R I E R B A R R I E R B A R R I
E R B A R R I E R
Barriers to Identifying, Accessing and
Understanding Data
What data exists? How do I access the
data? How do I know this data is what I
need? How can I tell someone what data I need?
How do I share my data with others? How do
I describe my data so others can understand
it?
User knows data exists and can access it but may
not know how to make
use of it due to lack of
under- standing of
what data represents
?
User is unaware this data exists
User knows this data existsbut cannot access it
because of
organizational and/or
technical barriers
Organization C
Organization A
Organization B
Data Strategy Approach Communities of
Interest, Metadata Registry
Data Strategy Approach Discovery Metadata
Data Strategy Approach Web Enabling,
Web-service Enabling
16
Publishing and Subscribing of Data
ServicesSupporting Both Known and Unanticipated
Authorized Users
System B
Data exchanged across engineered, well-defined
interfaces
System A
Known User of System A Data
Publish Structural and Semantic Metadata
Publish Data and Services
All Data Assets are Tagged with DoD Discovery
Metadata Specification (DDMS) Metadata
Publish Discovery Metadata
DoD Metadata Registry
Pull Structural and Semantic Metadata
Pull Data
DoD Discovery Catalogs
Query Catalogs and Registry
DoD Service Registry
System X
Shared Space
Leverages Service Oriented Architecture
Unanticipated Authorized User of System A Data
17

• Thank you for your attention!
• Dimitrios Georgakopoulos (dimitris_at_
  research.telcordia.com)

18
Backup Slides
19
Telcordias Events to Awareness Architecture
(E2A)
Awareness Specifications
Event Ontology
Routing Task Specifications
Content Routing and Coordination
Awareness Computation (AC)
Event Contextualization (EC)
Awareness
PrimitiveEvents
Contextualized Events
Axtionable Events (i.e. alerts task requests)

Users
Event Extraction Analysis (EA)
Proactive Event analysis Tasking
Event repository

• Continuous stream processing of events for real
  time event detection ?
• Event Subscriptions and tasks

20
Event Contexts and Context Management

• A Context typically contain information about
• Entities (e.g., actors or objects or interest)
• Activities and state changes of the entities
• Time interval of those activities and state
  changes
• Spatial coordinates in which the entities are
  situated
• Relationships of entities and activities to other
  contexts
• Contexts contain both current and historical info
• Context management
• E2A permits the initial modeling of one or more
  application specific contexts the relationships
  between them

21
A Simple Context for Surveillance
Facility context dynamically correlates and
tracks events from multiple cameras

• Facility Space Hierarchy
• Spaces are organized into a containment hierarchy
  with the rooms interconnected by portals
• Site-specific attributes e.g., name, secure,
  public, etc.
• Identities
• Partial information on specific people who may
  use the facility
• Site-specific attributes employee, security
  clearance,group, etc.
• Entities that move about the facility over time
• Usually people, though the idea extends to
  portable objects, like brief cases and documents
• Have a source-independent sequence of locations
  (supported by object tracking) of how the it
  changed positions over time
• Identity of the movable object may be known with
  some degree of certainty
• Pedigree information concerning the above

22
Event Contextualization

• Steps performed upon receipt of a primitive
  event
• Correlate event parameters and event source
  metadata with the information of the target and
  other related contexts
• Incrementally fuse the primitive event with the
  info already present in the context
• Incrementally publish the resulting
  contextualized events to its subscribers
• Example When a person enter a room in a
  facility, the location of the person is updated
  in the facility context and fused with the
  location of the camera

23
Telcordias Events to Awareness Architecture
(E2A)
Awareness Specifications
Event Ontology
Routing Task Specifications
Content Routing and Coordination
Awareness Computation (AC)
Event Contextualization (EC)
Awareness
PrimitiveEvents
Contextualized Events
Axtionable Events (i.e. alerts task requests)

Users
Event Extraction Analysis (EA)
Proactive Event analysis Tasking
Event repository

• Continuous stream processing of events for real
  time event detection ?
• Event Subscriptions and tasks

24
E2A Component responsibilities

• Event contextualization
• Injects primitive events
• Contextualizes and fuses events
• Awareness Computation
• Utilizes user-specified awareness specifications
  to compute complex events continuously and
  incrementally
• Proactively seek missing events
• Coordination
• Manages alert and tasking interactions with
  end-users
• Manages tasking of event sources
• Application context(s), event ontology, awareness
  specifications, and task specifications
• Permits application-specific customization

25
Awareness Specification

• VEAS-provided customization permits users to
  specify
• What types of events are of interest
• How to detect them
• When
• Where
• Which method to use
• Who should be alerted
• What/how event evidence and pedigree should be
  presented to each user

26
Event Ontology

• E2A surveillance ontology defines what type of
  events are of interest
• Event types are defined formally in OWL
• Existing event ontologies can be imported and
  used
• New event ontologies can be created and existing
  ones can be modified via Protégé to provide
  site-specific and situation-specific
  customizations
• Ontology provides an agreement about situation-
  and site-specific events of interest
• Example ZoneVisit
• Supported by Protégé, Awareness Computation

27
Awareness Specification (How Event Patterns are
Specified)

• Specifications
• Build from interconnected event operators
• Example Gales desk monitor detects if an
  object has been taken from her desk during her
  absence
• Operators
• Perform processing on events
• Examples generic filter, custom set difference
  Anybody but owner in target office
• Interconnections define contracts
• Specify the event flow between operators
• Define event types of the flowing events
• VEAS users author interconnections by utilizing
  event types defined in the surveillance ontology
• Example ZoneVisit event type flows from Owner
  in target office to Anybody but owner in target
  office

28
Core Awareness Operator Classes

• Contextualized event operators
• Subscribe to contextualized events and can be
  customized to filter such events
• Alert delivery operators
• submit alerts requests (by issuing actionable
  events) to E2As Coordination component
• Proactive event production operators
• submit task requests (by issuing actionable
  events) to E2As Coordination component
• Stream processing operators
• OR computes a union of its input streams
• Difference computes a set of difference of input
  streams
• Relational algebra operators
• Filtering culling of uninteresting events
• Joining combines related events from multiple
  sources into a composite event
• Grouping and aggregation regrouping and
  aggregations of events or multiple events
• Statistical and sampling operators
• Sampling operators can be added to compute
  changes in rate of occurrence of a specific event
  type
• Statistical operators can be introduce to utilize
  learned patterns of normal behavior to detect
  statistical anomalies
• Extensible pallet of operators

29
Telcordias Events to Awareness Architecture
(E2A)
Awareness Specifications
Event Ontology
Routing Task Specifications
Content Routing and Coordination
Awareness Computation (AC)
Event Contextualization (EC)
Awareness
PrimitiveEvents
Contextualized Events
Axtionable Events (i.e. alerts task requests)

Users
Event Extraction Analysis (EA)
Proactive Event analysis Tasking
Event repository

• Continuous stream processing of events for real
  time event detection ?
• Event Subscriptions and tasks

30
E2A Component responsibilities

• Event contextualization
• Injects primitive events
• Contextualizes and fuses events
• Awareness Computation
• Utilizes user-specified awareness specifications
  to compute complex events continuously and
  incrementally
• Proactively seek missing events
• Coordination
• Manages alert and tasking interactions with
  end-users
• Manages tasking of event sources
• Application context(s), event ontology, awareness
  specifications, and task specifications
• Permits application-specific customization

31
Coordination for Alert Delivery and Proactive
Event Production

• E2As coordination component embodies the
  capabilities of a workflow management system
• Rich-media dataflow type
• Accepts actionable events from Alert Delivery and
  Proactive Event Production operators
• Routes alerts and evidence to the user role(s)
  specified in the alert delivery operators
• Integrates external programs that can interact
  with event sources for
• tasking them to produce a specific event or
  events or a specific type
• managing them (e.g., changing their settings)