Introduction and Overview of Data and Information Fusion

1
Introduction and Overview of Data and Information
Fusion
James Llinas Research Professor, Director,
Emeritus Center for Multisource Information
Fusion University at Buffalo llinas_at_buffalo.edu
2
CMIF's Approach A Total Fusion Systems
Perspective

• Basic through Advanced RD In
• IF Based Situation and Threat Assessment
• Multiple-Sensor System Data Integration/Analysis
• Multi-Intelligence Information Environments
• COMINT, ELINT, OSINT, HUMINT, IMINT..
• Graph Analysis Methodology and Systems
• Applications
• Defense C4ISR Support and Tactical Applications
• Non-Defense Disaster Consequence Management,
  Critical Infrastructure Protection, Medical
  Informatics

Intelligence Analysis
Maritime Domain Awareness
Situation and Threat Assessment for Multiple
Application Domains
Focus on High Level Information Fusion and Graph
Analytics
Contents are CMIF Proprietary
3
CMIF Foundational Sciences and Methods
Sciences Graph Theory Information Fusion Situation Assessment Threat Assessment Resource Management Information Theory Mathematical Optimization Modeling and Simulation Knowledge Representation and Reasoning (KRR)
Business/Engineering Systems and Software Engineering Product Lifecycle Management Service Oriented Design Rich Web Applications Complex Event Processing Database Design Realtime Systems
Sampling of Methods
Complex Mathematical Optimizations
Advanced Techniques In Linguistic Processing
Modern Graphical Methods
Contents are CMIF Proprietary
4
Collaborative Decision SupportCMIF Command and
Control Lab
5
Center for Multisource Information Fusion
(CMIF)Flexibility in Research and Development

• Core Technology Research Development
• State University of New York at Buffalo, NY, USA
• Top level research university and scientific
  staff focus is basic research and proof of
  concept experiments
• In existence 20 years (Average grant revenue
  US8M)
• Collaborative Partner for Technology Transition
• CUBRC, a not-for-profit defense RD organization
  in Buffalo, New York, USA (135 people, US35M)
• Cleared, experienced staff, facilities focus is
  development and transition

Core Technology RD
Broad Base Of Defense RD
UNIVERSITY at BUFFALO
Transition Hardening
UB CUBRC
6
History of Information Fusion

• Dates to early 80sfairly young in the sense of
  technological historya maturing technology/field
  of study
• Driven by defense and intelligence needs
• Originally as a data compression device to
  digest huge amounts of sensed data as sensors
  advanced in capability (a push requirement)
• Later as an important element for decision
  support (a pull requirement)
• Matures to very broad range of application
• Robotics, medicine, imagery/remote sensing,
  intelligent transportation, conditioned-based
  maintenance, biometrics, medicine, etc

7
What is (Automated) Information Fusion?

• Information fusion is an Information Process
  (Software) comprising
• FUNCTIONS
• Alignment
• Association, correlation
• Combination of data and information from
• INPUTS
• Single and multiple sensors or sources to achieve
  
• OUTPUTS
• Refined Estimates of
• parameters, characteristics, events, behaviors
  and relations for/among observed entities in an
  observed field of view
• It is sometimes implemented as a Fully Automatic
  process or as a Human-Aiding process for Analysis
  and/or Decision Support

8
Data Fusion Definition
" Data Fusion is the process of combining data
(or information) for the purpose of estimating or
predicting some aspect of the world"
Steinberg, Bowman, and White, Revisions to the
JDL Data Fusion Model, NSSDF 1998
9
Most Simply--
Observation System
So that estimation algorithms (mathematical
techniques)orautomated reasoning methods
(artificial intelligence techniques) can produce
better estimates (than based on any single type
of data)
Real World
Multiple types of data
Related to things of interest
To improve estimates about those things
Observations (Multiple)
Association of Observations
Estimation
These Basic Ideas are Transferable to Many Types
of Problems
10
Basic Role of Fusion
Dec-Mkg Analysis etc
Estimates Of World States
Observational Means (Streaming)
Data Association
(Dynamic) Real States in the World
Common Referencing Alignment
Process Refinement
Evaluation
Actions
Requirements driven from here

• One means to satisfy user information needs for
  decision/analysis support,
• i.e., most frequently inserted to
  support human user

11
Everyday Data Fusion
Multinodal Fusion
Sound
Augmented Sensing
Smell
Taste
Images
Touch
Sensing Association
Pain
Balance
Temperature
Body Awareness (Proprioception
Robotic Multisensor Fusion
12
Sensor Fusion Exploits Sensor Commonalities and
Differences, Knowledge of Errors
Unknown Moving Object
POOR
GOOD
FAIR
FAIR
POOR
RADAR
FAIR
FAIR
POOR
GOOD
FAIR
FAIR
EO/IR
FAIR
FAIR
GOOD
FAIR
FAIR
FAIR
FAIR
C3I
SENSOR FUSION
Data Alignment
Data Association
State Estimation
Data Association Uses Overlapping Sensor
Capabilities so that State Estimation Can
Exploit their Synergies
13
A Persistent Focus Reduced Uncertainty
14
Multisource Data (Evidence) Association
M Observations From N Sensors
Tracks T
DATA ASSOCIATION
Multiple Observations Multiple Entities
Assigned Observations Resulting from some
Best way to decide which Observations should be
given to each State Estimator
15
What a Message looks like a Graphical
StructureAn Observational Evidence Atom--Not
a Point Measurement--
An Observation (descriptionRepresentation of an
Observation)
Synonyms
Includes judgments as well as observations
Multiple Relationships
Disconnected semantic fragments
Generally all elements have some type of
imperfection or error
Some errors Quantified
16
Design of the Association Process for Linguistic
(Msg) Inputs
Good Assignment Solution Graph Merging
Effective Semantic scoring
Human Observer 1
Linguistic, Textual (Semantic) Inputs
Hypotheses Scored via Semantic Similarity Scores
that accnt For uncertainty
Hypotheses Evaluation By high-dimensional Assignme
nt problem solution
Hypotheses Self-generated by node/arc content
Human Observer 2
Pick a node/arc, Search other graph For
associable elements (eg exploit ontology)
Apply JVC or other Modern assignment Problem
solution
Smart Graph Search
Interdependency with Text, Semantic Operations
17
Fusion of Realtime Data and A Priori Data Bases
TodayIncludes --Sociocultural Info --Social Media
Context
Realtime

• Decision-SpecificInformation
• Timely
• Accurate
• Consistent
• Structured
• Integrated

Decision Maker
A Priori and Realtime
DATA
INFORMATION
KNOWLEDGE
UNDERSTANDING
18
Data Fusion Functional Model (Jt. Directors of
Laboratories (JDL), 1993)
INFORMATION FUSION PROTOTYPE
JEM JWARN3 GCCS
Level 0Processing Sub-object DataAssociation
Estimation
Level 1Processing Single-ObjectEstimation
Level 2Processing SituationAssessment
Level 3Processing Threat/ImpactAssessment
Methods --Combinatorial Optimization --Linear/NL
Estimation --Statistical --Knowledge-based --Contr
ol Theoretic
Level 4Processing Adaptive ProcessRefinement
Data BaseManagement System
SupportDatabase
FusionDatabase
19
Information Fusion The Defense Context
Multiple types of sensor data
Related to things of interest in the Real World
To improve estimates about those things
Real World

• In the defense problem
• Non-cooperative,
• Unfriendly
• Deceptive

Associated or Correlated to the same object
or event or behavior
Fusion (Estimation) Techniques
!
20
Todays IF Process Design EnvironmentInformation
-space Motivation Exploitation of all Information
Weather
Dynamic Real World
Financial
Sensor Observations
Cultural
Human Observations
Numbers
Chat Twitter
Web
Language
Semantic Label
Political
SOFT
HARD
HARD SOFT
Contextual Information
Observational Data
Modern Fusion Process
A Priori Dynamic World Model
L4 Knowledge Mgmt
Declarative Knowledge Ontologies
World State Estimates
Learning Processes
21
The Soft Front-end Input
Unconstrained Vocabulary (Possibly different
languages)
(Digitized)
Trained Observer
Semantics
Computational Linguistics, NLP
Language Processing
Untrained Observer
Automated Text Extraction
Interview
Typical Atomic, Raw Data Input
RDF Triples ()
Bystander
22
Source Characterization
Hard Data
Calibration (Truth) Target
?1
?2
?3
?4
Pd (Obs Params)
?5
To Common Ref, Data Association
23
Some Distinctions in Hard and Soft Observational
Data
Totally distinct from Hard Sensors Philosophy
Relations not directly observablerequire
reasoning over properties of entities
Humans can also judge intangibles --emotional
state
Brower, J., (2001) "Relations without Polyadic
Properties Albert the Great on the Nature and
Ontological Status of Relations." Archiv für
Geschichte der Philosophie 83 22557.
24
Counterinsurgency Problem Environment
MURI Information Fusion Technology
Hard Sensing
Soft Sensing
COIN Decision Support
Counter-Insurgency
Context
Soft Operations
Kinetic Operations
Connable, B, Culture and COIN,
www.citadel.edu/.../Connable,20Culture20and20Co
unterinsurgency20Brief.ppt
25
Some Remarks on Ontology and Information Fusion

• Dr. James Llinas
• Research Professor, Director (Emeritus)
• Center for Multisource Information Fusion
• University at Buffalo and CUBRC
• llinas_at_buffalo.edu

26
Roles for Ontologies in IF Processes/Systems

• Reasonably reliable a priori Declarative
  Knowledge about some domain
• In the face of domains for which reliable a
  priori Procedural (dynamic) Knowledge is hard to
  specify
• Weak Knowledge problems
• As such, they provide a framework that connects
  Entities and Relationships
• Of fundamental concern for COIN, Ctr-Terrorism,
  Irregular Warfare re social structures and
  militarily-significant entity relations
• The basic construct of a Situation or a
  Threat and thus Level 2, 3 Fusion estimation

27
Complexities in Distributed and Networked Systems

• In modern Distributed/Networked Systems there are
  No single points of authority These systems are
  collages of Legacy systemsJoint/Multiservice
  systemsCoalition systems
• Nodal Ontologies for Fusion/Situational
  Estimation, and Communication-support Ontologies
  for Inter-Nodal Communications/Data-sharing (eg
  JC3IEDM)
• Harmonizing NLP Operations and Ontologies within
  and across such systems
• The issue of Uncertainty in Ontological
  specification
• Probabilistic and Non-Probabilistic Ontologies
• Is there an Inescapable need for Semantic
  Mediation?
• Mediator systems well-studied and developed

• Eg Gio Wiederhold (June 1, 1993). "Intelligent
  integration of information". ACM SIGMOD Record 22
  (2)
• (This was a major DARPA program)

28
Semantic Complexity

• Controlling Semantic Proliferation/Complexity
• Ontologies
• Controlled Languages
• Eg Battle Management Language
• Eg Shade, U., et al, From Battle Management
  Language (BML) to Automatic Information Fusion,
  Chapter in Information Fusion and Geographic
  Information Systems, Lecture Notes in
  Geoinformation and Cartography,Popovich, V.V.
  Claramunt, C. Devogele, Th. Schrenk, M.
  Korolenko, K. (Eds.), 2011, Springer
• Understanding complexity drivers in text
• Eg McDonald, D.D., Partially Saturated Referents
  as a Source of Complexity in Semantic
  Interpretation, Proceedings of NLP Complexity
  Workshop Syntactic and Semantic Complexity in
  Natural Language Processing Systems, 2000
• Measuring Semantic Complexity
• Eg, Pollard, S and Biermann, A.W., A Measure of
  Semantic Complexity for Natural Language Systems
  (2000) Proceedings of NLP Complexity Workshop
  Syntactic and Semantic Complexity in Natural
  Language Processing Systems, 2000

29
The Association Problem

• The Ontologically-specified World is
  controllablethe Real Data World is not
• While Ontologies can help in Fusion-based
  estimation and inferencing problems, the
  mechanics of exploitation will involve the
  associability of Real (uncontrolled) data to
  (controlled) Entities and Relations in the
  Ontologies
• Semantic similarity, metrics, degree (hops),
  etc
• Efficient algorithmseg Cloud implementations
• PhD-level research
• There is also the issue of Coveragein
  poorly-understood/known problems, how does one
  specify an Ontology that has adequate coverage?
• Issue of negative information

30
Summary

• Ontologies have a useful role in the design and
  development of Information Fusion systems
• Questions regarding issues of
• Authoritative control of semantics in distributed
  systems
• Acceptable, optimal methods for mediation
• Complexity of semantics
• Understanding, measuring, controlling
• Association of semantic terms and complex,
  high-dimensional semantic structures
• Seem to require further, continuing study to
  better define best ways to employ ontological
  information in complex, distributed, large-scale
  Information Fusion systems and applications

31
Unified Research on Network-based Hard and Soft
Information Fusion

• A CMIF 5-year Multidisciplinary University
  Research Initiative (MURI) Program
• Funded by the Army Research Office 7M
• UB/CMIF lead Penn State Tenn State
• Soft/NLP/Ontology Lead Prof Stu Shapiro, CSE
• Building TRACTOR Soft front-end

32
(No Transcript)
33
(No Transcript)
34
(No Transcript)