Multisensor Data Fusion

1

• Multisensor Data Fusion
• Andrés Navarro

2
Contents

• Introduction
• Definition
• Sensors and levels
• Fusion models
• Fusion techniques
• Distributed Data Fusion
• Scenarios
• Conclusions

3
Introduction

• Motivation
• Master Thesis
• First approach to sensor fusion
• Overview of state-of-the-art
• Guide for people who do not know about sensor
  fusion to get introduced to the issue

4
Introduction

• Why is sensor fusion needed?
• Research questions
• What is multisensor data fusion? Is there an
  unanimous definition?
• What models for multisensor data fusion exist in
  the literature? Do they have common descriptions?
  Do they contradict each other?
• What techniques or methods can be used in
  multisensor data fusion?
• Centralized or decentralized data fusion?
• What scenarios can multisensor data fusion be
  applied?

5
Definition

• JDL Data Fusion Lexicom
• A. Steinberg and C. Bowman.
• Wald

A process dealing with the association,
correlation, and combination of data and
information from single and multiple sources to
achieve refined position and identity estimates,
and complete and timely assessments of situations
and threats, and their significance. The process
is characterized by continuous refinements of its
estimates and assessments, and the evaluation of
the need for additional sources, or modification
of the process itself, to achieve improved
results.
Data fusion is the process of combining data or
information to estimate or predict entity states.
Data fusion is a formal framework in which are
expressed means and tools for the alliance of
data originating from different sources. It aims
at obtaining information of greater quality the
exact definition of greater quality will depend
upon the application.
6
Sensors and levels
Commensurate multisensor data
Non commensurate multisensor data
Feature-level Fusion
Information extraction
Direct Data Fusion
High-level Fusion
7
Fusion models

• JDL data fusion model
• Dasarathy's functional model
• Waterfall fusion process model
• Boyd Loop
• Thomopoulos' Fusion Model
• Durrant-Whyte architecture
• The Omnibus process model
• Endsley's Situation Awareness
• General Data Fusion Architecture

8
Fusion models

• JDL data fusion model
• Sources
• Sources preprocessing
• Level 1
• Level 2
• Level 3
• Level 4
• Database management system
• HCI

9
Fusion models

• JDL model revisions
• Drawbacks
• Different revisions
• New definitions

10
Fusion models

• Dasarathy's functional model
• Levels of abstraction
• Data
• Feature
• Decision
• Categorization of data fusion functions in terms
  of the type of data level at input/output.

11
Fusion models

• Waterfall fusion process model
• Fusion processin stages
• Omission offeedback dataflow is themajor
  limitation.

12
Fusion models

• Boyd Loop
• OODA cycle

13
Fusion models

• Thomopoulos' Fusion Model

14
Fusion models

• Durrant-Whyte achitecture

15
Fusion models

• The Omnibus process model

16
Fusion models

• Endsley's Situation Awareness

17
Fusion models

• General Data FusionArchitecture
• Network based
• Levels described as classes with attributes and
  functions.

18
Fusion models

• Models classification
• Elmenreich
• Abstract Generic Rigid
• Durrant-Whyte and Henderson
• Architecture Meta, Algorithmic, Conceptual,
  Logical and Execution
• Centralized - Decentralized
• Local Global
• Modular Monolithic
• Heterarchical - Hierarchical

19
Techniques

• Classification
• Overview
• Kalman Filter
• Probabilistic Inference
• Artificial Neural Networks
• Fuzzy Logic
• Support Algorithms
• Selection

20
Techniques

• Classified by

• JDL levels

21
Techniques

• Classified by
• JDL levels
• Type of method
• Fusion problems

22
Techniques

• Classified by

• JDL levels
• Type of method

23
Techniques

• Classified by
• JDL levels
• Type of method
• Fusion problems

24
Techniques

• Classified by

• JDL levels
• Type of method
• Fusion problems
• Data Association
• Estimation
• Identity declaration
• Decision-level identity fusion

25
Techniques

• Kalman Filter

Extended Kalman Filter
Discrete Kalman Filter
Assumed noise
26
Techniques

• Kalman Filter with INS
• Inertia System
• Good high frequency information
• Drift at a slow rate
• Other Position System
• Good data at low frequency, on the average
• High frequency noise

The Kalman Filter approach is instead to use the
statistical characteristics of the errors in both
the external information and the inertial
components to determine this optimal combination
of information. Actually, the filter
statistically minimizes the errors in the
estimates of the navigation parameters on an
ensemble average basis, no other means of
combining the data will outperform it, assuming
the internal model in the filter is adequate.
27
Techniques

• Kalman Filter with INS

• Direct implementation

• Filter fails ? System Fails
• High sample rate ? CPU load

• Indirect feedforward

• Erros in the inertial must remain of small
  magnitude

• Indirect feedback

28
Techniques

• Probabilistic Inference

• Bayesian Inference
• It can be used to discriminate between
  conflicting hypotheses
• Initial beliefs are needed before any evidence is
  ever collected
• Sensorfusion

29
Techniques

• Probabilistic Inference

• Bayesian Networks
• Probabilistic graphical model that represents a
  set of variables and their probabilistic
  interdependencies
• Algorithms to perform inference and learning

• Dynamic Bayesian Networks
• Extension of Bayesian networks that allows the
  representation of temporal information ? Signals

• Hidden Markov models
• Model for Markov process a stochastic process in
  which the probability distribution of the current
  state is conditionally independent of the path of
  past states

30
Techniques

• Probabilistic Inference
• Dempster-Shafer Theory
• Generalization of Bayesian Theory Instead of
  requiring probabilities for each question, belief
  functions are used.
• Two ideas

• Obtain degrees of belief for one question from
  subjective probabilities fora related question.
• Use Dempster's rule for combining these degrees
  of belief

Sensor1
Sensor2
m1(u0)?
m2(u0)?
m(u0)m1(u0)m2(u0)?
31
Techniques

• Probabilistic Inference
• Dempster-Shafer Theory
• Generalization of Bayesian Theory Instead of
  requiring probabilities for each question, belief
  functions are used.
• Two ideas

• Obtain degrees of belief for one question from
  subjective probabilities fora related question.
• Use Dempster's rule for combining these degrees
  of belief

Sensor1
Sensor2
m1(u0)?
m2(u2)?
m(u0)m1(u0)m2(u2)?
32
Techniques

• Probabilistic Inference
• Dempster-Shafer Theory
• Generalization of Bayesian Theory Instead of
  requiring probabilities for each question, belief
  functions are used.
• Two ideas

• Obtain degrees of belief for one question from
  subjective probabilities fora related question.
• Use Dempster's rule for combining these degrees
  of belief

Sensor1
Sensor2
m1(u0)?
m2(u1)?
Dempster's rule
33
Techniques

• Probabilistic Inference

• Generalized evidence processing theory
• Unifies the Bayesian theory with the D-S,
  combining their advantages and avoiding their
  disadvantages
• Each sensor collects evidence and assigns the
  evidence via probability masses unlike D-S, GEP
  assigns and combines probability masses based on
  the a priori conditional probability of the
  hypotheses.

34
Techniques

• Artificial Neural Networks
• Computational model of biological neural
  networks
• Densely interconnected set of artificial neurons
  simple units as perceptron
• Feed-forward / recurrent
• Non linear statistical data modelling
• They can learn a complex relationship between
  inputs and outputs, normally established by the
  unit weights
• Learning

• Units' weights ? Backpropagation algorithm
• Network structure

35
Techniques

• Artificial Neural Networks
• Neurons can be trained to represent sensory
  information and, through associative recall,
  complex combinations of the neurons can be
  activated in response to different sensory
  stimuli.
• The main advantage of neural networks for
  multisensor fusion is that there is no need of a
  model for the sensors or the uncertainties

36
Techniques

• Fuzzy Logic
• Fuzzy set theory
• Elements have degrees of membership to the
  different sets, differing from classical set
  theory, where elements belong or do not belong to
  a certain set.
• Rules IF antecedent THEN consequence
• Operators OR, AND and NOT
• Steps

• Fuzzyfcation
• Rule evaluation
• Aggregation
• Defuzzyfication

37
Techniques

• Fuzzy Logic
• Uncertainty in multisensor fusion can be directly
  represented in the inference process by allowing
  each proposition to be assigned a degree of
  truth.
• Fuzzy Fusion Network
• Input data
• Feature Extraction
• Feature Level Fusion
• Decision Level Fusion

38
Techniques

• Support Algorithms
• Required functions for the fusion system
• Library of basic numerical methods
• Database management
• Man-Machine Interaction
• Sensor Management

39
Techniques

• Techniques Selection
• Goals
• Maximum effectiveness Algorithms should make
  inferences with maximum specificity in the
  presence of uncertain and missing data, dealing
  with minimal or no available a priori
  information.
• Operational constraints and time constraints must
  be considered.
• Resource efficiency in CPU and communications
  load.
• Operational flexibility to account for
  operational needs with changing a priori data.
• Functional growth.

40
Techniques

• Techniques Selection
• Steps
• Identifying categories of data-processing
  techniques or algorithms.
• Surveying existing prototype and fielded data
  fusion systems.
• Analyzing system requirements.
• Analyzing and defining operational concepts for
  manual and automatic processes.
• Identifying preliminary algorithms
• Performing trade-off analyses of algorithm
  effectiveness versus required system resources.
• Preparing detailed designs and prototypes of
  selected algorithms.
• Refining and tuning the algorithms.

41
Distributed Data Fusion

• A collection of processing nodes, connected by
  communication links, in which none of the nodes
  has knowledge about the overall network
  topology.
• Requirements
• Dynamic Topology Management
• Information- Sharing Strategies
• Algorithms

42
Scenarios

• Image Data Fusion
• Advantages
• Reduction of overall uncertainty and increase of
  accuracy.
• New features in a scene can be perceived
• More timely information is available

43
Scenarios

• Robot Navigation

44
Scenarios

• Ambient Intelligence
• Electronic environments that are sensitive and
  responsive to the presence of people.
• Characteristics
• Embedded
• Context-aware
• Personalized
• Adaptive
• Anticipatory

45
Scenarios

• Augmented Reality
• Integration of virtual content in a real
  environment in real time.

• Alignment between virtual content and real world
  
• Fusion of tracking systems
• INS
• GPS
• Ultrasound
• Vision-based

Accurate position
46
Scenarios

• Augmented Reality
• Interaction with other perceptual systems
• Orienting, Auditory,Haptic, Taste, Smell
• Feature extraction ? Human Behaviour
• Sensors Position, orientation, body gestures,
  speech, vital signs, eyetracking...
• Human behaviour and experience models and
  simulations

47
Conclusions

• What is multisensor data fusion? Is there an
  unanimous definition?
• Most of definitions are restrictive to a certain
  terminology and applications
• A broader definition is needed to cover such a
  wide diversity of sensor fusion applications
• Wald's definition is chosen
• Discussion will continue

48
Conclusions

• What models for multisensor data fusion exist in
  the literature? Do they have common descriptions?
  Do they contradict each other?
• A common point in most of them is the need of
  divide the fusion process in levels of data
  abstraction.
• More disagreement is found in the idea of a
  cycling processing.
• Relationship between specification and usability
• Combine the underlying ideas for the final design.

49
Conclusions

• What techniques or methods can be used in
  multisensor data fusion?
• Data fusion at different levels of abstraction
  implies the use of multiple techniques.
• A layout or scheme for the implementation of any
  kind of sensor fusion application is not
  feasible.
• The design of the fusion algorithms is a lengthy
  task where multiple fusion techniques can be
  combined.

50
Conclusions

• Centralized or decentralized data fusion?
• The decentralized fashion has some advantages and
  some disadvantages comparing to the centralized
  one.
• The implementation of Distributed Data Fusion
  requires
• The use of certain fusion models that allow
  decentralization.
• Specific algorithms

51
Conclusions

• Final conclusion

Multisensor Data Fusion is a broad issue due to
the wide range of scenarios that it can be
applied to. Therefore, to find a definition, a
model or an algorithm scheme that is explicit,
meaning that it can be followed to implement a
real system, and at the same time usable for any
kind of application, is an unfeasible task.
Hence, a view of the different approaches,
theories and implementations in the issue of
sensor fusion can be presented, intending to be
useful as a collection of different ideas that
should be combined in the implementation of a
real fusion system.
52

• Multisensor Data Fusion
• Andrés Navarro