Michle ROMBAUT

1
Principles for Data Fusion Applications

• Michèle ROMBAUT
• LIS INPG Grenoble France

2
Why to fuse data?

• Assumption observation of a real system
• Improve the knowledge about the system
• state, evolution, classification
• What type of knowledge do we want to acquire?
• accurate evaluation of analogic state
• reliable evaluation of symbolic state

3
Symbolic data fusion

• Don't present analogic data fusion in this
  presentation.
• Samples of symbolic data fusion
• medical diagnosis
• human recognition (posture, action, activity,
  emotion, gesture)

4
Steps of data fusion

• Observation process
• Numerical to symbolic data conversion
• Definition of space of discernment
• Fusion process
• Decision process

5
Parts of the presentation

• Part 1 Problem analysis without taking into
  account accuracy, certainty, reliability
• Part 2 Problem analysis using main formalisms
• probability approach
• possibility approach
• Part 3 Evidence theory for data fusion
• transferable belief model, Dempster-Shafer theory

6
Part 1

• Problem analysis without taking into account
  accuracy, certainty, reliability

7
Observation process

• Need of sensors
• symbolic click of a mouse
• generally analogic measurement of analogic
  parameters (distance, length, )
• How to choose the parameters?
• difficult question
• must be pertinent regarding the objectives
• expert choice
• information quantity evaluation?

8
Observation process

• Developing hardware or software sensors
• particular or usual sensors (camera, microphone)
• particular or usual data processing (image and
  signal processing)
• Evaluation of the pertinent parameters

9
Space of discernment

• How to choose the symbols of interest?
• Must correspond to the needs of application
• emotion recognition what emotions? neutral
  emotion? combined emotions? grimaces?

Space of discernment ? H1, H2, , HN
10
Space of discernment

• How to choose the symbols of interest?
• Must correspond to the needs of application
• emotion recognition what emotions? neutral
  emotion? combined emotions? grimaces?
• Properties of the chosen symbols
• exclusivity? (one true and others false)
• exhaustivity? (open or closed world)

11
Space of discernment

• Choice of intermediate symbols
• Example of silhouette posture
• vertical - horizontal give information about
  posture
• Definition of pertinent spaces
• of discernment

12
Numerical to symbolic data conversion

• How to link numerical data to symbolic data?
• Bridge the semantic gap
• Need models of interpretation to conversion
  process
• statistic models supervised learning
• expert models

13
Numerical to symbolic data conversion

• Example Is a silhouette vertical or horizontal?
• Numerical parameter angle between the main axis
  of the silhouette and horizontal line
• Space of discernment ? Horizontal, Vertical

14
Numerical to symbolic data conversion
Problem of crisp sets (will be seen later)
15
Conversion on level
16
Conversion on event
P1
H1
H2
H1
t
Need a Flip-Flop for memory
17
Fusion process

• Logical combination
• Assuming there are several symbolic parameters
  estimation
• Could be redundant
• same space of discernment
• improve confidence about symbols
• Could be complementary
• different spaces of discernment
• improve accuracy (symbols more accurate)
• Fusion ? same space of discernment

18
Fusion process

• Redundant information
• Example
• measurement of main axis angle ? vertical or
  horizontal
• measurement of extreme axis angle ? vertical or
  horizontal
• Problem of the conflict between sources will be
  seen later

19
Fusion process

• Complementary information
• Example
• measurement of distances in the face ? small or
  large
• Need rules of combination
• Can be seen as refinement and coarsening process

20
Fusion process example

• Color sensor ?Cblue, red
• Shape sensor ?Scircle, square
• Object ?OA,B,C,D
• Final interest ?FCheap, Expensive

21
Fusion process example

• ?OA,B,C,D?C??S
• Refinement process
• blueA?B, redC?D
• circleB?D, squareA?C
• Coarsening process
• Expert knowledge
• cheapA?D, expensiveB?C

22
Fusion process example

• Rules of combination

Refinement process
23
Fusion process example

• Rules of combination

Refinement process
Coarsening process
cheapA?D, expensiveB?C
24
Fusion process for Sequence Recognition
t
25
Decision process

• Crisp sets no problem
• If uncertainty optimizing a criteria

26
Part 2

• Inacuracy and uncertainty using main formalisms
• probability
• possibility

27
Quality of the data

• Problems of inaccuracy, uncertainty, conflict
  between sources of information.
• Usual formalisms
• Probabilistic and Bayesian
• Fuzzy sets and possibility
• Belief theory (Transferable Belief Model)
• Same steps of the previous part
• Definition of space of discernment
• Logical rules of combination
• Decision criteria

28
Numerical to symbolic data conversion
Problem of crisp sets
29
Probabilistic approach

• Conversion model
• statistic supervised learning
• subjective expert knowledge model

Likelihood conditional probability For a given
? v?(V) p(?/V) v?(H) p(?/H)
30
Fuzzy set approach

• Membership degree that ? belongs to class A
  mA(?)
• Possibility p?(V) mV(?)

31
Fusion process

• Redundant information
• Same space of discernment
• Problem of the conflict between sources

32
Bayesian approach

• Using of Bayes theorem

33
Doubt and conflict
34
Fusion process for Sequence Recognition
Hidden Markov Model (HMM) Transitions between
states conditional probabilities
35
Possibility approach

• Lot of combination operators
• Three main classes
• Conjunctive operators T-norm
• Used for concordant information
• Disjunctive operators T-conorm
• Used for discordant information
• Adaptative operators according to a conflict
  index

36
Fusion process

• Complementary information different spaces of
  discernment (no conflict)
• Using rules of combination
• Refinement and coarsening process
• For sequence recognition Fuzzy Petri nets

37
Decision process

• Optimizing a criteria
• Choice of criteria application dependant
• Decision ? risks

38
Probabilistic approach

• Usual criteria
• maximum of probability a posteriori
• maximum of likelihood

39
Possibility approach

• Usual criteria
• Maximum of possibility optimistic decision
• Maximum of necessity careful approach

40
Part 3

• Evidence theory for data fusion
• Dempster-Shafer theory, Transferable belief model
  (TBM)
• Transferable belief model
• Developed by P. Smets http//iridia0.ulb.ac.be/p
  smets/
• http//www.hds.utc.fr/tdenoeux/

41
Transferable Belief Model

• Based on a Basic belief assignment (BBA)
• Defined on propositions i.e. power set of W ? 2W
  
• Link to Belief (Bel) and Plausibility (Pl)

42
Basic Belief Assignment
m 2W ? 0,1 A ? m(A)

• m(A) mass of evidence
• confidence on the subset A and doubt about the
  hypotheses included in A
• total of mass equal to 1

43
Doubt representation

• Doubt explicitly represented
• Total vacuous m (W) 1
• Example of Basic Belief Assignment
• Space of discernment W V,H
• Power set 2W ?,V, H, H,V
• W H,V ? (H ? V)
• Example m(H) 0.3 m(H ? V) 0.7

44
Example of doubt
Surprise and Fear
Transition between neutral and joy
45
Belief (Bel)

• Bel 2W ? 0,1
• A ? Bel(A)

Bel(A) belief that trueness is in A
46
Plausibility (Pl)

• Pl 2W ? 0,1
• A ? Pl(A)

Pl(A) plausibility that trueness is in A
47
Numerical to symbolic data conversion

• Statistical or expert knowledge
• Constructed on probability distributions or
  inspired to the fuzzy sets
• Same difficulties such as probability approach or
  fuzzy sets approach
• Explicitly model the doubt
• Lot of propositions (Denoeux, Smets, Appriou)

48
Proposition fuzzy sets inspired method

• Definition of belief that a measurement belongs
  to a class or union of classes

49
Proposition fuzzy sets inspired method

• Definition of belief that a measurement belongs
  to a class or union of classes

50
Fusion process

• Redundant information
• Conjunctive combination
• Value of conflict

51
Rules of combination

• Conjunctive combination based on intersection

W V,H
52
Rules of combination

• Conjunctive combination based on intersection

W V,H
m1,2 (V) m1(V).m2(V) m1(V).m2(V?H)
m1(V?H).m2(V)
53
Rules of combination

• Conjunctive combination based on intersection

W V,H
m1,2 (V) m1(V).m2(V) m1(V).m2(V?H)
m1(V?H).m2(V) m1,2 (H) m1(H).m2(H)
m1(H).m2(V?H) m1(V?H).m2(H) m1,2 (V?H)
m1(V?H).m2(V?H) K m1(V).m2(H) m1(H).m2(V)
54
Conflict management

• K quantity of conflict between sources
• Never normalize the masses loosing information
• Re-distribution of the conflict on vacuous
• Analyze of the problem
• Reliability of the sources
• Models of conversion
• Adapted decision

55
Rules of combination

• Complementary information
• Refinement and coarsening process

56
Fusion process for Sequence Recognition
Temporal belief filter and belief scheduler for
human activity recognition PhD of E. RAMASSO
(LIS-INPG) ICCASP2006, Fusion2006
57
Discounting

• Model the confidence about a source of
  information
• Depends on context parameters
• Discounting parameter ? ? 0,1

58
Example

• Example of Basic Belief Assignment
• Space of discernment W V,H
• m(H) 0.3 m(H ? V) 0.7
• New Basic Belief Assignment
• m'(H) ?.m(H) 0.3 ?
• m'(H ? V) m(H ? V) (1-?).m(H) (1-?) ?.m(H
  ? V) (1-?) 0.7?
• Doubt is increased

59
Decision process

• Optimizing a criteria
• Choice of criteria application dependant
• Decision ? risks
• Maximum of Belief (careful approach)
• Maximum of Plausibility (optimist approach)
• Maximum of Pignistic probabilities

60
Pignistic probability

• Smets says decision is made with probabilities
  ? need to bet
• Transformation
• Basic Belief Assignment defined on 2W
• to Pignistic Probability defined on W.

61
Problems to use TBM

• Difficulty to define models conversion ? same
  problem for probability, fuzzy sets
• Large amount of combinations ? working on power
  set
• Binary approach to code BBA
• N hypotheses N bits word code
• Union of hypotheses logical OR operator

62
Example for ? A, B, C

• 3 bits word
• A 100 B010 C 001
• A?B 100 OR 010 110
• Intersection logical AND
• Conjunctive rule m1(A?B).m2(A?C) m12(A)
• 110 AND 101 100
• m1(110).m2(101) m12(100)

63
Interests to use TBM

• All made by probabilistic approach can be done
  by TBM approach
• Doubt is better modeled
• Conflict in conjunctive combination is detected
  and quantified
• Reliability of the source can be taken into
  account
• Expert knowledge are relatively easy to take into
  account

64
To conclude on fusion process

• Common difficulties for all approaches
• Definition of the spaces of discernment
• Modelisation of the knowledge (expert and
  statistic)
• Definition of the combination rules
• Choice of the decision criteria (always taking a
  risk)

65
Comparison of approaches

• Results cannot be directly compared
• before decision different model of information
• after decision largely depending to the choice
  criteria
• How to choose?
• Fuzzy sets fuzziness of signal/image (analogic
  data)
• TBM used to model the knowledge (expert or
  statistic knowledge, adaptability)
• Probability used to make a decision
• TBM and Probability are complementary