Lectureship Early Career Fellowship

1
LectureshipEarly Career Fellowship
Fabio Cuzzolin INRIA Rhone-Alpes

• School of Technology, Oxford Brookes University
• 19/6/2008

2
Career path

• Masters thesis on gesture recognition at the
  University of Padova
• Visiting student, ESSRL, Washington University in
  St. Louis, and at the University of California at
  Los Angeles (2000)
• Ph.D. thesis on belief functions and uncertainty
  theory (2001)?
• Researcher at Politecnico di Milano with the
  Image and Sound Processing group (2003-2004)?
• Post-doc at the University of California at Los
  Angeles, UCLA Vision Lab (2004-2006)?
• Marie Curie fellow at INRIA Rhone-Alpes

3
Scientific production and collaborations

• collaborations with journals

IEEE PAMI
IEEE SMC-B
CVIU
Information Fusion
Int. J. Approximate Reasoning

• PC member for VISAPP, FLAIRS, IMMERSCOM, ISAIM
• currently 410 journal papers and 318 conference
  papers

4
My background
research
5
Computer Vision

• Action and gesture recognition
• Laplacian segmentation and matching of 3D shapes
• Bilinear models for invariant gaitID

Machine learning
Manifold learning for dynamical models
Uncertainty theory
Geometric approach to measures
Generalized total probability Vision applications
and developments
Discrete math
Unification of the notion of independence
New directions
6
HMMs for gesture recognition

• transition matrix A -gt gesture dynamics
• state-output matrix C -gt collection of hand poses

• Hand poses were represented by size functions
  (BMVC'97)?

7
Gesture classification

• EM to learn HMM parameters from an input sequence
• the new sequence is fed to the
• learnt gesture models
• they produce a likelihood
• the most likely model is chosen (if above a
  threshold)?
• OR new model is attributed the label of the
  closest one (using K-L divergence or other
  distances)?

HMM 1
HMM 2

HMM n
8
Compositional behavior of HMMs

• the model of the action of interest is embedded
  in the overall model ? clustering

• Cluttered model for two
• overlapping motions

• Reduced model for the fly gesture after
  clustering

9
Volumetric action recognition

• 2D approaches, feature extracted from images ?
  viewpoint dependence
• now available synchronized multi-camera systems
  Milano, BBC RD
• volumetric approach features are extracted from
  volumetric reconstructions of the body (ICIP'04)?

10
3D feature extraction

• k-means clustering of bodyparts

• Linear discriminant analysis (LDA) to estimate
  motion direction

• Locally linear embedding to find topological
  representation of the moving body

11
Computer Vision

• Action and gesture recognition
• Laplacian segmentation and matching of 3D shapes
• Bilinear models for invariant gaitID

Machine learning
Manifold learning for dynamical models
Uncertainty theory
Geometric approach to measures
Generalized total probability Vision applications
and developments
Discrete math
Unification of the notion of independence
New directions
12
Unsupervised coherent 3D segmentation

• to recognize actions we need to extract features
• segmenting moving articulated 3D bodies into
  parts
• along sequences, in a consistent way
• in an unsupervised fashion
• robustly, with respect to changes of the topology
  of the moving body
• as a building block of a wider motion analysis
  and capture framework
• ICCV-HM'07, CVPR'08, to submit to IJCV

13
Clustering after Laplacian embedding

• local neighbourhoods stable under articulated
  motion

• generates a lower-dim, widely separated embedded
  cloud
• less sensitive to topology changes than other
  methods
• less expensive then ISOMAP (refs. Jenkins,
  Chellappa)?

14
Algorithm

• K-wise clustering in the embedding space

15
Seed propagation along time

• To ensure time consistency clusters seeds have
  to be propagated along time
• Old positions of clusters in 3D are added to new
  cloud and embedded
• Result new seeds

16
Results

• Example of model recovery

• Coherent clustering along a sequence

17
Results - 2

• handling of topology changes

• missing data

18
Laplacian matching of dense meshes or voxelsets

• as embeddings are pose-invariant (for articulated
  bodies)?
• they can then be used to match dense shapes by
  simply aligning their images after embedding

• ICCV '07 NTRL, ICCV '07 3dRR, CVPR '08,
  submitted to ECCV'08, to submit to PAMI

19
Eigenfunction Histogram assignment

• Algorithm
• compute Laplacian embedding of the two shapes
• find assignment between eigenfunctions of the two
  shapes
• this selects a section of the embedding space
• embeddings are orthogonally aligned there by EM

20
Results

• Appls graph matching, protein analysis, motion
  capture
• To propagate bodypart segmentation in time
• Motion field estimation, action segmentation

21
Computer Vision

• Action and gesture recognition
• Laplacian segmentation and matching of 3D shapes
• Bilinear models for invariant gaitID

Machine learning
Manifold learning for dynamical models
Uncertainty theory
Geometric approach to measures
Generalized total probability Vision applications
and developments
Discrete math
Unification of the notion of independence
New directions
22
Bilinear models for gait-ID

• To recognize the identity of humans from their
  gait (CVPR '06, book chapter in progress)?
• nuisance factors emotional state, illumination,
  appearance, view invariance ... (literature
  randomized trees)??
• each motion possess several labels action,
  identity, viewpoint, emotional state, etc.

• bilinear models Tenenbaum can be used to
  separate the influence of style and content
  (to classify)?

23
Content classification of unknown style

• given a training set in which persons
  (contentID) are seen walking from different
  viewpoints (styleviewpoint)?
• an asymmetric bilinear model can learned from it
  through SVD
• when new motions are acquired in which a known
  person is being seen walking from a different
  viewpoint (unknown style)
• an iterative EM procedure can be set up to
  classify the content
• E step -gt estimation of p(cs), the prob. of the
  content given the current estimate s of the style
  
• M step -gt estimation of the linear map for
  unknown style s

24
Three-layer model

• Features projections of silhouette's contours
  onto a line through the center

• Three layer model
• each sequence is encoded as an HMM
• its C matrix is stacked in a single observation
  vector
• a bilinear model is learnt from those vectors

25
Results on CMU database

• Mobo database 25 people performing 4 different
  walking actions, from 6 cameras. Three labels
  action, id, view
• Compared performances with baseline algorithm
  and straight k-NN on sequence HMMs

26
Computer Vision

• Action and gesture recognition
• Laplacian segmentation and matching of 3D shapes
• Bilinear models for invariant gaitID

Machine learning
Manifold learning for dynamical models
Uncertainty theory
Geometric approach to measures
Generalized total probability Vision applications
and developments
Discrete math
Unification of the notion of independence
New directions
27
Learning manifolds of dynamical models

• Classify movements represented as dynamical
  models
• for instance, each image sequence can be mapped
  to an ARMA, or AR linear model, or a HMM
• Motion classification then reduces to find a
  suitable distance function in the space of
  dynamical models
• e.g. Kullback-Leibler, Fisher metric Amari
• when some a-priori info is available (training
  set)..
• .. we can learn in a supervised fashion the
  best metric for the classification problem!
• To submit to ECCV'08 MLVMA Workshop

28
Learning pullback metrics

• many algorithms take in input dataset and map it
  to an embedded space, but fail to learn a full
  metric (LLE, ISOMAP)?
• consider than a family of diffeomorphisms F?
  between the original space M and a metric space N
• the diffeomorphism F induces on M a pullback
  metric
• maximizing inverse volume finds the manifold
  which better interpolates the data (geodesics
  pass through crowded regions)?

29
Pullback metrics - detail

• case of linear maps Xing and Jordan'02,
  Shental'02

• Diffeomorphism on M

30
Space of AR(2) models

• given an input sequence, we can identify the
  parameters of the linear model which better
  describes it
• autoregressive models of order 2 AR(2)?
• Fisher metric on AR(2)?
• Compute the geodesics of the pullback metric on M

31
Results on action and ID rec

• scalar feature, AR(2) and ARMA models

32
Computer Vision

• Action and gesture recognition
• Laplacian segmentation and matching of 3D shapes
• Bilinear models for invariant gaitID

Machine learning
Manifold learning for dynamical models
Uncertainty theory
Geometric approach to measures
Generalized total probability Vision applications
and developments
Discrete math
Unification of the notion of independence
New directions
33
Uncertainty measures Intervals, credal sets

• a number of formalisms have been proposed to
  extend or replace classical probability

• assumption not enough evidence to determine the
  actual probability describing the problem
• second-order distributions (Dirichlet), interval
  probabilities
• credal sets

34
Multi-valued maps and belief functions

• suppose you have two different but related
  problems ...
• ... that we have a probability distribution for
  the first one
• ... and that the two are linked by a map one to
  many
• Dempster'68, Shafer'76

35
Belief functions as random sets

• probability on a finite set function p 2T -gt
  0,1 with
• p(A)?x e A m(x), where m T -gt 0,1 is a
  mass function

• probabilities are additive if A?B? then
  p(A?B)p(A)p(B)?

36
Geometric approach to uncertainty

• belief functions can be seen as points of a
  Cartesian space of dimension 2n-2
• belief space the space of all the belief
  functions on a given frame

• Each subset is a coordinate in this space

37
Approximation problem

• how to transform a measure of a certain family
  into a different uncertainty measure ? can be
  done geometrically

• Probabilities, fuzzy sets, possibilities are
  special cases of b.f.s

• IEEE Tr. SMC-B '07, IEEE Tr. Fuzzy Systems '07,
  AMAI '08, AI '08, IEEE Tr. Fuzzy Systems '08

38
Computer Vision

• Action and gesture recognition
• Laplacian segmentation and matching of 3D shapes
• Bilinear models for invariant gaitID

Machine learning
Manifold learning for dynamical models
Uncertainty theory
Geometric approach to measures
Generalized total probability Vision applications
and developments
Discrete math
Unification of the notion of independence
New directions
39
Total belief theorem

• generalization of the total probability theorem

• a-priori constraint

• conditional constraint

• introduces Kalman-like filtering for random sets

40
Graph of all solutions

• whole graph of candidate solutions

• admissible solution is found by following a path
  on the graph
• links to combinatorics and linear systems
• to submit to JRSS-B

41
Computer Vision

• Action and gesture recognition
• Laplacian segmentation and matching of 3D shapes
• Bilinear models for invariant gaitID

Machine learning
Manifold learning for dynamical models
Uncertainty theory
Geometric approach to measures
Generalized total probability Vision applications
and developments
Discrete math
Unification of the notion of independence
New directions
42
Model-free pose estimation

• estimating the pose (internal configuration) of
  a moving body from the available images

• if you do not have an a-priori model of the
  object ..
• Sun Torr BMVC'06, Rosales, Urtasun Brand,
  Grauman ICCV'03, Agarwal

43
Learning feature-pose maps

• ... learn a map between features and poses
  directly from the data
• given pose and feature sequences acquired by
  motion capture ..

• a multi-modal Gaussian density is set up on
  the feature space
• a map from each cluster to the set of poses
  whose feature values fall inside it (regression
  functions, EM)?

44
Evidential model

• similar to propagation in qualitative Markov
  trees
• MTNS'00, ISIPTA'05, to submit to Information
  Fusion

45
Information fusion by Dempsters rule

• several aggregation or elicitation operators
  proposed
• original proposal Dempsters rule

• b2
• m(?)0.1, m(a2 ,a3 ,a4)0.9

46
Performances

• comparison of three models left view only, right
  view only, both views

• left model

• estimate associated with the right model

• pose estimation yielded by the overall model

• ground truth

47
JPDA with shape info

• JPDA model independent targets

• shape model rigid links

• Dempsters fusion

• robustness clutter does not meet shape
  constraints
• occlusions occluded targets can be estimated
• CDC'02, CDC'04

48
Belief graphical models

• what happens when the original probability
  distribution belongs to a certain class?
• In particular belief functions induced by
  graphical models?

49
Imprecise classifiers

• application of robust statistics to vision
  problems
• imprecise classifiers
• class estimate is a belief function or a credal
  set Zaffalon, Cozman
• exploit only available evidence, represent
  ignorance

50
Credal networks

• belief networks or credal networks Shafer and
  Shenoy
• at each node a BF or a convex set of probs
• similar to generalized belief propagation ...
• message passing between nodes representing groups
  of variables

• algorithms to reduce complexity already exist

51
Computer Vision

• Action and gesture recognition
• Laplacian segmentation and matching of 3D shapes
• Bilinear models for invariant gaitID

Machine learning
Manifold learning for dynamical models
Uncertainty theory
Geometric approach to measures
Generalized total probability Vision applications
and developments
Discrete math
Unification of the notion of independence
New directions
52
Boolean independence

• independence can be defined in different ways in
  Boolean algebras, semi-modular lattices, and
  matroids
• Boolean independence is important in uncertainty
  theory

• a set of sub-algebras At of a Boolean algebra B
  are independent (IB) if

• example collection of power sets of the
  partitions of a given finite set

53
Relation with matroids?

• Matroids ? paradigm of abstract independence

• matroid (E, I?2E)
• ??I
• A?I, A?A then A?I
• A1?I, A2?I, A2gtA1 then ?x ? A2 s.t. A1?x?I
• graphic matroids dependent sets are circuits
• they have significant relationships BUT
• Boolean independence a form of
  anti-matroidicity?
• BCC'01, BCC'07, ISAIM'08, UNCLOG'08, subm.to
  Discrete Mathematics

54
Computer Vision

• Action and gesture recognition
• Laplacian segmentation and matching of 3D shapes
• Bilinear models for invariant gaitID

Machine learning
Manifold learning for dynamical models
Uncertainty theory
Geometric approach to measures
Generalized total probability Vision applications
and developments
Discrete math
Unification of the notion of independence
New directions
55
A multi-layer frameworkfor human motion analysis

• feedbacks act between different layers (e.g.
  integrated detection, segmentation,
  classification and pose estimation)?

56
Spatio-temporal action segmentation

• problem segmenting parts of the video(s)
  containing interesting motions
• global approach working on
  multidimensional volumes
• previous works object segmentation
  on the
  spatio-temporal volume for
  single frames Collins,
  Natarajan
• idea in a multi-camera setup, working on 3D
  clouds (hulls) motion fields time 7D volume
• proposal smoothing using tensor voting Medioni
  PAMI'05 shape detection on the obtained
  manifold

57
Stereo correspondence based on local image
structure

• problem finding correspondences between points
  in different view, using the local structure of
  the image
• Markov random fields disparity hidden variable
• one direction using local direction of the
  gradient or structure tensor to help the
  correspondence Zucker
• second option FRAME -gt large scale structures in
  MRF
• general potential for MRFs, local texture for
  correspondence?

58
Other developments

• 3D markerless motion capture
• Proposal data-driven pose estimation based on 3D
  representations
• unsupervised body model learning
• shape classification/ recognition in embedding
  spaces
• surveillance in crowded areas impossible to
  recover a 3D model
• ? information fusion techniques on multiple
  images
• handle conflict between different pieces of
  evidence