STRUCTURE FROM MOTION

1
STRUCTURE FROM MOTION

• Esther Antúnez Ortiz
• PRIP, Vienna University of Technology

2
Overview

• Introduction
• Object Tracking
• Object representation
• Feature selection
• Object detection
• Object Tracking Methods
• Graph-based Tracking
• Structure From Motion
• Problems not using structure
• Improvements using structure
• Example Comparison of Mean Shift with and
  without structure
• Summary

3
Introduction

• Tracking find correspondences between objects
  in consecutive frames

4
Overview

• Introduction
• Object Tracking
• Object representation
• Feature selection
• Object detection
• Object Tracking Methods
• Graph-based Tracking
• Structure From Motion
• Problems not using structure
• Improvements using structure
• Example Comparison of Mean Shift with and
  without structure
• Summary

5
Object Tracking

• Tracking estimation of the trajectory of an
  object in the image plane as it moves around a
  scene
• Applications
• motion-based recognition
• automated surveillance
• human-computer interaction
• traffic monitoring
• vehicle navigation

6
Object Tracking

• Tracking objects can be complex due to
• loss of information caused by projection of the
  3D world on a 2D image
• noise in images
• complex object motion
• non-rigid or articulated nature of objects
• partial and full object occlusions
• complex object shapes
• scene illumination changes
• real-time processing requirements

7
Object Tracking

• Constraints on the motion and/or appearance of
  objects to simplify the process of tracking
• Almost all tracking algorithms assume that the
  object motion is smooth with no abrupt changes
• Prior knowledge about the number and the size of
  objects, or the object appearance and shape, can
  also be used to simplify the problem

8
Object Tracking

• Numerous approaches for object tracking
• These primarily differ from each other in
• object representation
• image features
• modeling of motion, appearance and shape of the
  object

9
Object Tracking

• Process of tracking
• suitable representation of the object
• selection of image features
• strategy for detecting objects in a scene.

10
Object respresentation

• The most common shape representations are
• Points
• Primitive geometric shapes
• Object silhouette and contour

• Articulated shape models
• Skeletal models

11
Object respresentation

• And the most common ways to represent the
  appearance are
• Probability densities of object appearance
• Templates
• Active appearance models
• Multiview appearance models

12
Feature selection

• Closely related to the object representation
• Most of the algorithms use a combination of
  several features
• Some of the most common features
• Color
• Edges
• Optical flow
• Texture

13
Object detection

• Point detectors
• Background subtraction
• Segmentation
• Mean-Shift clustering
• Active contours
• Graphs-cuts
• Supervised learning

14
Overview

• Introduction
• Object Tracking
• Object Segmentation
• Feature selection
• Object detection
• Object Tracking Methods
• Graph-based Tracking
• Structure From Motion
• Problems not using structure
• Improvements using structure
• Example Comparison of Mean Shift with and
  without structure
• Summary

15
Object Tracking Methods

• Tasks of object tracker
• detecting the object and
• establishing correspondence between the object
  instances across frames
• Objects represented using the shape and/or
  appearance models described in previously
• We can divide the tracking methods in
• Point tracking
• Kernel tracking
• Silhouette tracking
• Tracking using structure

16
Point Tracking

• Objects represented by points
• Association based on the previous object state
  (position, motion,)
• Requires an external mechanism to detect the
  objects in every frame
• Two categories
• deterministic methods
• statistical methods

17
Kernel Tracking

• Kernel refers to the object shape and appearance
• Can be a rectangular template or an elliptical
  shape with an associated histogram
• Objects tracked by computing the motion of the
  kernel in consecutive frames

18
Silhouette Tracking

• These methods use the information encoded inside
  the object region
• Tracking performed by estimating the object
  region in each frame
• Silhouettes tracked by
• shape matching or
• contour evolution

19
Tracking using Structure

• These methods using structural information of the
  objects in addition to other features like color,
  for example.
• Objects divided in parts
• Certain relations and constrains among these
  parts are impose
• Example human body

20
Overview

• Introduction
• Object Tracking
• Object Segmentation
• Feature selection
• Object detection
• Object Tracking Methods
• Graph-based Tracking
• Structure From Motion
• Problems not using structure
• Improvements using structure
• Example Comparison of Mean Shift with and
  without structure
• Summary

21
Graph-based Tracking

• Graphs offer a way to represent the structure in
  a rich and compact manner
• Node attributes size, average color, position
• Edges specify the spatial relationships(adjacency,
  border) between the nodes
• In this way, each image of a sequence is
  segmented and represented as a region adjacency
  graph.

22
Graph-based methods using graph matching

• Used to associate structures acquired at
  different time instances
• Object tracking becomes a graph-matching problem
• Complexity of graph matching reduced with the
  segmentation

23
Graph-based methods not using graph matching

• Graphs are only used to represent structure
• But not for associating consecutive measurements

24
Others

• Some methods use graphs to represent
  task-specific prior knowledge in form of a graph
  structure

25
Overview

• Introduction
• Object Tracking
• Object Segmentation
• Feature selection
• Object detection
• Object Tracking Methods
• Graph-based Tracking
• Structure From Motion
• Problems not using structure
• Improvements using structure
• Example Comparison of Mean Shift with and
  without structure
• Summary

26
Structure From Motion

• Many tracking algorithms encounter difficulties
  with
• articulated objects
• targets undergoing pose variations
• partial occlusions
• Tracking the parts of the object can give
  information about the motion of the object to
  handle occlusions in a more robust way
• Added problem how to distinguish between the
  tracked parts

27
Structure From Motion

• Structural information is needed (i.e. tracking
  people)
• spatial relationships between the parts
• constraints on the parts and their connections
• Many tracking methods using graphs to describe
  the structure
• nodes (size, average color, position )
• spatial edges (spatial relationships adjacency,
  border)
• temporal edges(correspondence between moving
  parts)

28
Problems not using structure

• Occlusion by similar object
• Occlusion by equal object

29
Problems not using structure

• Complete occlusion by different object
• Change of objects appearance

30
Problems not using structure

• Combination of different kind of occlusions

31
Improvements using structure

• Structural information to solve some problems
• adjacency
• connectivity
• ratio between the sizes of the parts
• actual state of the parts
• constrains
• graphs are used to describe the structure
• nodes ? store attributes and
• edges ? represent the spatial relationships
• Graph matching problem

32
Example Comparison of Mean Shift with and
without structure

• Mean Shift algorithm used to associate the nodes
  of the structure between adjacent image frames
• Statistical procedure, which locates local
  density maxima in a given probability
  distribution
• Search window in the probability distribution
• Density maximum determined by weighted average
  computation
• Search window moved to the maximum until the
  algorithm converges
• Based in color histogram

33
Example Comparison of Mean Shift with and
without structure

• Mean Shift algorithm works reliably and robustly
  in videos with
• rigid target object
• no occlusions
• no similar or even equal objects like the target
  object
• constant lighting
• However occlusions are a serious problem
• Occlusions corrupt the color distribution

34
Example Comparison of Mean Shift with and
without structure

• Artner et al. propose to combine deterministic
  tracking of object parts with graph
  representation encoding structural dependencies
  between the parts
• MSER used to generate regions (vertices of the
  graph)
• Mean Shift trackers at each vertex
• Objective
• Structural energy minimization of the graph and
• Color histogram similarity maximization at the
  vertices

35
Example Comparison of Mean Shift with and
without structure

• Combination of Mean Shift and graph relaxation
• Objetive of graph relaxation maintain the
  tracked structure as similar as possible to the
  initial one

36
Example Comparison of Mean Shift with and
without structure

• Performance of Mean Shift with/without structure

37
Overview

• Introduction
• Object Tracking
• Object Segmentation
• Feature selection
• Object detection
• Object Tracking Methods
• Graph-based Tracking
• Structure From Motion
• Problems not using structure
• Improvements using structure
• Example Comparison of Mean Shift with and
  without structure
• Summary

38
Summary

• Many algorithms for object tracking in based on
  object representation and object features
• Problems with occlusions, non-rigid objects,
  unknown motions
• Structural information to improve the process of
  tracking

39
STRUCTURE FROM MOTION
THANK YOU!

• Esther Antúnez Ortiz
• PRIP, Vienna University of Technology