A Bayesian Computer Vision System for Modeling Human Interactions

1
A Bayesian Computer Vision System for Modeling
Human Interactions
Nuria Oliver, Barbara Rosario And Alex
Pentland _at_ Media Laboratory MIT Lectured by
Ehrlich Avi
2
Main Problem Figure-Ground Segregation
http//www.illusionworks.com
3
(No Transcript)
4
Background learning

• It build from sampling N images and computing the
  mean background image and covariance matrix.
• The moving objects wont have significant
  contribution to the mean background (because they
  wont appear in most of the N samples).

5
(No Transcript)
6
Segmentation and Tracking

• The RT computer vision creates 2-D blobs
  features for modeling each pedestrian.
• blob is a compact set of pixels that share some
  visual properties that are not shared by the
  surrounding pixels. (can be color, texture,
  brightness, motion, shading or combination of
  these).
• The main cue for clustering the pixels into blobs
  is motion.

7
Segmentation by background subtraction
Cont

• Now we can take any new frame and subtract it
  fromthe calculated background - any Euclidean
  distance from feature space between them that are
  bigger then a given threshold is a moving object.
• This motion mask is the input to the connected
  component algorithm that produces blob
  descriptions.

8
(No Transcript)
9
Tracking

• The trajectories of each blob is computed and
  saved to dynamic track memory. Each trajectory
  has associated to Kalman filter the predicts the
  blobs position and velocity in the next frame.
• Any blob is represent by RBG color, so its
  possible to overcome occlusion.
• In the subsequent frame Mahalanobis distance to
  determine the blob that is most likely to have
  the same identity.

10
(No Transcript)
11
Behavior Models

• Individual behavior and person-to-person
  interaction
• Behavior modeling using Statistical Directed
  Acyclic Graphs (DAG) or Probabilistic Inference
  Network (PIN).
• HMM (Hidden Markov Models) and CHMM (Coupled
  Hidden Markov Models) can be viewed as temporal
  PIN or DAG.They constitute a simple graphical
  way of representing casual dependencies between
  variables.they can hold incomplete data as well
  as uncertainty. They are trainable and easy to
  avoid overfitting.they has algorithms for doing
  predictions.

12
Behavior Models (Cont)
HMM and CHMM can offer framework for combining
prior knowledge and data and they are modular and
parallelizable. The goal is to represent the
behavior in terms of states and transition
between states over time.
13
(No Transcript)
14
Synthetic Behavioral Agents

• Synthetic agents that mimic human behavior in
  virtual environment.
• Agent can make 5 types of interaction and various
  kind of individual behavior.
• The virtual environment parameters were taken
  from real pedestrian scene.
• There are primitive behaviors and complex
  interactive ones.

15
Agent Behavior

• 5 types of interactions
• Inter1 Follow, reach and walk together.

16
Agent Behavior
Inter2 Approach, meet and go on separately.
17
Agent Behavior
Inter3 Approach, meet and go on
together. Inter4 Change direction in order to
meet. Approach, meet and continue
together. Inter5 Change direction in order to
meet. approach, meet and go on separately .
18
Experimental Results

• The goal is to have a system that will
  accurately interpret behavior and interaction
  within almost any pedestrian scene with little or
  no training
• Comparison of HMM and CHMM architectures.
• They used 11 to 75 sequences for training
  (depends on complexity and avoiding overfitting).
• The test was to classify
• 20 unseen new questions.

19
Experimental Results (cont)
Left CHMM (solid line) and HMM (dashed line)
black lines without no interaction and red with
no interaction Right ROC curve for CHMM (very
important in surveillance tasks).
HMM is the winner
20
Experimental Results (cont)

• models that were trained with synthetic data
  should be directly applicable to human data is
  it really possible?
• Data from the computer vision model was used
  (2D-blob features)Example of inter2The
  feature vectors is similar to the synthetic
  data.

21
Experimental Results (cont)

• HMM performed much worse then CHMM and they
  omitted their results.
• 2 type of learning synthetic only / synthetic
  and real data (8 example for each interaction
  only!)

22
Experimental Results (cont)

• Parameters sensitivity
• Models parameters (training with different
  agents dynamic) were changed by factors of 2.5
  and 5
• The same results obtained. (except for inter1)
• Conclusions
• The system is very robust and can be applied for
  real scene after very short practice with
  synthetic data only.
• The success in classifying interaction is almost
  100 without getting significant levels of false
  alarms.

23
Reverend Thomas Bayes 1702-1761