Change%20Detection%20in%20Stochastic%20Shape%20Dynamical%20Models%20with%20Application%20to%20Activity%20Recognition

1
Change Detection in Stochastic Shape Dynamical
Models with Application to Activity Recognition

• Namrata Vaswani
• Thesis Advisor Prof. Rama Chellappa

2
Acknowledgements

• Part of this work is joint work with Dr Amit Roy
  Chowdhury and Prof Rama Chellappa.

3
The Group Activity Recognition Problem
4
Problem Formulation

• The Problem
• Model activities performed by a group of moving
  and interacting objects (which can be people or
  vehicles or robots or diff. parts of human body).
  Use the models for abnormal activity detection
  and tracking
• Our Approach
• Treat objects as point objects landmarks.
• Changing configuration of objects deforming
  shape
• Abnormality change from learnt shape dynamics
• Related Approaches for Group Activity
• Co-occurrence statistics, Dynamic Bayes Nets

5
The Framework

• Define a Stochastic State-Space Model (a
  continuous state HMM) for shape deformations in a
  given activity, with shape scaled Euclidean
  motion forming the hidden state vector and
  configuration of objects forming the observation.
  
• Use a particle filter to track a given
  observation sequence, i.e. estimate the hidden
  state given observations.
• Define Abnormality as a slow or drastic change in
  the shape dynamics with unknown change
  parameters. We propose statistics for slow
  drastic change detection.

6
Overview

• The Group Activity Recognition Problem
• Slow and Drastic Change Detection
• Landmark Shape Dynamical Models
• Applications, Experiments and Results
• Principal Components Null Space Analysis
• Future Directions Summary of Contributions

7
A Group of People Abnormal Activity Detection
Normal Activity
Abnormal Activity
8
Human Action Tracking
Cyan Observed Green Ground Truth Red SSA
Blue NSSA
9
A Group of Robots
10
Applications to Speech/Audio Processing?

• Slow change detection
• Shape Dynamical Models Entire framework
  applicable to observations coming from acoustic
  sensors, only observation model will change
• Sensor fusion is easy combine audio/video
• PCNSA an algorithm for high dimensional pattern
  classification, applicable here as well

11
Overview

• The Group Activity Recognition Problem
• Slow and Drastic Change Detection
• Landmark Shape Dynamical Models
• Applications, Experiments and Results
• Principal Components Null Space Analysis

12
The Problem

• General Hidden Markov Model (HMM) Markov state
  sequence Xt, Observation sequence Yt
• Finite duration change in system model which
  causes a permanent change in probability
  distribution of state
• Change is slow or drastic Tracking Error
  Observation Likelihood do not detect slow
  changes. Use dist. of Xt
• Change parameters unknown use Log-Likelihood(Xt)
  
• State is partially observed use MMSE estimate of
  LL(Xt) given observations, ELL(Xt)Y1t) ELL
• Nonlinear dynamics Particle filter to estimate
  ELL

13
The General HMM
Qt(Xt1Xt)

State, Xt
State, Xt1
Xt2
?t(YtXt)
Observation, Yt1
Observation, Yt
Yt2
14
Related Work

• Change Detection using Particle filters, unknown
  change parameters
• CUSUM on Generalized LRT (Y1t) assumes finite
  parameter set
• Modified CUSUM statistic on Generalized LRT
  (Y1t)
• Testing if uj Pr(YtltyjY1t-1) are uniformly
  distributed
• Tracking Error (TE) error b/w Yt its
  prediction based on past
• Threshold negative log likelihood of observations
  (OL)
• All of these approaches use observation
  statistics Do not detect slow changes
• PF is stable and hence is able to track a slow
  change
• Average Log Likelihood of i.i.d. observations
  used often
• But ELL E-LL(Xt)Y1t (MMSE of LL given
  observations) in context of general HMMs is new

15
Particle Filtering
16
Change Detection Statistics

• Slow Change Propose Expected Log Likelihood
  (ELL)
• ELL Kerridge Inaccuracy b/w pt (posterior) and
  pt0 (prior)
• ELL(Y1t )E-log pt0 (Xt)Y1tEp-log pt0
  (Xt)K(pt pt0)
• A sufficient condition for detectable changes
  using ELL
• EELL(Y1t0) K(pt0pt0)H(pt0),
  EELL(Y1tc) K(ptcpt0)
• Chebyshev Inequality With false alarm miss
  probabilities of 1/9, ELL detects all changes
  s.t.
• K(ptcpt0) -H(pt0)gt3 vVarELL(Y1tc)
  vVarELL(Y1t0)
• Drastic Change ELL does not work, use OL or TE
• OL Neg. log of current observation likelihood
  given past
• OL -log Pr(YtY0t-1,H0) -logltptt-1 ,
  ?tgt
• TE Tracking Error. If white Gaussian observation
  noise, TE OL

17
ELL OL Slow Drastic Change

• ELL fails to detect drastic changes
• Approximating posterior for changed system
  observations using a PF optimal for unchanged
  system error large for drastic changes
• OL relies on the error introduced due to the
  change to detect it
• OL fails to detect slow changes
• Particle Filter tracks slow changes correctly
• Assuming change till t-1 was tracked correctly
  (error in posterior small), OL only uses change
  introduced at t, which is also small
• ELL uses total change in posterior till time t
  the posterior is approximated correctly for a
  slow change so ELL detects a slow change when
  its total magnitude becomes detectable
• ELL detects change before loss of track, OL
  detects after

18
A Simulated Example

• Change introduced in system model from t5 to t15

OL
ELL
19
Practical Issues

• Defining pt0(x)
• Use part of state vector which has linear
  Gaussian dynamics can define pt0(x) in closed
  form
• OR
• Assume a parametric family for pt0(x), learn
  parameters using training data
• Declare a change when either ELL or OL exceed
  their respective thresholds.
• Set ELL threshold to a little above H(pt0)
• Set OL threshold to a little above
  EOL0,0H(YtY1t-1)
• Single frame estimates of ELL or OL may be noisy
• Average the statistic or average no. of detects
  or modify CUSUM

20
Change Detection
Yes
Change (Slow)
ELLEp-log pt0(Xt) gt Threshold?
PF
ptt-1N pttN ptN
Yt
Yes
Change (Drastic)
OL -logltptt-1 , ?tgt gt Threshold?
21
Approximation Errors

• Total error lt Bounding error Exact filtering
  error PF error
• Bounding error Stability results hold only for
  bounded fns but LL is unbounded. So approximate
  LL by min-log pt0(Xt),M
• Exact filtering error Error b/w exact filtering
  with changed system model with original model.
  Evaluating ptc,0 (using Qt0 ) instead of ptc,c
  (using Qtc)
• PF Error Error b/w exact filtering with original
  model particle filtering with original model.
  Evaluating ptc,0,N which is a Monte Carlo
  estimate of ptc,0

22
Stability / Asymptotic Stability

• The ELL approximation error averaged over
  observation sequences PF realizations is
  eventually monotonically decreasing ( hence
  stable), for large enough N if
• Change lasts for a finite time
• Unnormalized filter kernels are mixing
• Certain boundedness (or uniform convergence of
  bounded approximation) assumptions hold
• Asymptotically stable if the kernels are
  uniformly mixing
• Use stability results of LeGland Oudjane
• Analysis generalizes to errors in MMSE estimate
  of any fn of state evaluated using a PF with
  system model error

23
Unnormalized filter kernel mixing

• Unnormalized filter kernel, Rt, is state
  transition kernel,Qt, weighted by likelihood of
  observation given state
• Mixing measures the rate at which the
  transition kernel forgets its initial condition
  or eqvtly. how quickly the state sequence becomes
  ergodic. Mathematically,
• Example LeGland et al State transition, Xt
  Xt-1nt is not mixing. But if Yth(Xt)wt, wt
  is truncated noise, then Rt is mixing

24
Complementary Behavior of ELL OL

• ELL approx. error, etc,0, is upper bounded by an
  increasing function of OLkc,0, tclt k lt t
• Implication Assume detectable change i.e.
  ELLc,c large
• OL fails gt OLkc,0,tcltkltt small gt ELL error,
  etc,0 smallgt ELLc,0 large gt ELL detects
• ELL fails gt ELLc,0 small gtELL error, etc,0
  large gt at least one of OLkc,0,tcltkltt large gt
  OL detects

25
Rate of Change Bound

• The total error in ELL estimation is upper
  bounded by increasing functions of the rate of
  change (or system model error per time step)
  with all increasing derivatives.
• OLc,0 is upper bounded by increasing function of
  rate of change.
• Metric for rate of change (or equivalently
  system model error per time step) for a given
  observation Yt DQ,t is

26
The Bound
Assume Change for finite time, Unnormalized
filter kernels mixing, Posterior state space
bounded
27
Implications

• If change slow, ELL works and OL does not work
• ELL error can blow up very quickly as rate of
  change increases (its upper bound blows up)
• A small error in both normal changed system
  models introduces less total error than a perfect
  transition kernel for normal system large error
  in changed system
• A sequence of small changes will introduce less
  total error than one drastic change of same
  magnitude

28
Possible Applications

• Abnormal activity detection, Detecting motion
  disorders in human actions, Activity Segmentation
• Neural signal processing detecting changes in
  stimuli
• Congestion Detection
• Video Shot change or Background model change
  detection
• System model change detection in target tracking
  problems without the tracker loses track

29
Approximation Errors

• Total error lt Bounding error Exact filtering
  error PF error
• Bounding error Stability results hold only for
  bounded fns but LL is unbounded.
  BEELLtc,c-ELLtc,c,M
• Exact filtering error Error b/w exact filtering
  with original system model with changed model,
  MEELLtc,c,M-ELLtc,0,M
• PF Error Error b/w exact filtering with changed
  model particle filtering with changed model,
  PEELLtc,0,M-ELLtc,0,M,N

30
Asymptotic Stability

• If (i) Change lasts for finite time, (ii)
  Unnormalized filter kernels are uniformly mixing,
  (iii) Bounded posterior state space increase
  of Mt (bound on expected value of LL) with t is
  polynomial and (iv) E?t lt ? for all t, then
• If (i), (ii), (iii) LL unbounded but expected
  value of its bounded approx. converges to true
  value uniformly in t and (iv), then
• Both 1. 2. imply Error avged. over obs.
  sequences PF runs is asymptotically stable

31
Stability

• If (i), (ii) Unnormalized filter kernels Mixing,
  (iii), then
• limN?8(Error avged over obs. seq. PF runs) is
  stable ?t eventually strictly decreasing
• 2. If (i), (ii) Mixing, (iii)Bounded
  posterior state space,
• limN?8(Error avged over PF runs)/Mt is stable
  almost surely for all obs. seq ?t strictly
  decreasing.

32
We have shown

• Asymptotic stability of errors in ELL estimation
  if change lasts for a finite time, unnormalized
  filter kernels are uniformly mixing some
  boundedness assumptions hold.
• Stability for large N if the kernels are only
  mixing
• ELL error upper bounded by an increasing fn. of
  OLc,0 ELL works when OL fails vice versa
• ELL error upper bounded by an increasing fn. of
  rate of change, with incr. derivatives of all
  orders. OLc,0 upper bounded by increasing fn. of
  rate of change
• Analysis generalizes to errors in MMSE estimate
  of any fn. of state evaluated using a PF with
  system model error

33
More Practical Issues

• Estimates from single frames are noisy and
  affected by outliers
• Average the no. of detects over past p time
  instants
• Or average the statistic over past p time
  instants
• aOL(p) (1/p) -log Pr(Yt-p1tY0t-p,H0)
• Either avg. ELL, aELL, or use joint ELL over
  past p states, jELL(p) (1/p)E-log
  pt-pt(Xt-pt)Yot
• Or, modify CUSUM for unknown change parameters,
  i.e. change if max1ltpltt dp gt ?, dp
  Statistic(p)Tp,t.

34
Overview

• The Group Activity Recognition Problem
• Slow and Drastic Change Detection
• Landmark Shape Dynamical Models
• Applications, Experiments and Results
• Principal Components Null Space Analysis

35
A Group of Human Body Parts
36
What is Shape?

• Shape is the geometric information that remains
  when location, scale and rotation effects are
  filtered out Kendall
• Shape of k landmarks in 2D
• Represent the X Y coordinates of the k points
  as a k-dimensional complex vector Configuration
• Translation Normalization Centered Configuration
• Scale Normalization Pre-shape
• Rotation Normalization Shape

37
Activities on the Shape Sphere in Ck-1
38
Related Work

• Related Approaches for Group Activity
• Co-occurrence Statistics
• Dynamic Bayesian Networks
• Shape for robot formation control
• Shape Analysis/Deformation
• Pairs of Thin plate splines, Principal warps
• Active Shape Models affine deformation in
  configuration space
• Deformotion scaled Euclidean motion of shape
  deformation
• Piecewise geodesic models for tracking on
  Grassmann manifolds
• Particle Filters for Multiple Moving Objects
• JPDAF (Joint Probability Data Association
  Filter) for tracking multiple independently
  moving objects

39
Motivation

• A generic and sensor invariant approach for
  activity
• Only need to change observation model depending
  on the landmark, the landmark extraction method
  and the sensor used
• Easy to fuse sensors in a Particle filtering
  framework
• Shape invariant to translation, zoom,
  in-plane rotation
• Single global framework for modeling and tracking
  independent motion interactions of groups of
  objects
• Co-occurrence statistics Req. individual joint
  histograms
• JPDAF Cannot model object interactions for
  tracking
• Active Shape Models good for only approx. rigid
  objects
• Particle Filter is better than the Extended
  Kalman Filter
• Able to get back in track after loss of track due
  to outliers,
• Handle multimodal system or observation process

40
The HMM

• Observation, Yt Centered configurations
• State, Xt?t, ct, st, ?t
• Current Shape (zt),
• Shape Velocity (ct) Tangent coordinate w.r.t.
  zt-1
• Scale (st),
• Rotation angle (?t)
• Use complex vector notation to simplify equations
• Use a particle filter to approximate the optimal
  non-linear filter, pt(dx) Pr(Xt?dxY0t)
  posterior state distribution conditioned on
  observations upto time t, by an N-particle
  empirical estimate of pt

41
Hidden Markov Shape Model
XtShape(zt), Shape Velocity(ct), Scale(st),
Rotation(?t)
Xt1
State Dynamics
Observation Model Yt h(Xt) wt ztstej?
wt wt i.i.d observation noise
Using complex notation
Observation, Yt Centered Configuration
Shape X Rotation, SO(2) X Scale, R Centered
Config, Ck-1
42
State Dynamics

• Shape Dynamics Linear Markov model on shape
  velocity
• Shape velocity at t in tangent space w.r.t.
  shape at t-1, zt-1
• Orthogonal basis of the tangent space, U(zt-1)
• Linear Gauss-Markov model for shape velocity
• Move zt-1 by an amount vt on shape manifold to
  get zt
• Motion (Scale, Rotation)
• Linear Gauss-Markov dynamics for log st,
  unwrapped ?t

43
The HMM
Observation Model Shape,Motion?Centered
Configuration
System Model Shape and Motion Dynamics
Motion Dynamics
Shape Dynamics

• Linear Gauss-Markov models for log st and ?t
• Can be stationary or non-stationary

44
Three Cases

• Non-Stationary Shape Activity (NSSA)
• Tangent space, U(zt-1), changes at every t
• Most flexible Detect abnormality and also track
  it
• Stationary Shape Activity (SSA)
• Tangent space, U(µ), is constant (µ is a mean
  shape)
• Track normal behavior, detect abnormality
• Piecewise Stationary Shape Activity (PSSA)
• Tangent space is piecewise constant, U(µk)
• Change time fixed or decided on the fly using
  ELL
• PSSA ELL Activity Segmentation

45
Stationary, Non-Stationary
Stationary Shape
Non-Stationary Shape
46
Stationary Shape Activity

• Mean shape is constant, so set ?t µ (Procrustes
  mean), for all t, ?t not part of state vector,
  learn mean shape using training data.
• Define a single tangent space w.r.t. µ shape
  dynamics simplifies to linear Gauss-Markov model
  in tangent space
• Since shape space is not a vector space, data
  mean may not lie in shape space, evaluate
  Procrustes mean an intrinsic mean on the shape
  manifold.

47
What is Procustes Mean?

• Proc mean µ, minimizes sum of squares of Proc
  distances of the set of pre-shapes from itself
• Proc distance is Euclidean distance between
  Proc fit of one pre-shape onto another
• Proc fit scale or rotate a pre-shape to
  optimally align it with another pre-shape
• Optimally minimum Euclidean distance between
  the two pre-shapes after alignment

48
Learning Procrustes Mean

• Procrustes mean of set of preshapes wi
  Dryden,Mardia

49
Procrustes DistanceDryden Mardia

• Translation Scale normaln. of config. ?
  Pre-Shape
• Procrustes fit of pre-shape w onto y
• Procrustes distance

50
Learning Stationary Shape Dynamics
Learn Procrustes mean, µ
µ
Learn AR model in tangent space
µ, Sv, A, Sn
51
Overview

• The Group Activity Recognition Problem
• Slow and Drastic Change Detection
• Landmark Shape Dynamical Models
• Applications, Experiments and Results
• Principal Components Null Space Analysis

52
Abnormal Activity Detection

• Define abnormal activity as
• Slow or drastic change in shape statistics with
  change parameters unknown.
• System is a nonlinear HMM, tracked using a PF
• This motivated research on slow drastic change
  detection in general HMMs
• Tracking Error detects drastic changes. We
  proposed a statistic called ELL for slow change.
• Use a combination of ELL Tracking Error and
  declare change if either exceeds its threshold.

53
Tracking to obtain observations

• CONDENSATION tracker framework
• State Shape, shape velocity, scale, rotation,
  translation, Observation Configuration vector
• Measurement model Motion detection locally
  around predicted object locations to obtain
  observation
• Predicted object configuration obtained by
  prediction step of Particle filter
• Predicted motion information can be used to move
  the camera (or any other sensor)
• Combine with abnormality detection for drastic
  abnormalities will not get observation for a set
  of frames, if outlier then only for 1-2 frames

54
Activity Segmentation

• Use PSSA model for tracking
• At time t, let current mean shape µk
• Use ELL w.r.t. µk to detect change time, tk1
  (segmentation boundary)
• At tk1, set current mean shape to posterior
  Procrustes mean of current shape, i.e.
• µk1largest eigenvector of EpztztSi1N
  zt(i) zt(i)
• Setting the current mean as above is valid only
  if tracking error (or OL) has not exceeded the
  threshold (PF still in track)

55
A Common Framework for

• Tracking
• Groups of people or vehicles
• Articulated human body tracking
• Abnormal Activity Detection / Activity Id
• Suspicious behavior, Lane change detection
• Abnormal action detection, e.g. motion disorders
• Human Action Classification, Gait recognition
• Activity Sequence Segmentation
• Fusing different sensors
• Video, Audio, Infra-Red, Radar

56
Possible Applications

• Modeling group activity to detect suspicious
  behavior
• Airport example
• Lane change detection in traffic
• Model human actions track a given sequence of
  actions, detect abnormal actions (medical
  application to detect motion disorders)
• Activity sequence segmentation unsupervised
  training
• Sensor independent Acoustic/Radar/Infra-Red,
  Sensor fusion
• Low level processing for Dynamic Bayesian
  Networks
• Medical Image Processing, e.g. Shape deformation
  models for heart

57
Apply to Gait Verification

• Model diff. parts of human body head, torso,
  fore hind arms legs as landmarks
• Learn the landmark shape dynamics for diff.
  peoples gait
• Verification Given a test seq. a possible
  match (say, from face recognition stage), verify
  if match is correct
• Start tracking test seq. using the shape
  dynamical model of the possible match
• If dynamics does not match at all, PF will lose
  track
• If dynamics is close but not correct, ELL w.r.t.
  the possible match will exceed its threshold

58
Gait Recognition

• System Identification approach
• Assuming the test seq. has negligible observation
  noise, learn the shape dynamical model parameters
  for the test sequence
• Find distance of parameters for test seq. from
  those for diff people in the database. Similar
  idea to Soatto, Ashok
• Match time series of shape velocity of probe
  gallery
• Save the shape velocity sequence for the diff
  people in database.
• Test sequence estimate the shape velocity seq.,
  use DTW Kale to match it against all peoples
  gait

59
Experiments and Results
60
Why Particle Filter?

• N does not increase (much) with incr. state dim.,
  Approx. posterior distrib. of state only in high
  probability regions (so fixed N works for all t
  state space at t Dt)
• Better than Extended KF because of asymptotic
  stability
• Able to track in spite of wrong initial
  distribution
• Get back in track after losing track due to an
  outlier observation
• Slowly changing system Able to track it and yet
  detect the change using ELL (explained later)
• Can handle Multi-Modal prior/posterior, EKF cannot

61
Time-Varying No. of Landmarks?

• Ill posed problem Interpolate the curve formed
  by joining the landmarks re-sample it to a
  fixed no. of landmarks k
• Experimented with 2 interpolation/re-sampling
  schemes
• Uniform Re-samples independently along x y
• Assumes observed landmarks are uniformly sampled
  from some continuous function of a dummy variable
  s
• All observed landmark get equal weight while
  re-sampling
• Very sensitive to change in of landmarks, but
  also able to detect abnormality caused by two
  closely spaced points
• Arc-length Parameterizes x y coordinates by
  the length of the arc upto that landmark
• Assumes observed landmarks are non-uniformly
  sampled points from a continuous fn. of length x
  (l), y (l)
• Smoothens out motion of closely spaced points,
  thus misses abnormality caused by two closely
  spaced points

62
Experiments

• Group Activity
• Normal activity Group of people deplaning
  walking towards airport terminal used SSA model
• Abnormality A person walks away in an un-allowed
  direction distorts the normal shape
• Simulated walking speeds of 1,2,4,16,32 pixels
  per time step (slow to drastic distortion in
  shape)
• Compared detection delays using TE and ELL
• Plotted ROC curves to compare performance
• Human actions
• Defined NSSA model for tracking a figure skater
• Abnormality abnormal motion of one body part
• Able to detect as well as track slow abnormality

63
Abnormality

• Abnormality introduced at t5
• Observation noise variance 9
• OL plot very similar to TE plot (both same to
  first order)

Tracking Error (TE)
ELL
64
ROC ELL

• Plot of Detection delay against Mean time b/w
  False Alarms (MTBFA) for varying detection
  thresholds
• Plots for increasing observation noise

Drastic Change ELL Fails
Slow Change ELL Works
65
ROC Tracking Error(TE)

• ELL Detection delay 7 for slow change ,
  Detection delay 60 for drastic
• TE Detection delay 29 for slow change,
  Detection delay 4 for drastic

Slow Change TE Fails
Drastic Change TE Works
66
ROC Combined ELL-TE

• Plots for observation noise variance 81
  (maximum)
• Detection Delay lt 8 achieved for all rates of
  change

67
Human Action Tracking
Normal Action SSA better than NSSA
Abnormality NSSA works, SSA fails
Green Observed, Magenta SSA, Blue NSSA
68
NSSA Tracks and Detects Abnormality
Abnormality introduced at t20
Tracking Error
ELL
Red SSA, Blue NSSA
69
Temporal Abnormality

• Abnormality introduced at t 5, Observation Noise
  Variance 81
• Using uniform re-sampling, Not detected using
  arc length

70
Overview

• The Group Activity Recognition Problem
• Slow and Drastic Change Detection
• Landmark Shape Dynamical Models
• Applications, Experiments and Results
• Principal Components Null Space Analysis

71
Typical Data Distributions
Apples from Apples problem All algorithms work
well
Apples from Oranges problem Worst case for
SLDA, PCA
72
PCNSA Algorithm

• Subtract common mean µ, Obtain PCA space
• Project all training data into PCA space,
  evaluate class mean, covariance in PCA space µi,
  Si
• Obtain class Approx. Null Space (ANS) for each
  class Mi trailing eigenvectors of Si
• Valid classification directions in ANS if
  distance between class means is significant
  WiNSA
• Classification Project query Y into PCA space,
  XWPCAT(Y- µ), choose Most Likely class, c, as

73
Classification Error Probability

• Two class problem. Assumes 1-dim ANS, 1 LDA
  direction
• Generalizes to M dim ANS and to non-Gaussian but
  unimodal symmetric distributions

74
Applications

• Image Video retrieval
• Applied to human action retrieval
• Hierarchical image/video retrieval PCNSA
  followed by LDA
• Activity Classification Abnormal Activity
  Detection

75
Applications
Face recognition, Large pose variation
Object recognition
Face recognition, Large expression variation
Facial Feature Matching
76
Discussion Ideas

• PCNSA test approximates the LRT (optimal Bayes
  solution) as condition no. of Si tends to
  infinity
• Fuse PCNSA and LDA get an algorithm very similar
  to Multispace KL
• For multiclass problems, use error probability
  expressions to decide which of PCNSA or SLDA is
  better for a given set of 2 classes
• Perform facial feature matching using PCNSA, use
  this for face registration followed by warping to
  standard geometry

77
Ongoing and Future Work

• Change Detection
• Implications of Bound on Errors is increasing fn.
  of rate of change
• CUSUM on ELL OL
• Quantitative performance analysis of ELL OL
• Find examples of mixing unnormalized filter
  kernels
• Non-Stationary Piecewise Stationary Shape
  Activities
• Application to sequences of different kinds of
  actions
• PSSA ELL for activity segmentation
• Joint tracking and abnormality detection
• Time varying number of Landmarks?
• What is best strategy to get a fixed no. k
  of landmarks?
• Can we deal with changing dimension of shape
  space?
• Multiple Simultaneous Activities, Multi-sensor
  fusion
• 3D Shape, General shape spaces

78
Contributions

• ELL for slow change detection, Stability of ELL
  approximation error
• Complementary behavior of ELL OL, ELL error
  proportional to rate of change with all
  increasing derivatives
• Stochastic dynamical models for landmark shapes
  NSSA, SSA, PSSA
• Modeling the changing configuration of a group of
  moving point objects as a deforming shape shape
  activity.
• Using ELL PSSA for activity segmentation
• PCNSA its error probability analysis,
  application to action retrieval, abnormal
  activity detection

79
Contributions

• Slow and drastic change detection in general HMMs
  using particle filters. We have shown
• Asymptotic stability / stability of errors in ELL
  approximation
• Complementary behavior of ELL OL for slow
  drastic changes
• Upper bound on ELL error is an increasing
  function of rate of change, with all increasing
  derivatives
• Stochastic state space models (HMMs) for
  simultaneously moving and deforming shapes.
• Stationary, non-stationary p.w. stationary
  cases
• Group activity human actions modeling,
  detecting abnormality
• NSSA for tracking slow abnormality, ELL for
  detecting it
• PSSA ELL Apply to activity segmentation

80
Other Contributions

• A linear subspace algorithm for pattern
  classification motivated by PCA
• Approximates the optimal Bayes classifier for
  Gaussian pdfs with unequal covariance matrices.
• Useful for apples from oranges type problems.
• Derived tight upper bound on its classification
  error probability
• Compared performance with Subspace LDA both
  analytically experimentally
• Applied to object recognition, face recognition
  under large pose variation, action retrieval.
• Fast algorithms for infra-red image compression

81
Special Cases

• For i.i.d. observation sequence, Yth(Xt)wt
• ELL(Y0t)E-log pt(Xt)Y0tE-log pt(Xt)Yt
• -log pt(h-1(Yt)-Eh-1(wt))- log
  pt(h-1(Yt))
• OL(Yt)const. if Eh-1(wt)0
• For zero (negligible) observation noise case,
  Yth(Xt)
• ELL(Y0t) E-log pt(Xt)Y0t -log
  pt(h-1(Yt))OL(Yt)const.

82
Particle Filtering Algorithm

• At t0, generate N Monte Carlo samples from
  initial state distribution, p0p0
• For all t,
• Prediction Given posterior at t-1 as an
  empirical distr., pt-1, sample from the state
  transition kernel Qt (xt-1,dxt) to generate
  samples from ptt-1
• Update/Correction
• Weight each sample of ptt-1 by probability of
  the observation given that sample, ?t(Ytx) pt
• Use multinomial sampling to resample from these
  weighted particles to generate particles
  distributed according to pt

83
Classification and Tracking Algorithms Using
Landmark Shape Analysis and their Application to
Face and Gait

• Namrata Vaswani
• Dept. of Electrical Computer Engineering
• University of Maryland, College Park
• http//www.cfar.umd.edu/namrata

84
Principal Component Null Space Analysis (PCNSA)
for Face Recognition
85
Related Work

• PCA uses projection directions with maximum
  inter-class variance but do not minimize the
  intra-class variance
• LDA uses directions that maximize the ratio of
  inter-class variance and intra-class variance
• Subspace LDA for large dimensional data, use PCA
  for dim. reduction followed by LDA
• Multi-space KL (similar to PCNSA)
• Other work ICA, Kernel PCA LDA, Neural nets

86
Motivation

• Example PCA or SLDA good for face recognition
  under small pose variation, PCNSA proposed for
  larger pose variation
• PCNSA addresses such Apples from Oranges type
  classification problems
• PCA assumes classes are well separated along all
  directions in PCA space Si Ss2I
• SLDA assumes all classes have similar directions
  of min. max. variance Si S, for all i
• If minimum variance direction for one class is
  maximum variance for the other a worst case for
  SLDA or PCA

87
Assumptions Extensions

• Assumptions required For each class,
• (i) an approx. null space exists,
• (ii) valid classification directions exist
• Progressive-PCNSA
• Defines a heuristic for choosing dimension of ANS
  when (ii) is not satisfied.
• Also defines a heuristic for new (untrained)
  class detection

88
Experimental Results

• PCNSA misclassified least, followed by SLDA and
  then PCA
• New class detection ability of PCNSA was better
• PCNSA most sensitive to training data size, PCA
  most robust