Sparsity Control for Robust Principal Component Analysis

1
Sparsity Control for Robust Principal Component
Analysis

• Gonzalo Mateos and Georgios B. Giannakis
• ECE Department, University of Minnesota
• Acknowledgments NSF grants no. CCF-1016605,
  EECS-1002180

Asilomar Conference November 10, 2010
2
Principal Component Analysis

• Motivation (statistical) learning from
  high-dimensional data

• Principal component analysis (PCA) Pearson1901
• Extraction of low-dimensional data structure
• Data compression and reconstruction
• PCA is non-robust to outliers Jolliffe86

• Our goal robustify PCA by controlling outlier
  sparsity

2
3
Our work in context

• Contemporary applications
• Anomaly detection in IP networks Huang et
  al07, Kim et al09
• Video surveillance, e.g., Oliver et al99

• Robust PCA
• Robust covariance matrix estimators
  Campbell80, Huber81
• Computer vision Xu-Yuille95, De la
  Torre-Black03
• Low-rank matrix recovery from sparse errors
  Wright et al09

• Hubers M-class and sparsity in linear regression
  Fuchs99

3
4
PCA formulations

• Training data

• Minimum reconstruction error
• Dimensionality reduction operator
• Reconstruction operator

• Maximum variance

• Factor analysis model

Solution
4
5
Robustifying PCA

• Least-trimmed squares (LTS) regression
  Rousseeuw87

LTS-based PCA for robustness
(LTS PCA)
is the -th order statistic among
Trimming constant determines breakdown point

• Q How should we go about minimizing ?

(LTS PCA) is nonconvex existence of
minimizer(s)?
A Try all subsets of size , solve, and
pick the best

• Simple but intractable beyond small problems

5
6
Modeling outliers
inlier

• Introduce auxiliary variables s.t.

outlier

• Inliers obey outliers
  something else
• Inlier noise are zero-mean i.i.d.
  random vectors

• Remarks
• and are
  unknown
• If outliers sporadic, then vector is sparse!

6
7
LTS PCA as sparse regression

• Lagrangian form

(P0)

• Justifies the model and its estimator (P0) ties
  sparsity with robustness

7
8
Just relax!

• (P0) is NP-hard relax

(P2)

• Role of sparsity controlling is central

• Q Does (P2) yield robust estimates ?

A Yap! Huber estimator is a special case
9
Entrywise outliers

• Use -norm regularization

(P1)
10
Alternating minimization
(P1)
10
11
Refinements

• Nonconvex penalty terms approximate better
  in (P0)

11
12
Online robust PCA

• Motivation Real-time data and memory limitations

• Exponentially-weighted robust PCA

12
13
Video surveillance
13
Data http//www.cs.cmu.edu/ftorre/
14
Online PCA in action

• Inliers

• Outliers

• Figure of merit angle between and

14
15
Concluding summary

• Sparsity control for robust PCA
• LTS PCA as -(pseudo)norm regularized regression
  (NP-hard)
• Relaxation (group)-Lassoed PCA M-type
  estimator
• Sparsity controlling role of central

• Batch and online robust PCA algorithms
• i) Outlier identification, ii) Robust subspace
  tracking
• Refinements via nonconvex penalty terms

• Tests on real video surveillance data for anomaly
  extraction

• Ongoing research
• Preference measurement conjoint analysis and
  collaborative filtering
• Robustifying kernel PCA and blind dictionary
  learning

15