Representation for Classification

1
Representation for Classification

• By Abhishek

2
Coverage

• yAx solution and its Importance
• A leaf from Compressed Sensing
• Donohos Model and Constraints
• A quick digression into Linear Programming
• Experiments Results

3
yAx

• yAx , Why is it important to solve it
• Say y is the output and x, the input than
  major problems in AI are basically finding that
  specific A that solves the Equation.
• Depends a lot on A !!
• If Number of equations are lesser than number of
  variables,
• its an underdetermined matrix and so not
  possible to solve for a unique solution.
• of variables of observations
• That means we need a balance between feature-set
  and of examples.

4
yAx

• of variables of observations
• Is it true for all cases?
• Not really

5
Compressed Sensing

• Suppose x is an unknown vector in Rm (like in
  an image or a signal) we need say m samples to
  reconstruct it correctly.
• But if we know priori that x is compressible by
  a transform coding with a known transform, we are
  allowed to acquire data about x by measuring
  n general linear functionals rather than m
  samples.
• If such functionals are well chosen and allowing
  for a degree of reconstruction error, then size
  of n can be dramatically smaller than m that
  is being considered.

6
yAx Compressed Sensing

• yAx can be solved by least squares too
• But a better way to understand x is solving for
  the sparsest x that solves the equation.
• An exhaustive search for sparsest solution is
  Intractable, NP-Hard.

7
Compressed Sensing

• One standard important Result in this field
• Consider an Underdetermined system of linear
  equations yAx with known dxn matrix and known
  y.
• Generally finding the sparsest solution is
  NP-Hard, but there is a thresholding phenomenon
  involved.
• If sparsest solution is sufficiently sparse, then
  it can be found by linear programming

8
Compressed Sensing AI

• Why is the result so important?
• If we make sure that such thresholding
  constraints are maintained, we can find such
  sparse solutions with lesser number of
  observations compared to the number of feature
  set.
• We can beat the condition
• of variables of observations

9
AI

• Our Idea
• Recall yAx
• If y is a classifier output, A a set of
  features, we try to solve for x.
• What will it give us?
• In this formation, x works as a set of weights
  on which we are judging y
• Using this we can basically see which feature set
  was responsible for classification.

10
AI

• Why is this important?
• AI domain has been restricted mainly to
  classification methods with a lesser importance
  placed on the choice of features that can be used
  for classification.
• When yAx is underdetermined there is too much
  freedom in the model. So typically those feature
  sets are used, after multiple experiments, that
  have the best classifying boundary.
• We are concerned with designing a method to
  better choose these feature sets.

11
Donohos Results

• A bit of Mathematics
• (NP) min ??x??0 subject to yAx, x0
• where ??x??0 counts of zeros
• (LP) min ??x??1 subject to yAx, x0
• where ??x??1 is the sum vector xx1,x2
• (known as part of convex optimization)
• When solution X is sparse enough , NP LP

12
Donohos Results

• A bit of Mathematics
• Where does this come from?
• Convex Polytopes Are basically Convex Polygons
  in higher dimensional space.

13
Donohos Results

• A bit of Mathematics
• Neighborliness
• If any subset of k or less vertices are in vertex
  set of a face.
• Simplices in dimension 2 and 3 are shown below.

14
Donohos Results

• A bit of Mathematics
• Neighborliness
• Simplices have the highest neighborliness.
• Except for Simplices, no polytope is d/2
  Neighborly. Where d is the dimension

15
Donohos Results

• A bit of Mathematics
• Donohos proof in Neighborliness
• K-neighborliness is completely equivalent to the
  statement yAx has a non-negative solution with
  at most k non-zeros.

16
Donohos Results

• A bit of Mathematics
• Donoho Jared has a set of standard theoretical
  and empirical proof on the thresholding
  phenomenon that is how much is recoverable by L1
  minimization

17

• X-axis represents the sparseness ?d/n
• Remember d of observations,nsize of feature
  set (size of A)
• Y-axis represents ?Threshold function.
• Interpretation
• For d/n0.1, if X has lesser than 0.2d nonzeros
  it has high probability of recovery. (signed
  case)
• 0.25d nonzeros it has high probability of
  recovery.(unsigned case)

18
Linear Programming

• How Linear Programming is done?
• Convex Polytopes and Linear programming are
  analogous as the search takes place on the vertex
  of the polytopes.
• Linear Programming has a favorable property as
  the cost function is linear, the optima it finds
  is also a global optima.
• Usually one either by Simplex Method( Moving from
  vertex to vertex) or by Interior Point
  methods(like Primal Dual Method)
• Done here by using L1-magic package.

19
Linear Programming

• Example
• Multiple Inequalities create a polygon.
• Simplex method Solution can be obtained by
  moving along the vertices.
• Primal Dual Method Moving Inside the polygon
  based on a specific dual cost function. (this is
  also the base for other interior point methods)
• Worst case solution is in Polynomial Time.

Image from Wikipedia
20
Experiments

• Experimental Images taken from
• COIL-20 Database from Columbia.
• Set of 20 objects
• With 72 different
• Orientations.

21
Experiments

• yAx, where y is the classifier output, A is
  the feature set
• A is constructed using 3 filters, Gabor, Wavelet
  (Db2) and Steerable Filters
• For each image, filters are applied to image at
  different resolution ranging from 100 to about
  3.
• Then Histogram is applied to the filter output at
  a specified bin size (20-200)

22
Duck Image with 30 intensity Bins
23
Duck Vs.Pot
24
2 Cars
25
2 Containers
26
Cat vs...
27
References

• Books
• Branko Grunbaum, Convex Polytopes
• Gunter Ziegler, Lectures on Convex Polytopes
• Convex Optimization by Stephen Boyd and Lieven
  Vandenberghe (Available online)
• Papers
• Maximal Sparsity Representation via L1
  Minimization. DLD Michael Elad
• Sparse Nonnegative solution of underdetermined
  Linear Equations by Linear Programming. DLD JT
• Neigborliness of Random-Projected Simplices in
  High Dimensions. DLD JT
• Thresholds for Recovery of Sparse Solutions via
  L1 Minimization. DLD JT
• Compressed Sensing. DLD
• Extensions to Compressed Sensing. Yaako Tsaig
  DLD
• High-Dimensionsal Centrally-Symmetric Polytopes
  with Neighborliness Proportional to Dimension.
  DLD
• L1-Magic Recovery of Sparse Signals via Convex
  programming. Emmanuel Candes Justin Romberg.
• Software
• L1-Magic http//www.acm.caltech.edu/l1magic/
• Sparselab http//sparselab.stanford.edu/

28
Thanks