Quadratic Classifiers (QC)

1
Quadratic Classifiers(QC)
2010 Scientific Computing

• J.-S. Roger Jang (???)
• CS Dept., National Taiwan Univ.
• http//mirlab.org/jang
• jang_at_mirlab.org

2
Bayes Classifier

• Bayes classifier ? A probabilistic framework for
  classification problem
• Conditional probability
• Bayes theorem

3
PDF Modeling

• Goal
• Find a PDF (probability density function) that
  can best describe a given dataset
• Steps
• Select a class of parameterized PDF
• Identify the parameters via MLE (maximum
  likelihood estimate) based on a given set of
  sample data
• Commonly used PDFs
• Multi-dimensional Gaussian PDF
• Gaussian mixture models (GMM)

4
PDF Modeling for Classification

• Procedure for classification based on PDF
• Training stage PDF modeling of each class based
  on the training dataset
• Test stage For each entry in the test dataset,
  pick the class with the max. PDF
• Commonly used classifiers
• Quadratic classifier (n-dim. Gaussian PDF)
• Gaussian-mixture-model classifier (GMM PDF)

5
1D Gaussian PDF Modeling

• 1D Gaussian PDF
• MLE of m and s
• Detailed derivation

6
1D Gaussian PDF Modeling via MLE

• MLE Maximum Likelihood Estimate
• Given a set of observations, find the parameters
  of the PDF such that the overall likelihood is
  maximized.
• Detailed derivation

7
1D Gaussian PDF Modeling via MLE
Normal dist. estimated by normal dist.
Uniform dist. estimated by normal dist.
8
d-dim. Gaussian PDF Modeling

• d-dim. Gaussian PDF g(x, m, S)
• MLE of m and S
• Detailed derivation

9
d-dim. Gaussian PDF Modeling

• d-dim. Gaussian PDF g(x, m, S)
• Likelihood of x in class j (governed by g(x, mj,
  Sj))

10
2D Gaussian PDF

• Bivariate normal density

Density function
Contours
11
2D Gaussian PDF Modeling

• gaussianMle.m

12
Steps of QC

• Training stage
• Select a type of Gaussian PDF
• Identify the PDF of each class
• Test stage
• Assign each sample to the class with the highest
  PDF value

Quiz!
13
Characteristics of QC

• If each class is modeled by an Gaussian PDF, the
  decision boundary between any two classes is a
  quadratic function.
• That is why it is called quadratic classifier.
• How to prove it?
• Different selections of the covariance matrix
• Constant times an identity matrix
• Diagonal matrix
• Full matrix (hard to use if the input dimension
  is large)

14
QC Results on Iris Dataset (I)

• Dataset IRIS dataset with the last two inputs

15
QC Results on Iris Dataset(II)

• PDF for each class

16
QC Results on Iris Dataset (III)

• Decision boundaries among classes

17
Quiz

• Quiz about QC
• Why the classifier is named "quadratic"?
• How do you train a quadratic classifier?
• How do you evaluate (test) a quadratic
  classifier?
• What is the major strength of a quadratic
  classifier?
• What is the major weakness of a quadratic
  classifier?
• If a quadratic classifier has a diagonal
  covariance matrix, does it fall back to a naive
  Bayes classifier? Why?