Classification

1
Classification

• A task of induction to find patterns

2
Outline

• Data and its format
• Problem of Classification
• Learning a classifier
• Different approaches
• Key issues

3
Data and its format

• Data
• attribute-value pairs
• with/without class
• Data type
• continuous/discrete
• nominal
• Data format
• flat

4
Sample data
5
Induction from databases

• Inferring knowledge from data
• The task of deduction
• infer information that is a logical consequence
  of querying a database
• Who conducted this class before?
• Which courses are attended by Mary?
• Deductive databases extending the RDBMS

6
Classification

• It is one type of induction
• data with class labels
• Examples -
• If weather is rainy then no golf
• If
• If

7
Different approaches

• There exist many techniques
• Decision trees
• Neural networks
• K-nearest neighbors
• Naïve Bayesian classifiers
• Support Vector Machines
• Ensemble methods
• Co-training
• and many more ...

8
A decision tree
9
Inducing a decision tree

• There are many possible trees
• lets try it on the golfing data
• How to find the most compact one
• that is consistent with the data?
• Why the most compact?
• Occams razor principle
• Issue of efficiency w.r.t. optimality

10
Information gain
and

• Entropy -
• Information gain - the difference between the
  node before and after splitting

11
Building a compact tree

• The key to building a decision tree - which
  attribute to choose in order to branch.
• The heuristic is to choose the attribute with the
  maximum IG.
• Another explanation is to reduce uncertainty as
  much as possible.

12
Learn a decision tree
Outlook
sunny
overcast
rain
Humidity
Wind
YES
high
normal
strong
weak
NO
YES
NO
YES
13
K-Nearest Neighbor

• One of the most intuitive classification
  algorithm
• An unseen instances class is determined by its
  nearest neighbor
• The problem is it is sensitive to noise
• Instead of using one neighbor, we can use k
  neighbors

14
K-NN

• New problems
• lazy learning
• large storage
• An example
• How good is k-NN?

15
Naïve Bayes Classifier

• This is a direct application of Bayes rule
• P(CX) P(XC)P(C)/P(X)
• X - a vector of x1,x2,,xn
• Thats the best classifier you can build
• But, there are problems

16
NBC (2)

• Assume conditional independence between xis
• We have
• An example
• How good is it in reality?

17
Classification via Neural Networks
Squash
?
A perceptron
18
What can a perceptron do?

• Neuron as a computing device
• To separate a linearly separable points
• Nice things about a perceptron
• distributed representation
• local learning
• weight adjusting

19
Linear threshold unit

• Basic concepts projection, thresholding

W vectors evoke 1
W .11 .6
L .7 .7
.5
20
E.g. 1 solution region for AND problem

• Find a weight vector that satisfies all the
  constraints

AND problem 0 0 0 0 1 0 1 0 0 1
1 1
21
E.g. 2 Solution region for XOR problem?
XOR problem 0 0 0 0 1 1 1 0 1 1
1 0
22
Learning by error reduction

• Perceptron learning algorithm
• If the activation level of the output unit is 1
  when it should be 0, reduce the weight on the
  link to the ith input unit by rLi, where Li is
  the ith input value and r a learning rate
• If the activation level of the output unit is 0
  when it should be 1, increase the weight on the
  link to the ith input unit by rLi
• Otherwise, do nothing

23
Multi-layer perceptrons

• Using the chain rule, we can back-propagate the
  errors for a multi-layer perceptrons.

Output layer
Hidden layer
Input layer