Classification algorithm overview

1
Classification algorithm overview

• LING 572
• Fei Xia
• Week 2 1/9/06

2
New time for lab sessions

• Time 3-4pm on Thursday (right after class)
  starting from this week.
• Location MGH 271
• Please feel free to bring your laptop
• Who did not receive the announcement?

3
Assignments

• Hand out every Thursday, explained at the lab
  session on the same day.
• Due at 11pm the Saturday in the following week.
• Ex Hw2 hand out on 1/11, due 1/20.
• ESubmit is ready for Hw1-Hw3. Give it a try.

4
Last time

• Course overview
• Mathematical foundation
• Basic concepts in the classification problem

5
Questions?
6
Important concepts

• Instance, InstanceList
• Labeled data, unlabeled data
• Training data, test data
• Feature, feature template
• Feature vector
• Attribute-value table
• Trainer, classifier
• Training stage, test stage

7
Training stage

• Estimate parameters
• Trainer InstanceList ? Classifier
• Mallet
• NaiveBayesTrainer t new NaiveBayesTrainer
  (parameters)
• Classifier c t.train (instanceList)

8
Input to learners attribute-value table
9
Output of learners a classifier

• A classifier
• f(x) y, x is input, y 2 C
• f(x) (ci, scorei), ci 2 C.
• A classifier fills out a decision matrix.

10
Testing stage

• Input new instances.
• Output a decision matrix.
• task find the best solution for new instances
• Classifier instance ? classification
• Malllet
• Classification cl c.classify (instance)

11
Evaluation

• Precision TP/(TPFP)
• Recall TP/(TPFN)
• F-score 2PR/(PR)
• Accuracy(TPTN)/(TPTNFPFN)
• F-score or Accuracy?
• Why F-score?

12
Steps for using a classifier
13
Steps for using a classifier

• Convert the task into a classification problem
  (optional)
• Split data into training/test/validation
• Convert the data into attribute-value table
• Training
• Decoding
• Evaluation

14
Important subtasks (for you)

• Converting the data into attribute-value table
• Define feature types
• Feature selection
• Convert an instance into a feature vector
• Understanding training/decoding algorithms for
  various algorithms.

15
Notation
16
How learners differ?
17
How learners differ?

• Modeling
• Training stage
• Test (decoding) stage

18
Modeling

• What to optimize given data x, find the class c
  that maximizes
• P(x, c)
• P(c x)
• P(x c)
• Decomposition
• Which variable conditions on which variable?
• What independence assumptions?

19
An example of different modeling
20
Two types of parameters

• Model parameters ones learned during the
  training. They are stored in a classifier.
• Ex Naïve Bayes
• p(ci) and p(fk ci)
• Internal (Non-model) parameters ones used to
  initialize a trainer, select features, etc.
• Ex iteration number, threshold for feature
  selection, Gaussian prior for MaxEnt, etc.

21
How learners differ?

• Modeling
• What function to optimize?
• How does the decomposition work?
• What kind of assumption is made?
• How many types of model parameters?
• How many internal (or non-model) parameters?
• How to handle multi-class problem?
• How to handle non-binary features?

22
How learners differ? (cont)

• Training how to estimate parameters?
• Decoding how to find the best solution?
• Weakness and strengths
• Simplicity (conceptual)
• Efficiency at training
• Efficiency at testing time
• Handling multi-class
• Theoretical validity
• Predication accuracy
• Stability and robustness
• Interpretablity
• Scalability
• Output topN
• Sparse data problem e.g., split data

23
A comparison chart(From Thorsten Joachims 2006
slides)
24
Topics not covered in this course

• Classification algorithms
• Neural network
• SVM
• Other ML methods
• Many sequence labeling models CRF
• Graphical models
• Theoretical properties
• Incremental induction

25
Assumptions made in this course

• Attribute-value table
• All attribute values are known.
• All the rows are available at the beginning. No
  incremental induction.
• Evaluation
• Different types of misclassification errors are
  equally important. If not, the evaluation metrics
  should be changed.

26
Using classification algorithms for
non-classification problems

• Regression problem the target attribute has an
  infinite number of values.
• Sequence labeling problem
• POS tagging
• Parsing