Multiclass Classification

1
Multiclass Classification

• Sariel Har-Peled, Dan Roth, Dav Zimak
• Department of Computer Science
• University of Illinois at Urbana-Champaign

2
Multiclass Classification in NLP

• Name/Entity Recognition
• Label people, locations, and organizations in a
  sentence
• PER Sam Houston,born in LOC Virginia, was
  a member of the ORG US Congress.
• Decompose into sub-problems
• Sam Houston, born in Virginia... ?
  (PER,LOC,ORG,?) ? PER (1)
• Sam Houston, born in Virginia... ?
  (PER,LOC,ORG,?) ? None (0)
• Sam Houston, born in Virginia... ?
  (PER,LOC,ORG,?) ? LOC (2)
• Many problems in NLP are decomposed this way
• Disambiguation tasks
• POS Tagging
• Word-sense disambiguation
• Verb Classification
• Semantic-Role Labeling

3
Transform the sub-problems

• Sam Houston, born in Virginia... ?
  (PER,LOC,ORG,?) ? PER (1)
• Transform each problem to feature vector
• Sam Houston, born in Virginia
• ? (Bob-, JOHN-, SAM HOUSTON, HAPPY, -BORN,
  --BORN,... )
• ? ( 0 , 0 , 1
  , 0 , 1 , 1 ,... )
• Transform each label to a class label
• PER ? 1
• LOC ? 2
• ORG ? 3
• ? ? 0
• Input 0,1d or Rd
• Output 0,1,2,3,...,k

4
Solving multiclass with binary learning

• Multiclass classifier
• Function f Rd ? 1,2,3,...,k
• Decompose into binary problems
• Not always possible to learn
• No theoretical justification (unless the problem
  is easy)

5
What we solve

• General framework
• Extend binary algorithms
• Theoretically justified
• Provably correct
• Generalizes well
• Verified Experimentally
• Naturally extends binary classification
  algorithms to mulitclass setting
• e.g. Linear binary separation induces linear
  boundaries in multiclass setting

6
Constraint Classification

• Examples (x,y)
• y ? Sk
• Sk partial order over class labels 1,...,k
• defines preference relation (lt) for class
  labels
• e.g. Multiclass 2lt1, 2lt3, 2lt4, 2lt5
• e.g. Multilabel 1lt3, 1lt4, 1lt5, 2lt3, 2lt4, 4lt5
• Constraint Classifier
• f X ? Sk
• f(x) is a partial order
• f(x) is consistent with y if (iltj) ? y ? (iltj)
  ?f(x)

7
Margin Generalization Bounds

• Linear Hypothesis space
• h(x) argsort vi.x
• vi, x ?Rd
• argsort returns permutation of 1,...,k
• CC margin-based bound
• ? min(x,y)?S min (i lt j)?y vi.x vj.x

• m - number of examples
• R - maxx x
• ? - confidence
• C - average constraints

8
VC-style Generalization Bounds

• Linear Hypothesis space
• h(x) argsort vi.x
• vi, x ?Rd
• argsort returns permutation of 1,...,k
• CC VC-based bound

• m - number of examples
• d - dimension of input space
• delta - confidence
• k - number of classes

9
Constraint Classification for NLP

• Represent relationship among output classes
• PER gt LOC, PER gt ORG ? Multiclass label PER
• Learn the relationship
• Multiclass
• Can robustly learn hypothesis space
• Provablly!
• Empircally!

10
Multilabel and Ranking Classification

• Woman
• Golfer
• Person
• Player

Multilabel Classification has l labels per example
11
Category Ranking (1)

• John 1 3 1
• Bob 4 1 4
• Scott 5 2 ?
• Natural order (bad) 1 ? 2 ? 3 ? 4 ? 5 (good)
• Multiclass/multilabel problem with a ranking over
  the classes.

5gt4gt3gt2gt1
12
The End