K nearest neighbor and Rocchio algorithm

1
K nearest neighborand Rocchio algorithm

• LING 572
• Fei Xia
• 1/11/2007

2
Announcement

• Hw2 is online now. It is due on Jan 20.
• Hw1 is due at 11pm on Jan 13 (Sat).
• Lab session after the class.
• Read DT Tutorial before next Tues class.

3
K-Nearest Neighbor (kNN)
4
Instance-based (IB) learning

• No training store all training instances.
• ? Lazy learning
• Examples
• kNN
• Locally weighted regression
• Radial basis functions
• Case-based reasoning
• The most well-known IB method kNN

5
kNN
6
kNN

• For a new document d,
• find k training documents that are closest to d.
• perform majority voting or weighted voting.
• Properties
• A lazy classifier. No training.
• Feature selection and distance measure are
  crucial.

7
The algorithm

• Determine parameter K
• Calculate the distance between query-instance and
  all the training instances
• Sort the distances and determine K nearest
  neighbors
• Gather the labels of the K nearest neighbors
• Use simple majority voting or weighted voting.

8
Picking K

• Use N-fold cross validation pick the one that
  minimizes cross validation error.

9
Normalizing attribute values

• Distance could be dominated by some attributes
  with large numbers
• Ex features age, income
• Original data x1(35, 76K), x2(36, 80K),
  x3(70, 79K)
• Assume age 2 0,100, income 2 0, 200K
• After normalization x1(0.35, 0.38),
• x2(0.36, 0.40), x3 (0.70, 0.395).

10
The Choice of Features

• Imagine there are 100 features, and only 2 of
  them are relevant to the target label.
• kNN is easily misled in high-dimensional space.
• ? Feature weighting or feature selection

11
Feature weighting

• Stretch j-th axis by weight wj,
• Use cross-validation to automatically
• choose weights w1, , wn
• Setting wj to zero eliminates this dimension
  altogether.

12
Similarity measure

• Euclidean distance
• Weighted Euclidean distance
• Similarity measure cosine

13
Voting

• Majority voting
• c arg maxc ?i ?(c, fi(x))
• Weighted voting weighting is on each neighbor
• c arg maxc ?i wi ?(c, fi(x))
• wi 1/dist(x, xi)
• ? We can use all the training examples.

14
Summary of kNN

• Strengths
• Simplicity (conceptual)
• Efficiency at training no training
• Handling multi-class
• Stability and robustness averaging k neighbors
• Predication accuracy when the training data is
  large
• Weakness
• Efficiency at testing time need to calc all
  distances
• Theoretical validity
• It is not clear which types of distance measure
  and features to use.

15
Rocchio Algorithm
16
Relevance Feedback for IR

• The issue plane vs. aircraft
• Take advantage of user feedback on relevance of
  docs to improve IR results
• User issues a short, simple query
• The user marks returned documents as relevant or
  non-relevant.
• The system computes a better representation of
  the information need based on feedback.
• Relevance feedback can go through one or more
  iterations.
• Idea it may be difficult to formulate a good
  query
• when you dont know the collection well,
  so iterate

17
Rocchio Algorithm

• The Rocchio algorithm incorporates relevance
  feedback information into the vector space model.
• Want to maximize sim (Q, Cr) - sim (Q, Cnr)
• The optimal query vector for separating relevant
  and non-relevant documents (with cosine sim.)
• Qopt optimal query Cr set of relevant doc
  vectors N collection size

18
Rocchio 1971 Algorithm (SMART)
qm modified query vector q0 original query
vector ?, ?, ? weights Dr set of known
relevant doc vectors Dnr set of known
irrelevant doc vectors
19
Relevance feedback assumptions

• Relevance prototypes are well-behaved.
• Term distribution in relevant documents will be
  similar
• Term distribution in non-relevant documents will
  be different from those in relevant documents
• Either All relevant documents are tightly
  clustered around a single prototype.
• Or There are different prototypes, but they have
  significant vocabulary overlap.
• Similarities between relevant and irrelevant
  documents are small.

20
Rocchio Algorithm for text classification

• Training time construct a set of prototype
  vectors, one vector per class.
• Testing time for a new document, find the most
  similar prototype vector.

21
Training time
Cj the set of positive examples for class
j. D the set of positive and negative
examples ?, ? weights
22
Why this formula?

• Rocchio shows when ??1, each prototype vector
  maximizes
• How maximizing this formula connects to the
  classification accuracy?

23
Testing time

• Given a new document d

24
kNN vs. Rocchio

• kNN
• Lazy learning no training
• Use all the training instances at testing time.
• Rocchio algorithm
• At the training time, calculate prototype
  vectors.
• At the test time, instead of using all the
  training instances, use only prototype vectors.
• Linear classifier not as expressive as kNN.

25
Summary of Rocchio

• Strengths
• Simplicity (conceptual)
• Efficiency at training
• Efficiency at testing time
• Handling multi-class
• Weakness
• Theoretical validity
• Stability and robustness
• Predication accuracy it does not work well when
  the categories are not linearly separable.

26
Additional slides
27
Three major design choices

• The weight of feature fk in document di e.g.,
  tf-idf
• The document length normalization
• The similarity measure e.g., cosine

28
Extending Rocchio?

• Generalized instance set (GIS) algorithm (Lam and
  Ho, 1998).