Ch8' InstanceBased Learning

1
Ch8. Instance-Based Learning

• ?????
• ????????
• ???
• 2000.09.27

2
Contents

• Introduction
• k-Nearest Neighbor Learning
• Distance-Weighted Nearest Neighbor Algorithm
• Remarks on k-Nearest Neighbor Algorithm
• A Note on Terminology
• Locally Weighted Regression
• Locally Weighted Linear Regression
• Remarks on Locally Weighted Regression
• Radial Basis Functions
• Case-Based Reasoning
• Remarks on Lazy and Eager Learning

3
INTRODUCTION

• Key Idea
• Instance-based approaches construct a different
  approximation to the target function for each
  distinct query instance that must be classified
• It constructs only a local approximation
• in the set of similar related instances of the
  new query instance
• Learning task simply stores the presented
  training data
• Advantage
• Process only local instance space not in the
  entire
• It can use more complex, symbolic representations
  for instances
• Perform better when the target function is very
  complex, but can still be described by a
  collection of less complex local approximations
• Disadvantage
• The cost of classifying new instances can be high
• It considers all attributes of the instances when
  attempting to retrieve similar training example

4
k-Nearest Neighbor Learning

• k-Nearest Neighbor Learning
• The nearest neighbors
• Similar instances in terms of the standard
  Euclidean distance
• Training algorithm
• For each training example ltx,f(x)gt, add the
  example to the list training_examples
• Classification algorithm
• Given a query instance xq to be classified,
• Let x1xk denote the k instances from
  training_examples that are nearest to xq
• When discrete-valued target functions, returns
• Where d(a,b) 1 if a b and where d(a,b) 0
  otherwise
• When continuous-valued target functions, returns

5
k-Nearest Neighbor Learning

• Distance-Weighted Nearest Neighbor Algorithm
• Weight the contribution of each of the k
  neighbors according to their distance to the
  query point xq, giving greater weight to closer
  neighbors
• When discrete-valued target functions
• When continuous-valued target functions

6
k-Nearest Neighbor Learning

• Remarks on k-Nearest Neighbor Algorithm
• Advantage
• Highly effective inductive inference method for
  many practical problems
• Robust to noisy training data
• Inductive bias
• The classification of an instance xq will be most
  similar to the classification of other instances
  that are nearby in Euclidean distance
• Disadvantage
• The distance between instances is calculated
  based on all attributes of the instance (curse of
  dimensionality)
• Weight each attribute differently for overcoming
  this problem, that is, stretch the axes in the
  Euclidean space
• Leave-one-out cross-validation is effective

7
Locally Weighted Regression

• Locally Weighted Regression
• Constructs an explicit approximation to f over a
  local region surrounding xq
• f may be a linear function, a quadratic function,
  a multilayer neural network, or some other
  functional form
• Locally Weighted Linear Regression
• The target function f is approximated near xq
• Fit w0wn to minimize the error
• Gradient descent in neural net
• Gradient descent in locally weighted linear
  regression

8
Radial Basis Functions

• Radial Basis Functions
• Global function approximation by linear
  combination of many local approximations
• It can approximate any function with arbitrarily
  small error
• Closely related to neural network
• It can be described a two-layer network
• Figure 8.2
• It is trained in a two-stage process
• Determine the number k, determine Ku for each xu
• xu can be all training example
• xu can be scattered uniformly throughout the
  instance space
• xu can be clusters of instances
• Train the weights wu
• It is more efficient than Backpropagation by
  separated two-stage training process

9
Case-based Reasoning

• Case-based Reasoning
• Instances or cases may be represented by rich
  symbolic descriptions
• May require a similarity metric different from
  Euclidean distance
• Multiple retrieved cases may be combined to form
  the solution to the new problem
• This is similar to the k-Nearest neighbor
  approach
• But, relying on knowledge-based reasoning rather
  than statistical methods
• There may be a tight coupling between case
  retrieval, knowledge-based reasoning, and problem
  solving

10
Remarks on Lazy and Eager Learning

• Lazy learning
• Wait for query before generalizing
• k-Nearest neighbor, locally weighted regression,
  case-based reasoning
• Eager learning
• Generalize before seeing query
• Radial basis function, almost others
• Difference
• Eager learner must create global approximation
• Lazy learner can create many local approximations
• If they use same H, lazy can create more complex
  functions