Ch. 4. Nonparameteric Techniques

1
Ch. 4. Nonparameteric Techniques
2
Nonparametric Density Estimation

• We have assumed that either
• The likelihoods p(x?i) were known (Likelihood
  Ratio Test)
• Or at least the parametric form of the
  likelihoods were known (Parameter Estimation)
• The methods that will be presented in this
  chapter do not afford such luxuries
• Instead, they attempt to estimate the density
  directly from the data without making any
  parametric assumptions about the underlying
  distribution

3
Histogram The simplest form of non-parametric
density estimation

• Divide the sample space into a number of bins and
  approximate the density at the center of each bin
  by the fraction of points in the training data
  that fall into the corresponding bin
• Drawbacks
• The final shape of the density estimate depends
  on the starting position of the bins
• The discontinuities of the estimate are not due
  to the underlying density, they are only an
  artifact of the chosen bin locations
• Curse of dimensionality, since the number of bins
  grows exponentially with the number of dimensions
• Histogram is unsuitable for most practical
  applications except for rapid visualization of
  results in one or two dimensions

4
Histogram and Curse of Dimensionality
5
Nonparametric Density Estimation
6
Nonparametric Density Estimation
7
Nonparametric Density Estimation
8
Nonparametric Density Estimation
9
Nonparametric Density Estimation
10
Method for Estimating Densities
11
Parzen Windows
12
Parzen Windows
13
Parzen Windows
As hn approaches 0, dn(x-xkc) approaches Dirac
delta function
14
Window (Kernel) Functions

• Gaussian
• Triangular
• Rectangular

15
Parzen Window Example
16
Parzen Window Estimation Example
17
Choice of Window Width

• The problem of choosing the window width is
  crucial in density estimation
• A large window width will over-smooth the density
  and mask the structure in the data
• A small window width will yield a density
  estimate that is spiky and very hard to interpret

18
Choice of Window Width
19
Parzen Window Estimation

• Unimodal Gaussian
• 100 data points were drawn the true density
  (left), the estimates using h1.06sN-1/5 (right)

• Bimodal Gaussian
• 100 data points were drawn the true density
  (left), the estimates using h1.06sN-1/5 (right)

20
Parzen Window Estimation Example
21
Parzen Window Estimation 1D Gaussian
For n??, estimates are the same as the true
distribution, regardless of window width
22
Parzen Window Estimation 2D Gaussian
23
Parzen Window Estimation 1D bimodal distribution
For n??, estimates are the same as the true
distribution, regardless of window width
24
Classification Based on Parzen Window Estimation
25
Classification Based on Parzen Window Estimation
26
Exercise
27
Kn Nearest Neighbor Estimation

• Remedy to the problem of the unknown best
  window function let the cell be a function of
  the training data

28
Kn Nearest Neighbor Estimation

• Eight points in 1-D k-NN density estimates, for
  k3 5

29
Kn Nearest Neighbor Estimation
30
Kn Nearest Neighbor Estimation
31
Kn Nearest Neighbor Estimation
32
Nearest Neighbor Rule
33
Nearest Neighbor Classification
34
Voronoi Tessellation

• Partitioning of the input space into Voronoi
  cells, each labeled
• by the category of the training points it
  contains

35
Metric and Nearest-Neighbor Classifier
36
Distance Metrics
37
Distance Metrics
38
(No Transcript)
39
Minimum Distance Classifier
40
The Uncritical Use of Euclidean Metric