Primer: Kernel Density Estimation

1
Primer Kernel Density Estimation

• Problem estimate a probability density function
  (pdf) on the basis of a finite sample
• Credits Duda, Hart, Stork (2001) Tarn Duong
• Applications e.g. pattern classification
• Parametric vs. Non-parametric techniques
• In parametric methods you assume a certain
  functional form of the pdf and only need to
  estimate the parameters that best fit the data.
• In non-parametric methods you dont make an
  assumption about the functional form of the pdf.
• Kernel density estimation is an example of a
  non-parametric technique.

2
Histograms
piece-wise constant shape can depend on choice
of bin centers!
3
Kernel Density Estimation

• Idea replace fixed bins with bins centered over
  data points, this corresponds to applying a
  rectangular kernel

4

• Popular choice Gaussian kernel
• Note kernel also called Parzen window function

some techniques for automatic selection of kernel
width
5
(No Transcript)
6

• Extension to more dimensions just replace
  univariate kernel with multi-variate kernel
• Example p-dimensional Gaussian product kernel

7
(No Transcript)
8
An Example from NeuroscienceSpike Trains and
Firing Rates

• Consider single neuron record times of spiking,
  e.g., n spikes at times ti, i 1,,n
• Neural response function
• Idea express spikes as Dirac d functions
• Allows to re-express sums over spikes as
  integrals over time

9
Measuring Approximate Firing Rates
linear filter, w(.) is the kernel, satisfying
B,C fixed and sliding rectangular filters (box
filter)
width100ms, fixed
width100ms, sliding
D gaussian filter kernel
E alpha filter (causal)
s100ms
1/a100ms
half-wave rectification
10
Discussion

• kernel density estimation is nice because no
  assumptions are necessary (assumptions can always
  be wrong)
• guaranteed to yield correct approximation in the
  limit of infinite data (which you never have,
  though). See, e.g., Duda, Hart, Stork (2001),
  chap. 4.
• representing and evaluating the pdf requires
  storing and summing over all data points, which
  can be problematic for large N
• amount of samples required for good approximation
  grows exponentially with the dimensionality of
  the space (curse of dimensionality)
• much less data may be needed for a parametric
  technique, where only a few parameters need to be
  estimated