Negative Results and Open Problems

1
Negative Results and Open Problems
NIPS 2002 workshops

• Isabelle Guyon
• Clopinet

2
Why?

• Negative results are seldom reported. A wealth of
  knowledge is lost.
• Negative results are sometimes more informative
  that positive results.
• Negative results sometimes point to important
  open problems.

3
History

• We better pay attention to negative results
  neural networks have been killed several times
  by negative results.

time
SVMs (1992)
Minski Papert (1969)
4
Negative prejudices

• 1) Against multilayer networks
• 1949 - MinskyPapert XOR problem.
• 1983 - Kirkpatrick et al simulated annealing.
• 1985 - HintonSejnowski Boltzmann Machine.
• 1986 Backprop.

5
Negative prejudices

• 2) Against polynomials
• 1975 - T. Poggio kernel trick for polynomials.
• 1977 - J. Schürmann feature selection.
• 1984 -T. Kohonen popularizes Poggios results.
• Lost popularity when back-prop arrived, 1986.
• Become an example of overfitting model.
• 1992 - SVMs Regain of interest.

6
Negative prejudices

• 3) Against kernel methods
• 1964 - Aizerman et al Potential functions.
• 1967 - Cover-Hart Nearest neighbors.
• 1982 - Hopfield nets, distributed associative
  memories. Parody of the Grand-Mother cell
  methods.
• 1992 - SVMs Regain of interest.

7
Negative prejudices

• 4) Against biased estimators
• 1922 - Fisher promotion of the use of unbiased
  estimators in statistics.
• 1971 - Vapnik-Chervonenkis theory biased
  estimators are good to get better
  generalization.
• (this example is a courtesy of V. Vapnik)

8
Other prejudices

• Against greedy search.
• For multivariate feature selection.
• For small VC dimension.
• For sparse solutions.
• For introducing domain knowledge.
• Your negative result here.

9
Some of my negative results

• SVM clustering does not work in high dimensions.
• Feature selection with correlation methods often
  works better than multivariate methods.
• Dumb linear, poly2 and Gaussian kernels are hard
  to beat.
• Hard to say of MSE and SVM which is best to train
  kernel classifiers.