Transductive Reliability Estimation for Kernel Based Classifiers

1
Transductive Reliability Estimation for Kernel
Based Classifiers
Dimitris Tzikas1, Matjaz Kukar2, Aristidis Likas1
tzikas_at_cs.uoi.gr, matjaz.kukar_at_fri.uni-lj.si,
arly_at_cs.uoi.gr

• 1Department of Computer Science, University of
  Ioannina, Greece
• 2Faculty of Computer and Information Science,
  University of Ljubljana, Slovenia

2
Introduction

• We wish to assess the reliability of single
  example classifications of kernel-based
  classifiers
• Support Vector Machine (SVM)
• Relevance Vector Machine (RVM)
• Such assessment is useful in risk-sensitive
  applications
• Weighted combination of several classifiers
• Reliability measures can be obtained directly
  from the classifier outputs
• We propose the use of the transduction
  reliability methodology to kernel-based
  classifiers

3
Kernel Classifiers

• Mapping function to the feature space
• Kernel function inner product in the feature
  space
• Kernel Classifier
• Training estimate w using training set D
• Prefer sparse solutions most wn?0
• SVM and RVM differ in the training method.

4
Support Vector Machine (SVM)

• SVM model (two-class)
• Maximize margin from the separating hyperplane in
  feature space
• subject to
• C is a hyperparameter to be prespecified

5
Reliability Measure for SVM

• The points near the decision boundary have lower
  reliability.
• Output ysvm(x) distance from the separating
  hyperplane (decision boundary).
• Transform the outputs to probabilities by
  applying the sigmoid function
• Define reliability measure

6
Relevance Vector Machine

• RVM model (two-class)
• Provides posterior probability for class C1
• RVM is a Bayesian linear model with hierarchical
  prior on weights w
• The hierarchical prior enforces sparse solutions

7
Relevance Vector Machine

• Compute by maximizing likelihood
• Many
• Compute w
• Incremental RVM
• Start from an empty model and a set of basis
  functions
• Incrementally add (and delete) terms
• Convenient for the transduction approach which
  requires retraining

8
RVM Reliability Measure

• Compute reliability estimate for the decision of
  input x as

9
Transductive Reliability Estimation (Kukar and
Kononenko, ECML 2002)

• The transductive methodology estimates
  reliability of individual classifications.
• Measures stability of the classifier after
• small perturbation to the training set (the test
  example with the class label is added to the
  training set)
• retraining of the classifier
• Assumption For reliable decisions, this process
  should not lead to significant model changes.
• The method can be applied to any classifier that
  outputs
• class posterior probabilities
• Transduction requires retraining ? incremental
  training methods are preferable

10
Transductive Reliability Estimation

• Assume a classifier CL1 and a training set
• Compute class posteriors pk and classify a test
  example.
• Objective Estimate reliability of decision
• Transductive step
• Add previous test example with the classification
  label to training set
• Train a classifier CL2
• Compute class posteriors qk and classify the test
  example.

11
Transductive Reliablility Estimation

• Difference between the class posterior vectors p
  and q of CL1 and CL2 is an estimate of
  reliability.
• Symmetric KL divergence
• Scale reliability values to 0, 1
• Reliable estimations
• How do we select threshold T?

12
Selecting the Threshold

• Use Leave-one-out to obtain classifications and
  reliability estimations TRE(x) for each example x
• For a threshold T
• We wish D1 to contain incorrectly classified
  examples
• D2 to contain correctly classified
  examples
• Select T that maximizes Information Gain
• check

13
Evaluation of reliability measures

• Transduction has been evaluated on several
  classifiers
• decision trees, Naïve Bayes
• We applied the transduction approach to SVM and
  RVM
• SVM is retrained from scratch with same
  hyperparameters
• For RVM we considered both retraining from
  scratch and incremental retraining
• Reliability measures ?RESVM, TRERVM and
  TRERVM(inc).
• TRERVM(inc).is computationally efficient (50
  100 times faster)
• We compare direct measures RESVM, RERVM with
  transductive measures.

14
Evaluation of reliability measures

• 3 UCI medical datasets (RBF kernel)
• 1 bioinformatics (linear kernel) dataset
  (leukemia)
• Cardiac Artery Disease (CAD) dataset (RBF kernel)
• Comparison with expert physicians
• Evaluation of reliability estimation methods
• Use Leave-one-out to decide for correct or
  incorrect classification of each example and
  compute the reliability estimates (RE(x),
  TRE(x)).
• For each dataset and measure determine the
  threshold that maximizes the information gain
• Use the maximum information gain to compare
  different reliability measures on each dataset

15
Evaluation on UCI Datasets

• Max IG of TRESVM is higher than RESVM
• Max IG of TRERVM(inc) is higher than TRERVM and
  RERVM (except hepatitis dataset)

16
Application on CAD (comparison to physicians)

• Coronary Artery Disease (CAD) dataset (University
  Clinical Centre, Ljubljana).
• 327 cases (228 positive, 99 negative)
• Physicians estimate reliability by computing a
  posterior probability based on diagnostic tests
  and other information.
• For posterior gt 0.9 or lt 0.1 diagnosis is assumed
  reliable.

17
Application on CAD
18
Conclusions

• We applied the transductive approach to
  kernel-based models
• Support Vector Machine (SVM)
• Relevance Vector Machine (RVM)
• We compared direct and transductive reliability
  measures on several datasets
• We also compared against physicians performance
  on a real dataset for Diagnosis of Coronary
  Artery Disease (CAD)
• The transductive approach seems to provide good
  estimates

19
Future work

• Examine incremental training methods for SVM.
• Define reliability measures based on the
  structural difference between the classifiers CL1
  and CL2.
• Use transduction to estimate strangeness of an
  example in the typicalness framework for
  confidence estimation (Kukar, KIS 2006)