Heat Diffusion Classifier on a Graph

1
Heat Diffusion Classifier on a Graph

• Haixuan Yang, Irwin King, Michael R. Lyu
• The Chinese University of Hong Kong
• Group Meeting
• 2006

2
Outline

• Introduction
• Heat Diffusion Model on a Graph
• Three Graph Inputs
• Connections with Other Models
• Experiments
• Conclusions and Future Work

3
Introduction

• Kondor Lafferty (NIPS2002)
• Construct a diffusion kernel on a graph
• Apply to a large margin classifier
• Lafferty Kondor (JMLR2005)
• Construct a diffusion kernel on a special
  manifold
• Apply to SVM
• Belkin Niyogi (Neural Computation 2003)
• Reduce dimension by heat kernel and local
  distance
• Tenenbaum et al (Science 2000)
• Reduce dimension by local distance

4
Introduction

• The ideas we inherit
• Local information
• relatively accurate in a nonlinear manifold.
• Heat diffusion on a manifold
• a generalization of the Gaussian density from
  Euclidean space to manifold.
• heat diffuses in the same way as Gaussian density
  in the ideal case when the manifold is the
  Euclidean space.
• The ideas we think differently
• Establish the heat diffusion equation directly on
  a graph
• three proposed candidate graphs.
• Construct a classifier by the solution directly.

5
Heat Diffusion Model on a Graph

• Notations

6
Heat Diffusion Model on a Graph

• Assumptions

7
Heat Diffusion Model on a Graph

• Solution

8
Heat Diffusion Model on a Graph

• Three candidate graphs
• KNN Graph
• Connect points j and i from j to i if j is one of
  the K nearest neighbors of i, measured by the
  Euclidean distance.
• SKNN-Graph
• Choose the smallest Kn/2 undirected edges, which
  amounts to Kn directed edges.
• Minimum Spanning Tree
• Choose the subgraph such that
• It is a tree connecting all vertices the sum of
  weights is minimum among all such trees.

9
Heat Diffusion Model on a Graph

• Illustration
• Manifold
• KNN Graph
• SKNN-Graph
• Minimum Spanning Tree

10
Heat Diffusion Model on a Graph

• Advantages and disadvantages
• KNN Graph
• Democratic to each node
• Resulting classifier is a generalization of KNN
• May not be connected
• Long edges may exit while short edges are removed
• SKNN-Graph
• Not democratic
• May not be connected
• Short edges are more important than long edges
• Minimum Spanning Tree
• Not democratic
• Long edges may exit while short edges are removed
• Connection is guaranteed
• Less parameter
• Faster in training and testing

11
Heat Diffusion Classifier (HDC)

• Choose a graph
• Compute the heat kernel
• Compute the heat distribution for each class
  according to the initial heat distribution
• Classify according to the heat distribution

12
Connections with other models

• The Parzen window approach (when the window
  function takes the normal form) is a special case
  of the HDC for the KNN and SKNN graphs (when?is
  small, Kn-1).
• KNN is a special case of the HDC for the KNN
  graph (when?is small, 1/ß0).
• In Euclidean space, the proposed heat diffusion
  model for the KNN graph (when K is set to be 2m,
  1/ß0) is a generalization of the solution
  deduced by Finite Difference Method.
• Hopefield Model (PNAS, 1982) is the original one
  which determines class by looking at immediate
  neighbors. (Thanks to the anonymous reviewer)

13
Experiments

• Experimental Setup
• Experimental Environments
• Hardware Nix Dual Intel Xeon 2.2GHz
• OS Linux Kernel 2.4.18-27smp (RedHat 7.3)
• Developing tool C
• Data Description
• 3 artificial Data sets and 6 datasets from UCI
• Comparison
• Algorithms
• Parzen windowKNNSVM KNN-HSKNN-HMST-H
• Results average of the ten-fold cross validation

14
Experiments

• Results

15
Conclusions and Future Work

• KNN-H, SKNN-H and MST-H
• Candidates for the Heat Diffusion Classifier on a
  Graph.
• Future Work
• Apply the asymmetric exp?H to SVM.
• Extend the current heat diffusion model further
  (from inside).
• DiffusionRank is a generalization of PageRank