1 Heat Diffusion Classifier on a Graph Haixuan Yang, Irwin King, Michael R. Lyu The Chinese University of Hong Kong Group Meeting 2006

2 Introduction Heat Diffusion Model on a Graph Three Graph Inputs Connections with Other Models Experiments Conclusions and Future Work Outline

3 Introduction Kondor & Lafferty (NIPS2002) Construct a diffusion kernel on a graph Apply to a large margin classifier Lafferty & Kondor (JMLR2005) Construct a diffusion kernel on a special manifold Apply to SVM Belkin & Niyogi (Neural Computation 2003) Reduce dimension by heat kernel and local distance Tenenbaum et al (Science 2000) Reduce dimension by local distance

4 Introduction  The ideas we inherit Local information  relatively accurate in a nonlinear manifold. Heat diffusion on a manifold  a generalization of the Gaussian density from Euclidean space to manifold.  heat diffuses in the same way as Gaussian density in the ideal case when the manifold is the Euclidean space.  The ideas we think differently Establish the heat diffusion equation directly on a graph  three proposed candidate graphs. Construct a classifier by the solution directly.

5 Heat Diffusion Model on a Graph  Notations

6 Heat Diffusion Model on a Graph  Assumptions

7 Heat Diffusion Model on a Graph  Solution

8 Heat Diffusion Model on a Graph  Three candidate graphs KNN Graph  Connect points j and i from j to i if j is one of the K nearest neighbors of i, measured by the Euclidean distance. SKNN-Graph  Choose the smallest K*n/2 undirected edges, which amounts to K*n directed edges. Minimum Spanning Tree  Choose the subgraph such that It is a tree connecting all vertices; the sum of weights is minimum among all such trees.

9 Heat Diffusion Model on a Graph  Illustration Manifold KNN Graph SKNN-Graph Minimum Spanning Tree

10 Heat Diffusion Model on a Graph  Advantages and disadvantages KNN Graph  Democratic to each node  Resulting classifier is a generalization of KNN  May not be connected  Long edges may exit while short edges are removed SKNN-Graph  Not democratic  May not be connected  Short edges are more important than long edges Minimum Spanning Tree  Not democratic  Long edges may exit while short edges are removed  Connection is guaranteed  Less parameter  Faster in training and testing

11 Heat Diffusion Classifier (HDC)  Choose a graph  Compute the heat kernel  Compute the heat distribution for each class according to the initial heat distribution  Classify according to the heat distribution

12 Connections with other models  The Parzen window approach (when the window function takes the normal form) is a special case of the HDC for the KNN and SKNN graphs (whenγis small, K=n-1).  KNN is a special case of the HDC for the KNN graph (whenγis small, 1/β=0).  In Euclidean space, the proposed heat diffusion model for the KNN graph (when K is set to be 2m, 1/β=0) is a generalization of the solution deduced by Finite Difference Method.  Hopefield Model (PNAS, 1982) is the original one which determines class by looking at immediate neighbors. (Thanks to the anonymous reviewer)

13 Experiments  Experimental Setup Experimental Environments  Hardware: Nix Dual Intel Xeon 2.2GHz  OS: Linux Kernel smp (RedHat 7.3)  Developing tool: C  Data Description 3 artificial Data sets and 6 datasets from UCI  Comparison Algorithms:  Parzen window KNN SVM KNN-H SKNN-H MST-H Results: average of the ten-fold cross validation Dataset Case s ClassesVariable Syn Syn Syn Breast-w68329 Glass21469 Iono Iris15034 Sonar Vehicle846418

14 Experiments  Results Dataset SVM KNN PWAKNN-HMST-HSKNN-H Syn Syn Syn Breast-w Glass Iono Iris Sonar Vehicle

15 Conclusions and Future Work  KNN-H, SKNN-H and MST-H Candidates for the Heat Diffusion Classifier on a Graph.  Future Work Apply the asymmetric exp{γH} to SVM. Extend the current heat diffusion model further (from inside). DiffusionRank is a generalization of PageRank