1. 1 Heat Diffusion Classifier on a Graph Haixuan Yang, Irwin King, Michael R. Lyu The Chinese University of Hong Kong Group Meeting 2006

2. 2 Introduction Heat Diffusion Model on a Graph Three Graph Inputs Connections with Other Models Experiments Conclusions and Future Work Outline

3. 3 Introduction Kondor & Lafferty (NIPS2002) Construct a diffusion kernel on a graph Apply to a large margin classifier Lafferty & Kondor (JMLR2005) Construct a diffusion kernel on a special manifold Apply to SVM Belkin & Niyogi (Neural Computation 2003) Reduce dimension by heat kernel and local distance Tenenbaum et al (Science 2000) Reduce dimension by local distance

4. 4 Introduction  The ideas we inherit Local information  relatively accurate in a nonlinear manifold. Heat diffusion on a manifold  a generalization of the Gaussian density from Euclidean space to manifold.  heat diffuses in the same way as Gaussian density in the ideal case when the manifold is the Euclidean space.  The ideas we think differently Establish the heat diffusion equation directly on a graph  three proposed candidate graphs. Construct a classifier by the solution directly.

5. 5 Heat Diffusion Model on a Graph  Notations

6. 6 Heat Diffusion Model on a Graph  Assumptions

7. 7 Heat Diffusion Model on a Graph  Solution

8. 8 Heat Diffusion Model on a Graph  Three candidate graphs KNN Graph  Connect points j and i from j to i if j is one of the K nearest neighbors of i, measured by the Euclidean distance. SKNN-Graph  Choose the smallest K*n/2 undirected edges, which amounts to K*n directed edges. Minimum Spanning Tree  Choose the subgraph such that It is a tree connecting all vertices; the sum of weights is minimum among all such trees.

9. 9 Heat Diffusion Model on a Graph  Illustration Manifold KNN Graph SKNN-Graph Minimum Spanning Tree

10. 10 Heat Diffusion Model on a Graph  Advantages and disadvantages KNN Graph  Democratic to each node  Resulting classifier is a generalization of KNN  May not be connected  Long edges may exit while short edges are removed SKNN-Graph  Not democratic  May not be connected  Short edges are more important than long edges Minimum Spanning Tree  Not democratic  Long edges may exit while short edges are removed  Connection is guaranteed  Less parameter  Faster in training and testing

11. 11 Heat Diffusion Classifier (HDC)  Choose a graph  Compute the heat kernel  Compute the heat distribution for each class according to the initial heat distribution  Classify according to the heat distribution

12. 12 Connections with other models  The Parzen window approach (when the window function takes the normal form) is a special case of the HDC for the KNN and SKNN graphs (whenγis small, K=n-1).  KNN is a special case of the HDC for the KNN graph (whenγis small, 1/β=0).  In Euclidean space, the proposed heat diffusion model for the KNN graph (when K is set to be 2m, 1/β=0) is a generalization of the solution deduced by Finite Difference Method.  Hopefield Model (PNAS, 1982) is the original one which determines class by looking at immediate neighbors. (Thanks to the anonymous reviewer)

13. 13 Experiments  Experimental Setup Experimental Environments  Hardware: Nix Dual Intel Xeon 2.2GHz  OS: Linux Kernel 2.4.18- 27smp (RedHat 7.3)  Developing tool: C  Data Description 3 artificial Data sets and 6 datasets from UCI  Comparison Algorithms:  Parzen window KNN SVM KNN-H SKNN-H MST-H Results: average of the ten-fold cross validation Dataset Case s ClassesVariable Syn-110022 Syn-210023 Syn-320023 Breast-w68329 Glass21469 Iono351234 Iris15034 Sonar208260 Vehicle846418

14. 14 Experiments  Results Dataset SVM KNN PWAKNN-HMST-HSKNN-H Syn-166.067.080.093.095.0 Syn-234.067.083.094.0 89.0 Syn-354.079.592.091.090.092.0 Breast-w96.894.196.696.995.999.4 Glass68.161.263.568.168.770.5 Iono93.783.289.296.3 Iris9697.395.398.092.094.7 Sonar88.580.353.990.991.894.7 Vehicle84.863.066.065.583.566.6

15. 15 Conclusions and Future Work  KNN-H, SKNN-H and MST-H Candidates for the Heat Diffusion Classifier on a Graph.  Future Work Apply the asymmetric exp{γH} to SVM. Extend the current heat diffusion model further (from inside). DiffusionRank is a generalization of PageRank