Adaptive Offset Subspace Self- Organizing Map with an Application to Handwritten Digit Recognition Huicheng Zheng, Pádraig Cunningham and Alexey Tsymbal August 20 th, 2006 Thanks to the MUSCLE Internal Fellowship ( MDM/KDD’06

2 Outline Introduction Learning self-organized subspaces –Adaptive Subspace Self-Organizing Map (ASSOM) Learning self-organized linear manifolds –Adaptive Offset Subspace Self-Organizing Map (AOSSOM) Experiments –Cluster separating –Handwritten digit recognition Conclusions

3 Fundamental Requirements in MDM Among others: Dimension reduction Visualization The Self-Organizing Map (SOM) meets these requirements and finds its use in MDM [J.-C. Lamirel et al., 2000] [R. Honda et al., 2000]

4 Learning Self-Organized Subspaces Pattern invariance classes pattern subspaces –For example: Translation-invariant learning Combining subspace learning and SOM [Kohonen. T., et al., 1997]: Adaptive Subspace Self- Organizing Map (ASSOM) rectangles circles triangles ……

5 Applications of the ASSOM to MDM Invariant-feature learning [Kohonen, T., et al., 1997] Speech processing [Hase, H., et al., 1996] Texture segmentation [Ruiz del Solar, J., 1998] Image retrieval [De Ridder, D., et al., 2000] Image classification [Zhang, B., et al., 1999]

6 The ASSOM does not learn linear manifolds adequately. Faces viewed from different perspectives, variants of handwritten digits can be handled by linear manifolds [Liu, Z. Q., 2001] [Hinton, G. E., et al., 1997]. Learning Linear Manifolds

7 Learning Self-Organized Linear Manifolds Principal Component Analysis Self-Organizing Map (PCASOM) [López-Rubio, E., et al., 2004] –Combining PCA with SOM –Computationally expensive Adaptive Manifold Self-Organizing Map (AMSOM) [Liu, Z. Q., 2001] –Attaching mean vectors to the ASSOM –No given objective function Adaptive Offset Subspace Self-Organizing Map (AOSSOM) [Proposed] –Attaching offset vectors to the ASSOM –Well defined objective function

8 Outline Introduction Learning self-organized subspaces –Adaptive Subspace Self-Organizing Map Learning self-organized linear manifolds –Adaptive Offset Subspace Self-Organizing Map Experiments –Cluster separating –Handwritten digit recognition Conclusions

9 Adaptive Subspace Self-Organizing Map The module arrays in ASSOM Rectangular topology Hexagonal topology A module representing the subspace L (j) j

10 Competition and Adaptation Repeatedly: –Competition: Winning module of –Adaptation: For the winner and the modules i in its neighborhood –Orthonormalize the basis vectors

11 Outline Introduction Learning self-organized subspaces –Adaptive Subspace Self-Organizing Map Learning self-organized linear manifolds –Adaptive Offset Subspace Self-Organizing Map Experiments –Cluster separating –Handwritten digit recognition Conclusions

12 Adaptive Offset Subspace Self-Organizing Map (AOSSOM): Linear Manifold Learning defines a set of linear manifold { A i } Learning of can be done by defining a criterion function: – defines an MSE criterion. – defines a spatial interaction between neighboring sites in the AOSSOM. Using stochastic gradient descent:

13 Outline Introduction Learning self-organized subspaces –Adaptive Subspace Self-Organizing Map Learning self-organized linear manifolds –Adaptive Offset Subspace Self-Organizing Map Experiments –Cluster separating –Handwritten digit recognition Conclusions

14 Cluster Separating by Learning Linear Manifolds (1) Nearly optimal parameters were used in all the experiments.

15 Cluster Separating by Learning Linear Manifolds (2) Instability was observed for the AMSOM when initial learning rate. A larger value has been used to alleviate the problem.

16 Four random clusters with 2-D Gaussian distributions. “AMSOM 1” uses the neighborhood-decreasing scheme proposed in [Liu, Z. Q., 2001]: A Gaussian function with the FWHM “AMSOM 2” uses the same neighborhood function as the AOSSOM: A Gaussian function with the FWHM Cluster Separating by Learning Linear Manifolds (3)

17 Cluster Separating by Learning Linear Manifolds (3) Boldface texts emphasize the cases where the AOSSOM worked the best (8 out of 10 times). Observation of projection error decreasing for D=1.

18 Outline Introduction Learning self-organized subspaces –Adaptive Subspace Self-Organizing Map Learning self-organized linear manifolds –Adaptive Offset Subspace Self-Organizing Map Experiments –Cluster separating –Handwritten digit recognition Conclusions

19 Application of the AOSSOM to Handwritten Digit Recognition Large variety of handwriting styles It is hard for hand-crafted feature extractor to cope with these variations. Variations create embedded manifolds in the feature space. Piece-wise linear approximation of embedded manifolds.

20 Handwritten Digit Image Database MNIST database composed of 60,000 training digits and 10,000 test digits [LeCun, Y., et al., 1998]. Some examples:

21 MNIST Learning with the AOSSOM One AOSSOM trained for each digit class For an input test digit image x, each AOSSOM M k reconstructs an output.

22 MNIST Learning Performance Summary H: Linear manifold dimension W: No. of rows and columns of AOSSOM

23 Outline Introduction Learning self-organized subspaces –Adaptive Subspace Self-Organizing Map Learning self-organized linear manifolds –Adaptive Offset Subspace Self-Organizing Map Experiments –Cluster separating –Handwritten digit recognition Conclusions

24 Conclusions The AOSSOM permits to learn organized linear manifolds with a well-defined criterion function. Higher robustness is observed in the experiments with the AOSSOM than with the AMSOM. The AOSSOM is well suited for MDM tasks where data structures are best handled by piece- wise linear manifolds.

25 References De Ridder, D., et al., 2000: The adaptive subspace map for image description and image database retrieval. SSPR&SPR. Hase, H., et al., 1996: Speech signal processing using Adaptive Subspace SOM (ASSOM). Technical Report NC95-140, The Inst. of Electronics, Information and Communication Engineers, Tottori University, Koyama, Japan. Hinton, G. E., et al., 1997: Modeling the manifolds of images of handwritten digits. IEEE Trans. Neural Networks, 8(1):65–74. Honda, R., et al., 2000: Semantic indexing and temporal rule discovery for time-series satellite images. In S. J. Simoff and O. R. Zaïane, editors, Proc. First Int. Workshop on Multimedia Data Mining, in conjunction with Sixth ACM SIGKDD Int. Conf. on Knowledge Discovery and Data Mining, pp. 82–90, Boston, MA, USA. Kohonen, T., et al., 1997: Self-Organized formation of various invariant-feature filters in the adaptive-subspace SOM. Neural Computation 9(6).

26 References Lamirel, J.-C., et al., 2000: A Self Organizing Map (SOM) extended model for information discovery in a digital library context. In S. J. Simoff and O. R. Zaïane, editors, Proc. First Int. Workshop on Multimedia Data Mining, in conjunction with Sixth ACM SIGKDD Int. Conf. on Knowledge Discovery and Data Mining, pp. 60–66, Boston, MA, USA. LeCun, Y., et al., 1998: Gradient-based learning applied to document recognition. Proc. IEEE, 86(11):2278–2324. Liu, Z. Q., 2001: Retrieving faces using Adaptive Subspace Self- Organising Map. In Proc. International Symposium on Intelligent Multimedia, Video and Speech Processing, pp. 377–380, Hong Kong. López-Rubio, E., et al., 2004: A Principal Components Analysis Self- Organizing Map. Neural Networks, 17(2), pp. 261–270. Ruiz del Solar, J., 1998: Texsom: texture segmentation using Self- Organizing Maps. Neurocomputing 21(1–3). Zhang, B., et al., 1999: Handwritten digit recognition by adaptive- subspace self-organizing map (ASSOM). IEEE Trans. on Neural Networks 10(4).

27 Thanks and questions?