1. Intelligent Database Systems Lab 國立雲林科技大學 National Yunlin University of Science and Technology 1 On-line Learning of Sequence Data Based on Self-Organizing Incremental Neural Network Shogo Okada, Osamu Hasegawa IJCNN, 2008 Presented by Hung-Yi Cai 2010/10/27

2. Intelligent Database Systems Lab N.Y.U.S.T. I. M. 2 Outlines  Motivation  Objectives  Methodology  Experiments  Conclusions  Comments

3. Intelligent Database Systems Lab N.Y.U.S.T. I. M. 3 Motivation  Numerous on-line, continuously learning algorithms based on neural networks have been proposed but continuously learning algorithms for non-sequential data and for static data.  Advantages of online learning and continuously learning systems ─ adapt to new data and improve recognition performance continuously ─ not require collection of large amounts of prior training data for batch learning

4. Intelligent Database Systems Lab N.Y.U.S.T. I. M. 4 Objectives  To propose the online, continuously learning method for sequence data based on the SOINN- DTW (Self-Organizing Incrementally Neural Network-Dynamic Time Warping) method.  To use SOINN’s on-line learning function and enhance SOINN-DTW to produce this online, continuously learning method.

5. Intelligent Database Systems Lab N.Y.U.S.T. I. M. Methodology  In the SOINN-DTW model, the output distribution of state is approximated using a SOINN, which can grow incrementally and accommodate input data.  The output distribution is represented in a self- organizing manner; the number of states is set in the learning process. 5

6. Intelligent Database Systems Lab N.Y.U.S.T. I. M. Methodology 6 The framework of SOINN-DTW SOINN On-line Learning Dynamic Time Warping Parameter estimation Recognition

7. Intelligent Database Systems Lab N.Y.U.S.T. I. M. The detail of SOINN 7 First LayerSecond Layer

8. Intelligent Database Systems Lab N.Y.U.S.T. I. M. The output of SOINN 8

9. Intelligent Database Systems Lab N.Y.U.S.T. I. M. Methodology 9 The framework of SOINN-DTW SOINN On-line Learning Dynamic Time Warping Parameter estimation Recognition

10. Intelligent Database Systems Lab N.Y.U.S.T. I. M. The process of On-line Learning 10

11. Intelligent Database Systems Lab N.Y.U.S.T. I. M. The process of On-line Learning 11

12. Intelligent Database Systems Lab N.Y.U.S.T. I. M. Experiments  Experimental Setting ─ We used two kinds of HMM for comparison with on- line SOINN-DTW method. Trained in MAP adaptation using on-line training data. Used all training data. ─ The parameters used for feature extraction and the feature vector are: 12

13. Intelligent Database Systems Lab N.Y.U.S.T. I. M. Experiments  Experimental Procedure a) We used 30 data per class (30 data 48 class 1440 data) as initial (batch) training data. Recognition accuracy was the baseline result. b) On-line learning was done and the model was updated using 500 data from all classes. c) We tested the test dataset. d) We repeated procedure b) c) until the recognition accuracy converged. 13

14. Intelligent Database Systems Lab N.Y.U.S.T. I. M. Experiments  Parameter setting ─ Through preliminary experimentation using development datasets, we set parameters for SOINN and parameters for SOINN-DTW. ƛ = 10000, α d = 10000. L = 8, L* = 5, α = 0.25. 14

15. Intelligent Database Systems Lab N.Y.U.S.T. I. M. Experiments  Experimental Results ─ The recognition accuracy of baseline and the recognition accuracy after on-line learning when the on-line training data are 3000 15

16. Intelligent Database Systems Lab N.Y.U.S.T. I. M. Experiments  Experimental Results 16

17. Intelligent Database Systems Lab N.Y.U.S.T. I. M. 17 Conclusions  Enhancing SOINN-DTW method to make it capable of on-line learning of sequence data.  Performance results from a TIMIT phoneme recognition task showed that, using on-line SOINN-DTW, through continuous on-line learning with new (on-line) data based on scarce batch training data, the classification accuracy is improved step by step.

18. Intelligent Database Systems Lab N.Y.U.S.T. I. M. 18 Comments  Advantages ─ The algorithm presents an on-line, continuously learning mechanism for sequence data.  Applications ─ Incremental Neural Network