1. Seungchan Lee Intelligent Electronic Systems Human and Systems Engineering Department of Electrical and Computer Engineering Software Release and Support Vector Machine Research Presentation:

2. ISIP: Research Presentation Page 1 of 24 Overview Software Release  Isip_lm_tester  Isip_network_builder  Debugging utility : Purify Verification System  Isip_verify  Support Vector Machine Audio File Generation Next Plans

3. ISIP: Research Presentation Page 2 of 24 Isip_lm_tester, Isip_network_builder Dummy Symbol generation problem  Sentence generation terminated when met with dummy symbol at the highest level.  Dummy Symbol should not show at the output sentence.  Include Dummy Symbol check routine Exclude Symbol generation problem  When turn on exclude symbol flag, lm_tester should not generate exclude symbol.  It need to modify HierarchicalSearch class.  Isip_network_buider  Add save option for ABNF, BNF  Need to correct dummy symbol generation problem  When generating dummy symbol without any subgraph, it generates error message.

4. ISIP: Research Presentation Page 3 of 24 Debugger Utility Purify  What is problem?  Compilation error : When instrumenting purify, it generates error message.  It is not easy to figure out the reason because we have so many linking process when compiling.  How to resolve?  Simple program without IFC class works fine  Narrow down which classes are problem.  Exclude all linking process, and then add one class repeatedly.  Solution :  After track down the linking process, I can figure out the problem is originated from sphere utility.  How can correct it?  I’m currently doing this.

5. ISIP: Research Presentation Page 4 of 24 Isip_verify When doing HMM training, it generates segmentation fault.  This happens at the end of the program related to HierarchicalDigraph object.  Recently, we have many changes in IFC classes, but this problem might exists sometimes ago. When doing SVM training, it generates checksum error.  This error did not happen before I was recompiling whole repository.  isip_verify utilty also need to be throughly investigated using purify utility

6. ISIP: Research Presentation Page 5 of 24 Audio File Generation Load one or two SWB CDs Select 100 conversations For each conversation, strip the NIST header Grab every other byte starting with the first byte (first channel) and put that into a raw audio file; PRESERVE the 8-bit ulaw data (do not use or convert to 16-bit) Convert this file to Sun ".au" using Sox

7. ISIP: Research Presentation Page 6 of 24 What to Learn? Audio File format .au file format  Widely used in UNIX machine and originated by SUN.  Header + Variable length information + audio data  Support various encoding types  NIST SPHERE file  Raw format  PERL Programming Language  It is simple programming language which performs extracting and printing out information from a text file.  Interpreted Language ( not compiled)  Conversion Utilities  w_decode  Sox  “od” command

8. ISIP: Research Presentation Page 7 of 24 Why Support Vector Machine ? This is new learning technology to be noticed recently. Even though it has been situated as a subfield of machine learning, it still have many issues about theory and algorithm. To be more familiar with verification system, it is required to review one field for the next step.

9. ISIP: Research Presentation Page 8 of 24 How it works? Suppose we have low dimensional feature space. It is consist of positive examples and negative examples How about the following case? How can we classify this?

10. ISIP: Research Presentation Page 9 of 24 How it works? Simple idea : Low dimensional feature space map into high dimensional feature space using kernel function.

11. ISIP: Research Presentation Page 10 of 24 How can we determine maximum margin? To explain this, we need to know the following concepts.  Margin concepts  Lagrange multiplier  Primal and dual representation  Karush-Kuhn-Tucker Conditions (KKT)  Risk Bounds and Minimization Maximal Margin classifier

12. ISIP: Research Presentation Page 11 of 24 Hyperplane Linear classification Input space X is split into two parts by the hyperplane defined by the equation Objective Function

13. ISIP: Research Presentation Page 12 of 24 Margin Geometric margin of two pointsThe margin of training set

14. ISIP: Research Presentation Page 13 of 24 Maximal Margin Classifier The Simplest model, but works only for data which are linearly separable in the feature space.  easy to understand and main building block for more complex SVMs Margin w H1 H2 Plus-plane = Minus-plane = Separating hyperplane = Classify as.. +1 if -1 if

15. ISIP: Research Presentation Page 14 of 24 Maximal Margin Classifier Margin w Computing the margin width

16. ISIP: Research Presentation Page 15 of 24 Maximal Margin Classifier Margin w Computing the margin width

17. ISIP: Research Presentation Page 16 of 24 Maximal Margin Classifier Margin w How to transform this optimization problem into dual problem? Hypothesis can be described as a linear combination of the training points. Lagrange

18. ISIP: Research Presentation Page 17 of 24 Maximal Margin Classifier Margin w How to transform this optimization problem into dual problem?

19. ISIP: Research Presentation Page 18 of 24 Maximal Margin Classifier Margin w How to transform this optimization problem into dual problem?

20. ISIP: Research Presentation Page 19 of 24 Maximal Margin Classifier Margin w How to transform this optimization problem into dual problem? Only these points are involved for the weight vector.

21. ISIP: Research Presentation Page 20 of 24 Maximal Margin Classifier Margin w How to transform this optimization problem into dual problem?

22. ISIP: Research Presentation Page 21 of 24 Maximal Margin Classifier Margin w How to transform this optimization problem into dual problem?

23. ISIP: Research Presentation Page 22 of 24 Review Maximal Margin SVMs Can be slow in practice Dose not control the number of support vector (Sparseness) Only one degree of freedom is the choice of kernel  model selection Cannot be used non linear separable feature space  many real world problems deal with nonlinear, noisy data. However, it is a starting point for the more sophisticated SVMs.

24. ISIP: Research Presentation Page 23 of 24 Next Plan Software Release  Resolve purify compilation problem  Examine memory leak problem using purify utility  Track down remaining bugs  Test several cases Verification System  Do NIST 2003 Experiment using new isip_verify  Implemenation techniques of support vector machine  Algorithm comparison between several SVM softwares  Resolve memory leak problems

25. ISIP: Research Presentation Page 24 of 24 Reference An introduction to Support Vector Machines and other kernel-based learning methods by “Nello Cristianini and John Shawe-Taylor”, 2000, Cambridge Press Support Vector Machines Tutorial Slides by Andrew W. Moore http://www.autonlab.org/tutorials/svm15.pdf Practical Perl Programming http://www.cs.cf.ac.uk/Dave/PERL/