1. Project 1: Classification Using Neural Networks 2009. 03. 23 Kim, Kwonill kikim@bi.snu.ac.kr kikim@bi.snu.ac.kr Biointelligence laboratory Artificial Intelligence

2. Contents Project outline Description on the data set Description on tools for ANN Guide to Writing Reports  Style  Mandatory contents  Optional contents Submission guide / Marking scheme Demo 2 (C) 2008, SNU Biointelligence Laboratory

3. 3 Outline Goal  Understand MLP & machine learning deeper  Practice researching and technical writing Handwritten digits problem (classification)  To predict the class labels (digits) of handwritten digit data set  Using Multi Layer Perceptron (MLP)  Estimating several statistics on the dataset Data set  Variation of the ‘Handwritten digit data set’  http://archive.ics.uci.edu/ml/datasets/Pen- Based+Recognition+of+Handwritten+Digits http://archive.ics.uci.edu/ml/datasets/Pen- Based+Recognition+of+Handwritten+Digits

4. Handwritten Digit Data Set (1/2) Original Data Set Description  Digit database of 11,000 samples from every 44 writers  http://archive.ics.uci.edu/ml/datasets/Pen- Based+Recognition+of+Handwritten+Digits http://archive.ics.uci.edu/ml/datasets/Pen- Based+Recognition+of+Handwritten+Digits  16 attributes  (x t, y t ), t = 1, …, 8  0 ~ 100  Label (Class)  0, 1, 2, …, 9 4 (C) 2008, SNU Biointelligence Laboratory

5. Handwritten Digit Data Set (2/2) Constitution  Preprocessed data (*.arff, *.csv)  Total data(pendigits_total_set, 1099) = training data(pendigits_training, 749) + test data(pendigits_test, 350)  Data description(pendigits.names)  For WEKA(*.arff) 5 (C) 2008, SNU Biointelligence Laboratory

6. 6 Tools for Experiments with ANN Source codes - Choose anything!!  Free software  Weka (recommended)  MATLAB tool box (Toolboxes  Neural Network)  ANN libraries (C, C++, JAVA, …) Web sites  http://www.cs.waikato.ac.nz/~ml/weka/ http://www.cs.waikato.ac.nz/~ml/weka/  http://www.faqs.org/faqs/ai-faq/neural-nets/part5/ http://www.faqs.org/faqs/ai-faq/neural-nets/part5/

7. Reports Style English only!! Scientific journal-style  How to Write A Paper in Scientific Journal Style and Format  http://abacus.bates.edu/~ganderso/biology/resources/writing/HTWsections.html http://abacus.bates.edu/~ganderso/biology/resources/writing/HTWsections.html 7 (C) 2008, SNU Biointelligence Laboratory Experimental process Section of Paper What did I do in a nutshell? Abstract What is the problem?Introduction How did I solve the problem? Materials and Methods What did I find out? Results What does it mean? Discussion Who helped me out? Acknowledgments (optional) Whose work did I refer to? Literature Cited Extra InformationAppendices (optional)

8. Report Contents – Mandatory (1/2) System description  Used software and running environments Result graphs and tables Analysis & discussion (Very Important!!) 8 (C) 2008, SNU Biointelligence Laboratory

9. Report Contents – Mandatory (2/2) Basic experiments  Changing # of epochs (Draw learning curve)  Various # of Hidden Units 9 (C) 2008, SNU Biointelligence Laboratory # Hidden Units TrainTest Average  Std. Dev. BestWorstAverage  Std. Dev. BestWorst Setting 1accuracy Setting 2 Setting 3 

10. Report Contents – Optional Various experimental settings  Normalization  Learning rates  Structure of MLP  Feature selection  Activation functions  Learning algorithm  … Evaluation techniques  ROC curve  k-fold Crossvalidation  … 10 (C) 2008, SNU Biointelligence Laboratory

11. 11 (C) 2008, SNU Biointelligence Laboratory Submission Guide Due date: Apr. 15th (Wed.) 15:00 Submit both ‘hardcopy’ and ‘email’  Hardcopy submission to the office (301-419 )  E-mail submission to jakramate@bi.snu.ac.kr jakramate@bi.snu.ac.kr  Subject : [AI Project1 Report] Student number, Name  Length: report should be summarized within 12 pages.  If you build a program by yourself, submit the source code with comments We are NOT interested in the accuracy and your programming skill, but your creativity and research ability. If your major is not a C.S, team project with a C.S major student is possible (Use the class board to find your partner and notice the information of your team to the 1 st project TA(jakramate@bi.snu.ac.kr) by Mar. 27th)jakramate@bi.snu.ac.kr

12. Marking Scheme 40 points for experiment & analysis  Extra 4 points for additional expriments 20 points for report 6 points for overall organization Late work  - 10% per one day  Maximum 7 days * The Maximum Score is Changed 12 (C) 2008, SNU Biointelligence Laboratory

13. References Materials about Weka  Weka GUI guide (PPT)  Explorer guide (PDF)  Experimenter guide (PDF) 13 (C) 2008, SNU Biointelligence Laboratory

14. WEKA Demo 14 (C) 2008, SNU Biointelligence Laboratory

15. Matlab 15 (C) 2008, SNU Biointelligence Laboratory

16. QnA MLP is the simplest form of contemporary neural networks. (you can see other forms in the ‘ANN’ section of Wikipedia: http://en.wikipedia.org/wiki/Artificial_neural_network) http://en.wikipedia.org/wiki/Artificial_neural_network Neural network is sometimes called as ANN (artificial neural network) to stress the difference with the original neural network in the brain or central nervous system. Learning in neural networks consists in the optimization of weights by gradient descent process. To get the global optimum, we need to try not just several configurations of parameters, but also various random starting points.  When you use weka, you need to try several ‘randomSeed’ for this reason 16 (C) 2008, SNU Biointelligence Laboratory