Dynamics of Training Noh, Yung-kyun Mar. 11, 2003

Slides:

Advertisements

Similar presentations

Multilayer Perceptrons 1. Overview  Recap of neural network theory  The multi-layered perceptron  Back-propagation  Introduction to training  Uses.

Advertisements

Biointelligence Laboratory, Seoul National University

Introduction to Training and Learning in Neural Networks n CS/PY 399 Lab Presentation # 4 n February 1, 2001 n Mount Union College.

Particle Swarm Optimization (PSO)  Kennedy, J., Eberhart, R. C. (1995). Particle swarm optimization. Proc. IEEE International Conference.

1 A Statistical Mechanical Analysis of Online Learning: Seiji MIYOSHI Kobe City College of Technology

Back-propagation Chih-yun Lin 5/16/2015. Agenda Perceptron vs. back-propagation network Network structure Learning rule Why a hidden layer? An example:

The loss function, the normal equation,

The Nature of Statistical Learning Theory by V. Vapnik

Prénom Nom Document Analysis: Artificial Neural Networks Prof. Rolf Ingold, University of Fribourg Master course, spring semester 2008.

Neural Networks Marco Loog.

Estimation of Oil Saturation Using Neural Network Hong Li Computer System Technology NYC College of Technology –CUNY Ali Setoodehnia, Kamal Shahrabi Department.

Artificial Neural Networks

Gene based diagnostic prediction of cancers by using Artificial Neural Network Liya Wang ECE/CS/ME539.

Biointelligence Laboratory, Seoul National University

Kernel Classifiers from a Machine Learning Perspective (sec ) Jin-San Yang Biointelligence Laboratory School of Computer Science and Engineering.

11 CSE 4705 Artificial Intelligence Jinbo Bi Department of Computer Science & Engineering

Appendix B: An Example of Back-propagation algorithm

Rotation Invariant Neural-Network Based Face Detection

Classification / Regression Neural Networks 2

An informal description of artificial neural networks John MacCormick.

Cross strait Quad-reginal radio science and wireless technology conference, Vol. 2, p.p ,2011 Application of fuzzy LS-SVM in dynamic compensation.

CHAPTER 5 S TOCHASTIC G RADIENT F ORM OF S TOCHASTIC A PROXIMATION Organization of chapter in ISSO –Stochastic gradient Core algorithm Basic principles.

Artificial Intelligence Chapter 3 Neural Networks Artificial Intelligence Chapter 3 Neural Networks Biointelligence Lab School of Computer Sci. & Eng.

Multi-Layer Perceptron

Survey of Kernel Methods by Jinsan Yang. (c) 2003 SNU Biointelligence Lab. Introduction Support Vector Machines Formulation of SVM Optimization Theorem.

Insight: Steal from Existing Supervised Learning Methods! Training = {X,Y} Error = target output – actual output.

Introduction to Neural Networks Introduction to Neural Networks Applied to OCR and Speech Recognition An actual neuron A crude model of a neuron Computational.

Face Image-Based Gender Recognition Using Complex-Valued Neural Network Instructor :Dr. Dong-Chul Kim Indrani Gorripati.

Neural Networks Presented by M. Abbasi Course lecturer: Dr.Tohidkhah.

Neural Networks Teacher: Elena Marchiori R4.47 Assistant: Kees Jong S2.22

Hazırlayan NEURAL NETWORKS Backpropagation Network PROF. DR. YUSUF OYSAL.

Neural Networks Vladimir Pleskonjić 3188/ /20 Vladimir Pleskonjić General Feedforward neural networks Inputs are numeric features Outputs are in.

Introduction to Neural Networks Freek Stulp. 2 Overview Biological Background Artificial Neuron Classes of Neural Networks 1. Perceptrons 2. Multi-Layered.

Perceptrons Michael J. Watts

Dynamic Neural Network Control (DNNC): A Non-Conventional Neural Network Model Masoud Nikravesh EECS Department, CS Division BISC Program University of.

Learning Kernel Classifiers 1. Introduction Summarized by In-Hee Lee.

Neural Networks The Elements of Statistical Learning, Chapter 12 Presented by Nick Rizzolo.

Meta-controlled Boltzmann Machine toward Accelerating the Computation Tran Duc Minh (*), Junzo Watada (**) (*) Institute Of Information Technology-Viet.

Biointelligence Lab School of Computer Sci. & Eng. Seoul National University Artificial Intelligence Chapter 8 Uninformed Search.

Learning: Neural Networks Artificial Intelligence CMSC February 3, 2005.

Learning with Neural Networks Artificial Intelligence CMSC February 19, 2002.

CSSE463: Image Recognition Day 14

Deep Feedforward Networks

Learning with Perceptrons and Neural Networks

Computer Science and Engineering, Seoul National University

ECE 539 Project Jialin Zhang

Ranga Rodrigo February 8, 2014

CSE 473 Introduction to Artificial Intelligence Neural Networks

Announcements HW4 due today (11:59pm) HW5 out today (due 11/17 11:59pm)

Biological and Artificial Neuron

Biological and Artificial Neuron

Artificial Intelligence Methods

Gene expression profiling diagnosis through DNA molecular computation

Aapo Hyvärinen and Ella Bingham

Artificial Intelligence Chapter 3 Neural Networks

Biological and Artificial Neuron

Neural Networks Chapter 5

Subhayu Basu et al. , DNA8, (2002) MEC Seminar Su Dong Kim

Biointelligence Laboratory, Seoul National University

Artificial Intelligence Chapter 3 Neural Networks

Learning Control for Dynamically Stable Legged Robots

Recursively Adapted Radial Basis Function Networks and its Relationship to Resource Allocating Networks and Online Kernel Learning Weifeng Liu, Puskal.

Artificial Intelligence Chapter 3 Neural Networks

Neural networks (1) Traditional multi-layer perceptrons

Masoud Nikravesh EECS Department, CS Division BISC Program

Artificial Intelligence 10. Neural Networks

Artificial Intelligence Chapter 3 Neural Networks

Ch 3. Linear Models for Regression (2/2) Pattern Recognition and Machine Learning, C. M. Bishop, Previously summarized by Yung-Kyun Noh Updated.

Structure of a typical back-propagated multilayered perceptron used in this study. Structure of a typical back-propagated multilayered perceptron used.

Artificial Intelligence Chapter 3 Neural Networks

Presentation transcript:

Dynamics of Training Noh, Yung-kyun Mar. 11, 2003 NIPS'1996 Volume 9, pp141-147 Noh, Yung-kyun Mar. 11, 2003

(C) 2003, SNU BioIntelligence Lab Introduction(1/3) Training guided by empirical risk minimization does not always minimize the expected risk. Namely overfitting A new description which is directly dependent on the actual traing steps. We will examine empirical risk and expected risk as functions of the traing time. Restrict ourselves to a quite simple neural network model. (C) 2003, SNU BioIntelligence Lab

(C) 2003, SNU Biointelligence Lab Introduction(2/3) Single layer perceptron with N-dim. Examples Nonlinear output function (C) 2003, SNU Biointelligence Lab

(C) 2003, SNU Biointelligence Lab Introduction(3/3) Learning by examples attempts to minimize this. We are interested in this.(generalization error or expected risk) R, Q : order parameters (C) 2003, SNU Biointelligence Lab

Dynamical approach(1/5) Dynamics Weights w.r.t. input data Dynamics of (C) 2003, SNU Biointelligence Lab

Dynamical approach(2/5) : P/N : trace. Expressed as Integration over eigenvalues When < 1 When > 1 (C) 2003, SNU Biointelligence Lab

Dynamical approach(3/5) (C) 2003, SNU Biointelligence Lab

Dynamical approach(4/5) (C) 2003, SNU Biointelligence Lab

Dynamical approach(5/5) (C) 2003, SNU Biointelligence Lab

(C) 2003, SNU Biointelligence Lab Conclusion The behavior of the learning and the training error during the whole training process. How good this theory describes errors and actual number of training steps. With sufficiently large , two batch training steps are necessary to reach the optimal convergence rate. Thermodynamic description of the training process can be added. This method could be extened towards other, more realistic models. (C) 2003, SNU Biointelligence Lab