Adavanced Numerical Computation 2008, AM NDHU Radial Basis Function Supervised Learning Adavanced Numerical Computation 2008, AM NDHU

Adavanced Numerical Computation 2008, AM NDHU Neural Networks A neural network (NN), in the case of artificial neurons called artificial neural network (ANN) or simulated neural network (SNN), is an interconnected group of natural or artificial neurons that uses a mathematical or computational model for information processing based on a connectionistic approach to computation. In most cases an ANN is an adaptive system that changes its structure based on external or internal information that flows through the network. In more practical terms neural networks are non-linear statistical data modeling or decision making tools. They can be used to model complex relationships between inputs and outputs or to find patterns in data. However, the paradigm of neural networks - i.e., implicit, not explicit , learning is stressed - seems more to correspond to some kind of natural intelligence than to the traditional symbol-based Artificial Intelligence, which would stress, instead, rule-based learning. Adavanced Numerical Computation 2008, AM NDHU

Adavanced Numerical Computation 2008, AM NDHU |Neural_Networks-A Comprehensive Foundation -Simon&Haykin.pdf Bernard Widrow Home Adavanced Numerical Computation 2008, AM NDHU

Function Approximation High-dimensional nonlinear function approximation Linear combination of basis functions Projective basis function Radial basis function Adavanced Numerical Computation 2008, AM NDHU

Function approximation Data oriented function approximation Paired data sampled from an unknown target function f : Rd R (x[t], y[t]),t=1,…,N Desired target (output) Predictor (input) Adavanced Numerical Computation 2008, AM NDHU

Adavanced Numerical Computation 2008, AM NDHU

Perceptron Projective basis function 1 tanh y x a Adavanced Numerical Computation 2008, AM NDHU

Adavanced Numerical Computation 2008, AM NDHU Post-tanh projection Posterior Weight Linear projection Post-nonlinear function Adavanced Numerical Computation 2008, AM NDHU

Multiple (Multi-layer) perceptrons b1 tanh r1 1 b2 a1 tanh r2 a2 x bM ……. rM aM tanh rM+1 1 Adavanced Numerical Computation 2008, AM NDHU

MLP (multilayer perceptrons) Weight sum of post-tanh projections Adavanced Numerical Computation 2008, AM NDHU

Supervised learning of multilayer perceptrons Applications Supervised learning of multilayer perceptrons Adavanced Numerical Computation 2008, AM NDHU

Adavanced Numerical Computation 2008, AM NDHU

Adavanced Numerical Computation 2008, AM NDHU

Adavanced Numerical Computation 2008, AM NDHU

Adavanced Numerical Computation 2008, AM NDHU

Adavanced Numerical Computation 2008, AM NDHU

Adavanced Numerical Computation 2008, AM NDHU

Adavanced Numerical Computation 2008, AM NDHU

Adavanced Numerical Computation 2008, AM NDHU

Adavanced Numerical Computation 2008, AM NDHU

Adavanced Numerical Computation 2008, AM NDHU

Adavanced Numerical Computation 2008, AM NDHU

Adavanced Numerical Computation 2008, AM NDHU

Adavanced Numerical Computation 2008, AM NDHU

Adavanced Numerical Computation 2008, AM NDHU Radial basis function 1 r b tanh y x a w x exp Adavanced Numerical Computation 2008, AM NDHU

Adavanced Numerical Computation 2008, AM NDHU RBF Network function Adavanced Numerical Computation 2008, AM NDHU

Adavanced Numerical Computation 2008, AM NDHU RBF Network y exp exp exp exp .. .. x Adavanced Numerical Computation 2008, AM NDHU

Why Post-Nonlinear Projection? Why projection? Linear projection versus Radial basis function Why hyper-tangent Hyper-tangent function versus exponential function Why Euclidean distance? Connections between exponential functions and hyper-tangent functions Adavanced Numerical Computation 2008, AM NDHU

Adavanced Numerical Computation 2008, AM NDHU Neural Networks Books Papers Neural networks of radial basis functions reference Adavanced Numerical Computation 2008, AM NDHU

Adavanced Numerical Computation 2008, AM NDHU Network parameter a vector of d elements scalar Adavanced Numerical Computation 2008, AM NDHU

Adavanced Numerical Computation 2008, AM NDHU Supervised Learning Derive optimal built-in parameters in  for approximating the target function underlying given paired training data Quantitative Criterion Mean square approximating error Adavanced Numerical Computation 2008, AM NDHU

Adavanced Numerical Computation 2008, AM NDHU Training and Testing Both training and testing data are oriented from the same target function Supervised learning is given training data and aimed to derive optimal built-in parameters The derived built-in parameters is verified with testing data Adavanced Numerical Computation 2008, AM NDHU

Adavanced Numerical Computation 2008, AM NDHU Training set Desired output MLP Network output Adavanced Numerical Computation 2008, AM NDHU

Adavanced Numerical Computation 2008, AM NDHU Training set Desired output RBF Network output Adavanced Numerical Computation 2008, AM NDHU

Adavanced Numerical Computation 2008, AM NDHU Mean square errors Adavanced Numerical Computation 2008, AM NDHU

Unconstrained optimization Adavanced Numerical Computation 2008, AM NDHU

Revisit hyper-plane Fitting Adavanced Numerical Computation 2008, AM NDHU

Adavanced Numerical Computation 2008, AM NDHU Sampling Blue : training data Red : testing data Adavanced Numerical Computation 2008, AM NDHU

Adavanced Numerical Computation 2008, AM NDHU Fitting Fitting blue training data The performance is reflected by the mean square testing error mean square training error 0.159647 mean square testing error 0.160200 Adavanced Numerical Computation 2008, AM NDHU

Verification of Generalization Training phase Given a training set, Testing phase Verify F(x;opt) by a testing set, which is assumed unknown during training phase Both training and testing sets are assumed oriented from the same generating process Adavanced Numerical Computation 2008, AM NDHU

Performance evaluation Mean square error of training Substitute each predictor in a training set to a network function Determine the mean square error of approximating the desired target by the network output Mean square error of testing indicates generalization capability of a network function Adavanced Numerical Computation 2008, AM NDHU

Mean square training error Adavanced Numerical Computation 2008, AM NDHU

Mean square testing error Adavanced Numerical Computation 2008, AM NDHU

Adavanced Numerical Computation 2008, AM NDHU Prediction testing data S_test ={(x[t],y[t]} training data S ={(x[t],y[t]} y[t] + x[t] Learning determine y_hat sum - Mean square testing Error Adavanced Numerical Computation 2008, AM NDHU

Adavanced Numerical Computation 2008, AM NDHU Over-fitting Extremely low training error but high testing error Over-fitting implies false generalization Adavanced Numerical Computation 2008, AM NDHU

Adavanced Numerical Computation 2008, AM NDHU Table lookup Over-fitting S: training set Adavanced Numerical Computation 2008, AM NDHU

Adavanced Numerical Computation 2008, AM NDHU MATLAB programming RBF(Radial Basis Function) Gradient descent method Conjugate gradient method Adavanced Numerical Computation 2008, AM NDHU

Adavanced Numerical Computation 2008, AM NDHU Data structure Flow charts Algorithm K-means Gradient descent method Conjugate gradient method Adavanced Numerical Computation 2008, AM NDHU

Adavanced Numerical Computation 2008, AM NDHU Reference painless-conjugate-gradient.pdf Nonlinear conjugate gradient method - Wikipedia, the free encyclopedia Adavanced Numerical Computation 2008, AM NDHU

Adavanced Numerical Computation 2008, AM NDHU

Adavanced Numerical Computation 2008, AM NDHU Network function Adavanced Numerical Computation 2008, AM NDHU

Adavanced Numerical Computation 2008, AM NDHU Network parameter Adavanced Numerical Computation 2008, AM NDHU

Adavanced Numerical Computation 2008, AM NDHU Data Structure Net Net.u : receptive field Mxd Net.si : variance Mx1 Net.w : posterior weights M+1 Net.M : number of hidden units Net.d : data dimension Adavanced Numerical Computation 2008, AM NDHU

Adavanced Numerical Computation 2008, AM NDHU LM_iniNet_RBF.m Adavanced Numerical Computation 2008, AM NDHU

Adavanced Numerical Computation 2008, AM NDHU Input & output x: Nxd N: data size d: data dimension y: Nx1 Net demo_LM_RBF.m Adavanced Numerical Computation 2008, AM NDHU

Adavanced Numerical Computation 2008, AM NDHU RBF y exp x .. Adavanced Numerical Computation 2008, AM NDHU

RBF network Evaluation y=0 function y=eva_f(Net,x) EXIT m=1:M Adavanced Numerical Computation 2008, AM NDHU

Adavanced Numerical Computation 2008, AM NDHU Cross distances X: Nxd u: Mxd Adavanced Numerical Computation 2008, AM NDHU

Adavanced Numerical Computation 2008, AM NDHU square error Function2 function E= DC(Net,x,y) Approximating error e=y-yhat Mean square error mean(e.^2) E=0 E=E/N exit i=1:N Adavanced Numerical Computation 2008, AM NDHU

Adavanced Numerical Computation 2008, AM NDHU Derivative derivative.m gtheta=[ gu gsi gw ] % gtheta : N x (Md+2M+1) % gu : N x Md % gsi : N x M % gw : N x ( M+1) Adavanced Numerical Computation 2008, AM NDHU

Adavanced Numerical Computation 2008, AM NDHU Since t runs from 1 to N and θ contains (Md+2M+1) elements Gtheta is a matrix with N rows and (Md+2M+1) columns gtheta : N x (Md+2M+1) Adavanced Numerical Computation 2008, AM NDHU

Adavanced Numerical Computation 2008, AM NDHU gtheta : N x (Md+2M+1) gu : N x Md Adavanced Numerical Computation 2008, AM NDHU

Adavanced Numerical Computation 2008, AM NDHU gu=[ ] exit t=1:N B=[ ] gu=[gu;B] m=1:M Adavanced Numerical Computation 2008, AM NDHU

Adavanced Numerical Computation 2008, AM NDHU

Adavanced Numerical Computation 2008, AM NDHU gtheta : N x (Md+2M+1) gsi : N x M Adavanced Numerical Computation 2008, AM NDHU

Adavanced Numerical Computation 2008, AM NDHU gtheta : N x (Md+2M+1) gw : N x (M+1) Adavanced Numerical Computation 2008, AM NDHU

Adavanced Numerical Computation 2008, AM NDHU

Gradient descent method Initialize , i=0 and set Calculate Update network parameters If halting condition hold, exit otherwise go to step 2 LM.m Adavanced Numerical Computation 2008, AM NDHU

Adavanced Numerical Computation 2008, AM NDHU Flow Chart i=0; guess ~ halting condition Adavanced Numerical Computation 2008, AM NDHU

Adavanced Numerical Computation 2008, AM NDHU Calculate delta_theta Update theta Adavanced Numerical Computation 2008, AM NDHU

Adavanced Numerical Computation 2008, AM NDHU Project Learning RBF or MLP by the LM method Derivation Matlab codes Testing Adavanced Numerical Computation 2008, AM NDHU

Adavanced Numerical Computation 2008, AM NDHU Derivative Adavanced Numerical Computation 2008, AM NDHU

Adavanced Numerical Computation 2008, AM NDHU Derivative Adavanced Numerical Computation 2008, AM NDHU

Adavanced Numerical Computation 2008, AM NDHU Matlab coding Main program Matlab Functions Calculate Adavanced Numerical Computation 2008, AM NDHU

Adavanced Numerical Computation 2008, AM NDHU Matlab coding Matlab functions Calculate Adavanced Numerical Computation 2008, AM NDHU

Adavanced Numerical Computation 2008, AM NDHU Matlab coding Matlab functions Calculate Adavanced Numerical Computation 2008, AM NDHU

Adavanced Numerical Computation 2008, AM NDHU Test Give two examples to test your matlab codes for learning RBF or MLP networks by the gradient descent method Adavanced Numerical Computation 2008, AM NDHU

Adavanced Numerical Computation 2008, AM NDHU Evaluation GD_iniNet Form u Form si Form w RBF evaluation Calculation of mse Determine gtheta gu gsi gw Update theta Delta_theta Gradient Halting condition Executable Good performance 2D function approximation Adavanced Numerical Computation 2008, AM NDHU