1. Adavanced Numerical Computation 2008, AM NDHU
Radial Basis Function
Supervised Learning
Adavanced Numerical Computation 2008, AM NDHU

2. Adavanced Numerical Computation 2008, AM NDHU
Neural Networks
A neural network (NN), in the case of artificial neurons called artificial neural network (ANN) or simulated neural network (SNN), is an interconnected group of natural or artificial neurons that uses a mathematical or computational model for information processing based on a connectionistic approach to computation. In most cases an ANN is an adaptive system that changes its structure based on external or internal information that flows through the network.
In more practical terms neural networks are non-linear statistical data modeling or decision making tools. They can be used to model complex relationships between inputs and outputs or to find patterns in data.
However, the paradigm of neural networks - i.e., implicit, not explicit , learning is stressed - seems more to correspond to some kind of natural intelligence than to the traditional symbol-based Artificial Intelligence, which would stress, instead, rule-based learning.
Adavanced Numerical Computation 2008, AM NDHU

3. Adavanced Numerical Computation 2008, AM NDHU
|Neural_Networks-A Comprehensive Foundation -Simon&Haykin.pdf
Bernard Widrow Home
Adavanced Numerical Computation 2008, AM NDHU

4. Function Approximation
High-dimensional nonlinear function approximation
Linear combination of basis functions
Projective basis function
Radial basis function
Adavanced Numerical Computation 2008, AM NDHU

5. Function approximation
Data oriented function approximation
Paired data sampled from an unknown target function f : Rd R
(x[t], y[t]),t=1,…,N
Desired target (output)
Predictor (input)
Adavanced Numerical Computation 2008, AM NDHU

6. Adavanced Numerical Computation 2008, AM NDHU

7. Perceptron Projective basis function1
tanh y x a
Adavanced Numerical Computation 2008, AM NDHU

8. Adavanced Numerical Computation 2008, AM NDHU
Post-tanh projection
Posterior Weight
Linear projection
Post-nonlinear function
Adavanced Numerical Computation 2008, AM NDHU

9. Multiple (Multi-layer) perceptronsb1
tanh r1 1 b2 a1
tanh r2 a2 x bM
……. rM aM
tanh
rM+1 1
Adavanced Numerical Computation 2008, AM NDHU

10. MLP (multilayer perceptrons)
Weight sum of post-tanh projections
Adavanced Numerical Computation 2008, AM NDHU

11. Supervised learning of multilayer perceptrons
Applications
Supervised learning of multilayer perceptrons
Adavanced Numerical Computation 2008, AM NDHU

12. Adavanced Numerical Computation 2008, AM NDHU

13. Adavanced Numerical Computation 2008, AM NDHU

14. Adavanced Numerical Computation 2008, AM NDHU

15. Adavanced Numerical Computation 2008, AM NDHU

16. Adavanced Numerical Computation 2008, AM NDHU

17. Adavanced Numerical Computation 2008, AM NDHU

18. Adavanced Numerical Computation 2008, AM NDHU

19. Adavanced Numerical Computation 2008, AM NDHU

20. Adavanced Numerical Computation 2008, AM NDHU

21. Adavanced Numerical Computation 2008, AM NDHU

22. Adavanced Numerical Computation 2008, AM NDHU

23. Adavanced Numerical Computation 2008, AM NDHU

24. Adavanced Numerical Computation 2008, AM NDHU

25. Adavanced Numerical Computation 2008, AM NDHU
Radial basis function 1 r b
tanh y x a w x
exp
Adavanced Numerical Computation 2008, AM NDHU

26. Adavanced Numerical Computation 2008, AM NDHU
RBF Network function
Adavanced Numerical Computation 2008, AM NDHU

27. Adavanced Numerical Computation 2008, AM NDHU
RBF Network y
exp
exp
exp
exp .. .. x
Adavanced Numerical Computation 2008, AM NDHU

28. Why Post-Nonlinear Projection?
Why projection?
Linear projection versus Radial basis function
Why hyper-tangent
Hyper-tangent function versus exponential function
Why Euclidean distance?
Connections between exponential functions and hyper-tangent functions
Adavanced Numerical Computation 2008, AM NDHU

29. Adavanced Numerical Computation 2008, AM NDHU
Neural Networks
Books
Papers
Neural networks of radial basis functions reference
Adavanced Numerical Computation 2008, AM NDHU

30. Adavanced Numerical Computation 2008, AM NDHU
Network parameter
a vector of d elements
scalar
Adavanced Numerical Computation 2008, AM NDHU

31. Adavanced Numerical Computation 2008, AM NDHU
Supervised Learning
Derive optimal built-in parameters in  for approximating the target function underlying given paired training data
Quantitative Criterion
Mean square approximating error
Adavanced Numerical Computation 2008, AM NDHU

32. Adavanced Numerical Computation 2008, AM NDHU
Training and Testing
Both training and testing data are oriented from the same target function
Supervised learning is given training data and aimed to derive optimal built-in parameters
The derived built-in parameters is verified with testing data
Adavanced Numerical Computation 2008, AM NDHU

33. Adavanced Numerical Computation 2008, AM NDHU
Training set
Desired
output
MLP
Network
output
Adavanced Numerical Computation 2008, AM NDHU

34. Adavanced Numerical Computation 2008, AM NDHU
Training set
Desired
output
RBF
Network
output
Adavanced Numerical Computation 2008, AM NDHU

35. Adavanced Numerical Computation 2008, AM NDHU
Mean square errors
Adavanced Numerical Computation 2008, AM NDHU

36. Unconstrained optimization
Adavanced Numerical Computation 2008, AM NDHU

37. Revisit hyper-plane Fitting
Adavanced Numerical Computation 2008, AM NDHU

38. Adavanced Numerical Computation 2008, AM NDHU
Sampling
Blue : training data
Red : testing data
Adavanced Numerical Computation 2008, AM NDHU

39. Adavanced Numerical Computation 2008, AM NDHU
Fitting
Fitting blue training data
The performance is reflected by the mean square testing error
mean square training error
mean square testing error
Adavanced Numerical Computation 2008, AM NDHU

40. Verification of Generalization
Training phase
Given a training set,
Testing phase
Verify F(x;opt) by a testing set, which is assumed unknown during training phase
Both training and testing sets are assumed oriented from the same generating process
Adavanced Numerical Computation 2008, AM NDHU

41. Performance evaluation
Mean square error of training
Substitute each predictor in a training set to a network function
Determine the mean square error of approximating the desired target by the network output
Mean square error of testing indicates generalization capability of a network function
Adavanced Numerical Computation 2008, AM NDHU

42. Mean square training error
Adavanced Numerical Computation 2008, AM NDHU

43. Mean square testing error
Adavanced Numerical Computation 2008, AM NDHU

44. Adavanced Numerical Computation 2008, AM NDHU
Prediction
testing data
S_test ={(x[t],y[t]}
training data
S ={(x[t],y[t]}
y[t] +
x[t]
Learning
determine y_hat
sum -
Mean square testing Error
Adavanced Numerical Computation 2008, AM NDHU

45. Adavanced Numerical Computation 2008, AM NDHU
Over-fitting
Extremely low training error but high testing error
Over-fitting implies false generalization
Adavanced Numerical Computation 2008, AM NDHU

46. Adavanced Numerical Computation 2008, AM NDHU
Table lookup
Over-fitting
S: training set
Adavanced Numerical Computation 2008, AM NDHU

47. Adavanced Numerical Computation 2008, AM NDHU
MATLAB programming
RBF(Radial Basis Function)
Gradient descent method
Conjugate gradient method
Adavanced Numerical Computation 2008, AM NDHU

48. Adavanced Numerical Computation 2008, AM NDHU
Data structure
Flow charts
Algorithm
K-means
Gradient descent method
Conjugate gradient method
Adavanced Numerical Computation 2008, AM NDHU

49. Adavanced Numerical Computation 2008, AM NDHU
Reference
painless-conjugate-gradient.pdf
Nonlinear conjugate gradient method - Wikipedia, the free encyclopedia
Adavanced Numerical Computation 2008, AM NDHU

50. Adavanced Numerical Computation 2008, AM NDHU

51. Adavanced Numerical Computation 2008, AM NDHU
Network function
Adavanced Numerical Computation 2008, AM NDHU

52. Adavanced Numerical Computation 2008, AM NDHU
Network parameter
Adavanced Numerical Computation 2008, AM NDHU

53. Adavanced Numerical Computation 2008, AM NDHU
Data Structure
Net
Net.u : receptive field Mxd
Net.si : variance Mx1
Net.w : posterior weights M+1
Net.M : number of hidden units
Net.d : data dimension
Adavanced Numerical Computation 2008, AM NDHU

54. Adavanced Numerical Computation 2008, AM NDHU
LM_iniNet_RBF.m
Adavanced Numerical Computation 2008, AM NDHU

55. Adavanced Numerical Computation 2008, AM NDHU
Input & output
x: Nxd
N: data size
d: data dimension
y: Nx1
Net
demo_LM_RBF.m
Adavanced Numerical Computation 2008, AM NDHU

56. Adavanced Numerical Computation 2008, AM NDHU
RBF y
exp x ..
Adavanced Numerical Computation 2008, AM NDHU

57. RBF network Evaluation
y=0
function y=eva_f(Net,x)
EXIT
m=1:M
Adavanced Numerical Computation 2008, AM NDHU

58. Adavanced Numerical Computation 2008, AM NDHU
Cross distances
X: Nxd
u: Mxd
Adavanced Numerical Computation 2008, AM NDHU

59. Adavanced Numerical Computation 2008, AM NDHU
square error
Function2
function E= DC(Net,x,y)
Approximating error
e=y-yhat
Mean square error
mean(e.^2)
E=0
E=E/N
exit
i=1:N
Adavanced Numerical Computation 2008, AM NDHU

60. Adavanced Numerical Computation 2008, AM NDHU
Derivative
derivative.m
gtheta=[ gu gsi gw ]
% gtheta : N x (Md+2M+1)
% gu : N x Md
% gsi : N x M
% gw : N x ( M+1)
Adavanced Numerical Computation 2008, AM NDHU

61. Adavanced Numerical Computation 2008, AM NDHU
Since t runs from 1 to N and θ contains (Md+2M+1) elements
Gtheta is a matrix with N rows and (Md+2M+1) columns
gtheta : N x (Md+2M+1)
Adavanced Numerical Computation 2008, AM NDHU

62. Adavanced Numerical Computation 2008, AM NDHU
gtheta : N x (Md+2M+1)
gu : N x Md
Adavanced Numerical Computation 2008, AM NDHU

63. Adavanced Numerical Computation 2008, AM NDHU
gu=[ ]
exit
t=1:N
B=[ ]
gu=[gu;B]
m=1:M
Adavanced Numerical Computation 2008, AM NDHU

64. Adavanced Numerical Computation 2008, AM NDHU

65. Adavanced Numerical Computation 2008, AM NDHU
gtheta : N x (Md+2M+1)
gsi : N x M
Adavanced Numerical Computation 2008, AM NDHU

66. Adavanced Numerical Computation 2008, AM NDHU
gtheta : N x (Md+2M+1)
gw : N x (M+1)
Adavanced Numerical Computation 2008, AM NDHU

67. Adavanced Numerical Computation 2008, AM NDHU

68. Gradient descent method
Initialize , i=0 and set
Calculate
Update network parameters
If halting condition hold, exit
otherwise go to step 2
LM.m
Adavanced Numerical Computation 2008, AM NDHU

69. Adavanced Numerical Computation 2008, AM NDHU
Flow Chart
i=0; guess
~ halting
condition
Adavanced Numerical Computation 2008, AM NDHU

70. Adavanced Numerical Computation 2008, AM NDHU
Calculate delta_theta
Update theta
Adavanced Numerical Computation 2008, AM NDHU

71. Adavanced Numerical Computation 2008, AM NDHU
Project
Learning RBF or MLP by the LM method
Derivation
Matlab codes
Testing
Adavanced Numerical Computation 2008, AM NDHU

72. Adavanced Numerical Computation 2008, AM NDHU
Derivative
Adavanced Numerical Computation 2008, AM NDHU

73. Adavanced Numerical Computation 2008, AM NDHU
Derivative
Adavanced Numerical Computation 2008, AM NDHU

74. Adavanced Numerical Computation 2008, AM NDHU
Matlab coding
Main program
Matlab Functions
Calculate
Adavanced Numerical Computation 2008, AM NDHU

75. Adavanced Numerical Computation 2008, AM NDHU
Matlab coding
Matlab functions
Calculate
Adavanced Numerical Computation 2008, AM NDHU

76. Adavanced Numerical Computation 2008, AM NDHU
Matlab coding
Matlab functions
Calculate
Adavanced Numerical Computation 2008, AM NDHU

77. Adavanced Numerical Computation 2008, AM NDHU
Test
Give two examples to test your
matlab codes for learning RBF or MLP
networks by the gradient descent method
Adavanced Numerical Computation 2008, AM NDHU

78. Adavanced Numerical Computation 2008, AM NDHU
Evaluation
GD_iniNet
Form u
Form si
Form w
RBF evaluation
Calculation of mse
Determine gtheta gu
gsi gw
Update theta
Delta_theta
Gradient
Halting condition
Executable
Good performance
2D function approximation
Adavanced Numerical Computation 2008, AM NDHU