1. Radial Basis Function Networks
In the name of God
Institute for advanced studies in basic sciences
Radial Basis Function Networks
Yousef Akhlaghi

2. This seminar is an introduction to radial basis function networks as linear neural networks.
Definition :
Radial basis function (RBF) networks are a special class of single hidden-layer feed forward neural networks for application to problems of supervised learning.
Reference: Radial Basis Function Networks, Mark J. Orre,

3. Linear models have been studied in statistics for about 200 years and the theory is applicable to RBF networks which are just one particular type of linear model. However the fashion for neural networks which started in the mid-80’s has given rise to new names for concepts already familiar to statisticians.
Equivalent terms in statistics and neural networks:

4. Supervised Learning:
Problem to be solved: Estimate a function from some example input-output pairs with little or no knowledge of the form of the function.
DIFFERENT NAMES OF THIS PROBLEM:
Nonparametric regression
Function approximation
System identification
Inductive learning
IN NEURAL NETWORK : Supervised learning
The function is learned from the examples or training set contains elements which consist of paired values of the independent input variable and the dependent output variable.

5. Parametric Regression:
Two main regression problems in statistics:
- parametric regression
- nonparametric regression
Parametric Regression:
In parametric regression the form of the functional relationship between the dependent and independent variables is known but may contain parameters whose values are unknown and capable of being estimated from the training set.
For example fitting a straight line:
parameters
to a bunch of points
Important point: The free parameters as well as the dependent and independent variables have meaningful interpretations like initial concentration or rate.

6. Nonparametric regression:
There is no or very little a priori knowledge about the form of the true function which is being estimated.
The function is still modeled using an equation containing free parameters but Typically this involves using many free parameters which have no physical meaning in relation to the problem. In parametric regression there is typically a small number of parameters and often they have physical interpretations.
In neural networks including radial basis function networks:
- Models are nonparametric and their weights and other parameters have no particular meaning in relation to the problems to which they are applied.
-The primary goal is to estimate the underlying function or at least to estimate its output at certain desired values of the input

7. Linear Models: Nonlinear Models:
A linear model for a function takes the form:
The model ‘f ’ is expressed as a linear combination of a set of ‘m’ fixed functions often called basis functions by analogy with the concept of a vector being composed of a linear combination of basis vectors.
Nonlinear Models:
if the basis functions can change during the learning process then the model is nonlinear.

8. An example
Almost the simplest polynomial is the straight line:
which is a linear model whose two basis functions are:
and whose parameters (weights) are:

9. The two main advantages of the linear character of RBF networks:
1- Keeping the mathematics simple it is just linear algebra (the linearly weighted structure of RBF networks)
2- There is no optimization by general purpose gradient descent algorithms (without involving nonlinear optimization).

10. Radial Functions:
Their characteristic feature is that their response decreases (or increases) monotonically with distance from a central point. The centre, the distance scale, and the precise shape of the radial function, are parameters of the model, all fixed if it is linear.
A typical radial function is the Gaussian:
Its parameters are
c : centre
r : width (spread)

11. Local modelling with radial basis function networks
B. Walczak , D.L. Massart
Chemometrics and Intelligent Laboratory Systems 50 (2000) 179–198

12. From exact fitting to RBFNs

13. RBF methods were originally developed for exact interpolation of a set of data points in a multidimensional space. The aim of exact interpolation is to project every input vector , onto the corresponding target , to find a function such that:
According to the radial basis function approach, exact mapping can be performed using a set of m basis functions (one for each data point) with the form , where is some nonlinear function, and denotes distance between and , usually Euclidean distance. Then the output of the mapping can be presented as linear combinations of these basis functions:
where denotes weights, and denote input object and the center of basis function, respectively.

14. In matrix notation the above equation is:
For a large class of functions, the matrix Φ is non-singular, and eq. 3 can be solved:
The basis functions can have different forms. The most popular among them is the Gaussian function:
controlling the smoothness properties of the interpolating function.

16. || xi – xj || Euclidean Distance of center j from object i
= [(xi1 - x1j)2 + (xi2–x2j)2 + …+ (xin-xnj)2]0.5

17. S Exact Fitting f × × × # Hidd. nodes = # objects object w center
Distance (x) S × ×
output ×
# Hidd. nodes = # objects
Exact Fitting

18. 2) Bias parameters are included in the linear sum.
In practice, we do not want exact modeling of the training data, as the constructed model would have a very poor predictive ability, due to fact that all details noise, outliers are modeled.
To have a smooth interpolating function in which the number of basis functions is determined by the fundamental complexity of the data structure, some modifications to the exact interpolation method are required.
By introducing at least the first two modifications to the exact interpolation method we obtain RBFN.
1) The number K of basis functions need not equal the number m of data points, and is typically much less than m.
2) Bias parameters are included in the linear sum.
3) The determination of suitable centers becomes part of the training process.
4) Instead of having a common width parameters, each basis function is given its own width σj whose value is also determined during training.

19. Architecture of RBFN

20. RBFN can be presented as a three-layer feedforward structure.
The input layer serves only as input distributor to the hidden layer.
Each node in the hidden layer is a radial function, its dimensionality being the same as the dimensionality of the input data.
The output is calculated by a linear combination . i.e. a weighted sum of the radial basis functions plus the bias, according to:

21. In matrix notation:
Nod1
Nod2
Nodk
bias
object1

22. Training algorithms

23. RBF network parameters
The centers of the RBF activation functions
The spreads of the Gaussian RBF activation functions
The weights from the hidden to the output layer

24. subset selection: forward selection backward elimination
Different subsets of basis functions can be drawn from the same fixed set of candidates. This is called subset selection in statistics.
forward selection
starts with an empty subset
added one basis function at a time (the one that most reduces the sum-squared-error)
until some chosen criterion stops
backward elimination
starts with the full subset
removed one basis function at a time ( the one that least increases the sum-squared-error)
until the chosen criterion stops decreasing

25. Initially, the basis functions are centered on data objects.
Orthonormalization of basis functions:
The Gram–Schmidt orthogonalization procedure is used to replace the k basis functions by the set of K orthogonal vectors
describing the same space.
Initially, the basis functions are centered on data objects. :
The set of vectors ui , which are a linear combination of the functions , can be used directly to model y. To model y, the consecutive orthonormal vectors ui (i =1,2, m) are introduced until the network performance reaches the desired level of approximation error.

26. Nonlinear neural network
Nonlinear training algorithm:
Nonlinear neural network
Apply the gradient descent method for finding centers, spread and weights, by minimizing the cost function (in most cases squared error). Back-propagation adapts iteratively the network parameters considering the derivatives of the cost function with respect to those parameters.
Drawback: Back-propagation algorithm may require several iterations and can get stuck into a local minima of the cost function

27. Radial Basis Function Neural Net in MATLAB nnet-Toolbox
function [net, tr] = newrb (p, t, goal, spread, mn, df)
NEWRB adds neurons to the hidden layer of a radial basis network until it meets the specified mean squared error goal.
P RxQ matrix of Q input vectors.
T SxQ matrix of Q target class vectors.
GOAL Mean squared error goal, default = 0.0.
SPREAD - Spread of radial basis functions, default = 1.0.
MN Maximum number of neurons, default is Q.
DF Number of neurons to add between displays, default = 25.
The following steps are repeated until the network's mean squared error falls below GOAL or the maximum number of neurons are reached:
1) The network is simulated with random weight
2) The input vector with the greatest error is found
3) A neuron (basis function) is added with weights equal to that vector.
4) The output layer weights are redesigned to minimize error.

28. THANKS