1. 8/10/2015 1 RBF NetworksM.W. Mak Radial Basis Function Networks 1. Introduction 2. Finding RBF Parameters 3. Decision Surface of RBF Networks 4. Comparison between RBF and BP

2. 8/10/2015 2 RBF NetworksM.W. Mak 1. Introduction MLPs are highly non-linear in the parameter space  gradient descent  local minima l RBF networks solve this problem by dividing the learning into two independent processes. w 

3. 8/10/2015 3 RBF NetworksM.W. Mak l RBF networks implement the function w i  i and c i can be determined separately  Fast learning algorithm l Basis function types

4. 8/10/2015 4 RBF NetworksM.W. Mak l For Gaussian basis functions Assume the variance  across each dimension are equal

5. 8/10/2015 5 RBF NetworksM.W. Mak l To write in matrix form, let

6. 8/10/2015 6 RBF NetworksM.W. Mak 2. Finding the RBF Parameters l Use the K-mean algorithm to find c i

7. 8/10/2015 7 RBF NetworksM.W. Mak K-mean Algorithm step1:K initial clusters are chosen randomly from the samples to form K groups. step2:Each new sample is added to the group whose mean is the closest to this sample. step3:Adjust the mean of the group to take account of the new points. step4:Repeat step2 until the distance between the old means and the new means of all clusters is smaller than a predefined tolerance.

8. 8/10/2015 8 RBF NetworksM.W. Mak Outcome:There are K clusters with means representing the centroid of each clusters. Advantages: (1) A fast and simple algorithm. (2) Reduce the effects of noisy samples.

9. 8/10/2015 9 RBF NetworksM.W. Mak Use K nearest neighbor rule to find the function width  k-th nearest neighbor of c i l The objective is to cover the training points so that a smooth fit of the training samples can be achieved

10. 8/10/2015 10 RBF NetworksM.W. Mak Centers and widths found by K-means and K-NN

11. 8/10/2015 11 RBF NetworksM.W. Mak l Determining weights w using the least square method where d p is the desired output for pattern p

12. 8/10/2015 12 RBF NetworksM.W. Mak Let E be the total-squared error between the actual output and the target output

13. 8/10/2015 13 RBF NetworksM.W. Mak Note that Problems (1) Susceptible to round-off error. (2) No solution if is singular. (3) If is close to singular, we get very large component in w.

14. 8/10/2015 14 RBF NetworksM.W. Mak Reasons (1) Inaccuracy in forming (2) If A is ill-conditioned, small change in A introduces large change in (3)If A T A is close to singular, dependent columns in A T A exist e.g. two parallel straight lines. x y

15. 8/10/2015 15 RBF NetworksM.W. Mak singular matrix : If the lines are nearly parallel, they intersect each other at i.e. or So, the magnitude of the solution becomes very large; hence overflow will occur. The effect of the large components can be cancelled out if the machine precision is infinite.

16. 8/10/2015 16 RBF NetworksM.W. Mak If the machine precision is finite, we get large error. For example, Finite machine precision => Solution: Singular Value Decomposition

17. 8/10/2015 17 RBF NetworksM.W. Mak xpxp K-means K-Nearest Neighbor Basis Functions Linear Regression cici cici ii A w l RBF learning process

18. 8/10/2015 18 RBF NetworksM.W. Mak l RBF learning by gradient descent we have Apply

19. 8/10/2015 19 RBF NetworksM.W. Mak we have the following update equations

20. 8/10/2015 20 RBF NetworksM.W. Mak Elliptical Basis Function networks : function centers : covariance matrix

21. 8/10/2015 21 RBF NetworksM.W. Mak l K-means and Sample covariance K-means : if Sample covariance : l The EM algorithm

22. 8/10/2015 22 RBF NetworksM.W. Mak EBF Vs. RBF networks EBF Vs. RBF networks RBFN with 4 centersEBFN with 4 centers

23. 8/10/2015 23 RBF NetworksM.W. Mak EBF Network’s output Elliptical Basis Function Networks

24. 8/10/2015 24 RBF NetworksM.W. Mak RBFN for Pattern Classification MLP RBF HyperplaneKernel function The probability density function (also called conditional density function or likelihood) of the k-th class is defined as

25. 8/10/2015 25 RBF NetworksM.W. Mak According to Bays’ theorem, the posterior prob. is where P(C k ) is the prior prob. and It is possible to use a common pool of M basis functions, labeled by an index j, to represent all of the class-conditional densities, i.e.

26. 8/10/2015 26 RBF NetworksM.W. Mak

27. 8/10/2015 27 RBF NetworksM.W. Mak

28. 8/10/2015 28 RBF NetworksM.W. Mak Hidden node’s output posterior prob. of thej-th set of features in the input. weight posterior prob. of class membership, given the presence of thej-th set of features. No bias term

29. 8/10/2015 29 RBF NetworksM.W. Mak Comparison of RBF and MLP To learn more about NN hardware, see http://www.particle.kth.se/~lindsey/HardwareNNWCourse/home.html