1. Slide number 1 EE3P BEng Final Year Project Group Session 2 Processing Patterns using a Multi- Layer Perceptron (MLP) Martin Russell

2. Slide number 2 Objectives To show you how to classify patterns using a type of artificial neural network To introduce the Multi Layer Perceptron (MLP) To describe the MLP software provided and to show you how to use it To set some practical work to be completed before the next session

3. Slide number 3 The Application Classification of spoken vowels from their first 2 formant frequencies, F1 and F2 Use a well-known set of data, collected by Peterson & Barney in 1952 Called the “Peterson-Barney” data

4. Slide number 4 Petersen Barney data 76 people (33 male, 28 female, 15 children) each spoke 2 examples of 10 vowels – 1520 utterances. For each utterance, Peterson & Barney measured: –F0: the fundamental frequency –F1, F2 and F3: the frequencies of the first 3 formants We will just use F1 and F2

5. Slide number 5 The Vowels 1.IY “beet” 2.IH “bit” 3.EH “bet” 4.AE “bat” 5.AA “bott” 6.AO “bought” 7.UH “book” 8.UW “boot” 9.AH “but” 10.ER “bird”

6. Slide number 6 Peterson & Barney F1 F2 data

7. Slide number 7 The Basic Idea F1F2 IY IH EH AE AA AO UH UW AH ER MLP e.g: F1 F2 correspond to ‘AO’ 0 0 0 0 0 1 0 0 0 0

8. Slide number 8 Understanding the MLP ? F1F2 “AA” F1F2 “AA” F1F2 “AA”

9. Slide number 9 A Multi-Layer Perceptron (MLP) Input units Output units “hidden” units, “artificial neurones”

10. Slide number 10 An “artificial neurone” x y 1 (constant) w1w1 w2w2 w3w3 h Input to h is given by: I(h) = w 1 x + w 2 y + w 3 Output from h is given by: O(h) = g(I(h)) ‘Sigmoid’ activation function

11. Slide number 11 The ‘sigmoid’ activation function g Value of k

12. Slide number 12 Artificial neurone (summary) The input I(h) to the hidden unit h is the weighted sum of the outputs from the inout units above it If I(h) > 0, O(h) ~ 1 (neurone ‘fires’) If I(h) < 0, O(h) ~ 0 (neurone doesn’t ‘fire’)

13. Slide number 13 Back to the P&B data For the Peterson & Barney application we use: –3 input units: 1 st for F1, 2 nd for F2, and 3 rd set equal to 1 –10 output units, 1 for each vowel class –Variable number of hidden units

14. Slide number 14 Example ‘P&B’ MLP 3 input units 10 output units

15. Slide number 15 ‘Winner takes all’ coding This is the scheme for coding the outputs, or ‘targets’ for the MLP If F1, F2 correspond to IY, target is (1,0,0,0,0,0,0,0,0,0) If F1, F2 correspond to IH, target is (0,1,0,0,0,0,0,0,0,0) If F1, F2 correspond to EH, target is (0,0,1,0,0,0,0,0,0,0) …. If F1, F2 correspond to ER, target is (0,0,0,0,0,0,0,0,0,1)

16. Slide number 16 “Black Art”: input scaling With MLPs it is normal to scale each input so that the average is 1 and variance is 1 For the Peterson & Barney F1 and F2 data I have: –Replaced F1 with (F1 – av(F1))/sd(F1) –Replaced F2 with (F2 – av(F2))/sd(F2) av = average, sd = standard deviation

17. Slide number 17 Data file format 200 3 10 0.560000 1.719298 11 0 0 0 0 0 0 0 0 0 0.576000 1.754386 11 0 0 0 0 0 0 0 0 0 0.800000 1.459649 10 1 0 0 0 0 0 0 0 0 0.768000 1.480702 10 1 0 0 0 0 0 0 0 0 1.180000 1.333333 10 0 1 0 0 0 0 0 0 0 1.166000 1.291228 10 0 1 0 0 0 0 0 0 0 1.360000 1.298246 10 0 0 1 0 0 0 0 0 0 1.370000 1.249123 10 0 0 1 0 0 0 0 0 0 1.320000 0.961403 10 0 0 0 1 0 0 0 0 0. ‘header’ information Input values (F1 F2 1) Target values

18. Slide number 18 Data Files There are two data files –pbData_m_f1f2_norm_train (the training set) –pbData_m_f1f2_norm_test (the test set) Both are on the web site

19. Slide number 19 How to build an MLP 1.First decide how many hidden units 2.Then choose initial estimates for each of the MLP parameters. Often these are chosen at random Use MLPInit.c from website MLPInit.c sets the weights randomly

20. Slide number 20 Using MLPInit MLPInit 3 5 10 init_3x5x10 Creates an MLP with: 3 input units 5 hidden units 10 output units Result is stored in the file init_3x5x10

21. Slide number 21 Example output from MLPInit 3 5 10 0.416415 0.418549 0.242699 0.333669 0.199291 0.195465 0.477893 0.556194 0.287547 0.280483 0.388121 0.103558 0.201107 0.378784 0.520226 0.303600 0.012010 0.215942 0.176979 0.171643 0.183424 0.113759 0.223094 0.365760 0.073800 0.210656 0.178952 0.069903 0.289522 0.161591 0.268101 0.260700 0.036500 0.067274 0.300626 0.237691 0.101108 0.058603 0.211381 0.318471 0.139368 0.296624 0.081525 0.020991 0.279433 0.046923 0.592494 0.538003 0.102141 0.095765 0.310450 0.022514 0.281300 0.479829 0.263548 0.201131 0.115437 0.117549 0.219977 0.252530 0.098656 0.306403 0.377581 0.066146 0.082592 See website

22. Slide number 22 MLP Training Suppose you have: –Input values 1.212 3.127 1, and –Target values 0 0 0 0 1 0 0 0 0 0 If you apply 1.212 3.127 1 to the input of your MLP, then you would like output to be target The (squared) difference between the target output and actual output is called the Error By adjusting the MLP weights this error can be reduced The algorithm for reducing this error is called Error Back-Propagation

23. Slide number 23 MLPTrain.c MLPTrain.c implements the Error Back- Propagation training algorithm It’s available on the website

24. Slide number 24 How to use MLPTrain.c Type: MLPTrain initMLP dataFile numIt trainedMLP initMLP is the ‘initial’ MLP (e.g. ini_3x5x10) dataFile is the training data file (e.g. pbData_m_f1f2_train) numIt is the number of iterations trainedMLP is the file for the optimised MLP Example: MLPTrain ini_3x5x10 pbData_m_f1f2_train 10000 mlp_3x5x10

25. Slide number 25 Example MLPTrain ini_3x5x10 pbData_m_f1f2_train 50000 mlp_50000_3x5x10

26. Slide number 26 Testing the MLP Use the test set pbData_m_f1f2_norm_test –Consists of 200 test F1 F2 patterns Want to input each test pattern and see if the MLP gives the ‘correct’ output pattern The output is correct if the output unit with the biggest value corresponds to the ‘1’ in the target

27. Slide number 27 Testing the MLP Example 1: –Output = 0.1 0.85 0.2 0.0 0.0 0.12 0.0 0.0 0.0 0.0 –Target= 0 1 0 0 0 0 0 0 0 0 –Correct Example 2: –Output = 0.1 0.85 0.2 0.0 0.0 0.12 0.0 0.0 0.0 0.0 –Target= 0 0 1 0 0 0 0 0 0 0 –Error

28. Slide number 28 MLPClass.c To test your MLP use MLPClass.c from the website MLPClass mlpFile testData Where: –mlpFile is the trained MLP file –testData is the test data file Example: MLPClass mlp_3x5x10 pbData_m_f1f2_test

29. Slide number 29 Note on the C programs All of the C programs are ANSI C compatible They were developed under linux, but that doesn’t matter They should compile on any platform I recommend the command-line C compiler under MS visual studio.net

30. Slide number 30 More “black art” The programs MLPTrain and MLPClass need a set of parameters These are stored in a file called mlpconfig (see website) This file must be in your directory when you run MLPTrain or MLPClass

31. Slide number 31 mlpconfig 1.0 0.0 BATCH This file is on the website Options for are BATCH or SEQUENTIAL is just k from slides 10 and 11

32. Slide number 32 Tasks Download C programs from the website Compile them Download Peterson & Barney data from the website Follow the steps outlined in these slides to build and test some MLPs –How many hidden units do you need? –How many iterations of training are needed? –What recognition rate can you achieve? –What the ‘learning rate’ and ‘momentum’ and how do they affect the results? Find out more about how an MLP works