Lecture 10 Artificial Neural Networks Dr. Jianjun Hu mleg.cse.sc.edu/edu/csce833 CSCE833 Machine Learning University of South Carolina Department of Computer Science and Engineering

Outline Midterm moved to March 15 th Neural Network Learning Self-Organizing Maps  Origins  Algorithm  Example Lecture Notes for E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1) 2

Neuron: no division, only 1 axon Lecture Notes for E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1) 3

4 Neural Networks Networks of processing units (neurons) with connections (synapses) between them Large number of neurons: Large connectitivity: 10 5 Parallel processing Distributed computation/memory Robust to noise, failures

Lecture Notes for E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1) 5 Understanding the Brain Levels of analysis (Marr, 1982) 1. Computational theory 2. Representation and algorithm 3. Hardware implementation Reverse engineering: From hardware to theory Parallel processing: SIMD vs MIMD Neural net: SIMD with modifiable local memory Learning: Update by training/experience

Lecture Notes for E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1) 6 Perceptron (Rosenblatt, 1962)

Lecture Notes for E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1) 7 What a Perceptron Does Regression: y=wx+w 0 Classification: y=1(wx+w 0 >0) w w0w0 y x x 0 =+1 w w0w0 y x s w0w0 y x

Lecture Notes for E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1) 8 K Outputs Classification : Regression :

Lecture Notes for E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1) 9 Training Online (instances seen one by one) vs batch (whole sample) learning:  No need to store the whole sample  Problem may change in time  Wear and degradation in system components Stochastic gradient-descent: Update after a single pattern Generic update rule (LMS rule):

Lecture Notes for E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1) 10 Training a Perceptron: Regression Regression (Linear output):

Lecture Notes for E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1) 11 Classification Single sigmoid output K>2 softmax outputs

Lecture Notes for E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1) 12 Learning Boolean AND

Lecture Notes for E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1) 13 XOR No w 0, w 1, w 2 satisfy: (Minsky and Papert, 1969)

Lecture Notes for E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1) 14 Multilayer Perceptrons (Rumelhart et al., 1986)

Lecture Notes for E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1) 15 x 1 XOR x 2 = (x 1 AND ~x 2 ) OR (~x 1 AND x 2 )

Lecture Notes for E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1) 16 Backpropagation

Lecture Notes for E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1) 17 Regression Forward Backward x

Lecture Notes for E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1) 18 Regression with Multiple Outputs zhzh v ih yiyi xjxj w hj

Lecture Notes for E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1) 19

Lecture Notes for E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1) 20

Lecture Notes for E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1) 21 whx+w0whx+w0 zhzh vhzhvhzh

Lecture Notes for E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1) 22 Two-Class Discrimination One sigmoid output y t for P(C 1 |x t ) and P(C 2 |x t ) ≡ 1-y t

Lecture Notes for E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1) 23 K>2 Classes

Lecture Notes for E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1) 24 Multiple Hidden Layers MLP with one hidden layer is a universal approximator (Hornik et al., 1989), but using multiple layers may lead to simpler networks

Lecture Notes for E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1) 25 Improving Convergence Momentum Adaptive learning rate

Lecture Notes for E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1) 26 Overfitting/Overtraining Number of weights: H (d+1)+(H+1)K

Lecture Notes for E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1) 27

Lecture Notes for E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1) 28 Tuning the Network Size Destructive Weight decay: Constructive Growing networks (Ash, 1989) (Fahlman and Lebiere, 1989)

Lecture Notes for E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1) 29 Dimensionality Reduction

Lecture Notes for E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1) 30

Lecture Notes for E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1) 31 Learning Time: sequential learning Applications:  Sequence recognition: Speech recognition  Sequence reproduction: Time-series prediction  Sequence association Network architectures  Time-delay networks (Waibel et al., 1989)  Recurrent networks (Rumelhart et al., 1986)

Lecture Notes for E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1) 32 Time-Delay Neural Networks

Lecture Notes for E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1) 33 Recurrent Networks

Lecture Notes for E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1) 34 Unfolding in Time

Self-Organizing Maps : Origins Self-Organizing Maps  Ideas first introduced by C. von der Malsburg (1973), developed and refined by T. Kohonen (1982)  Neural network algorithm using unsupervised competitive learning  Primarily used for organization and visualization of complex data  Biological basis: ‘brain maps’ Teuvo Kohonen

Self-Organizing Maps SOM - Architecture  Lattice of neurons (‘nodes’) accepts and responds to set of input signals  Responses compared; ‘winning’ neuron selected from lattice  Selected neuron activated together with ‘neighbourhood’ neurons  Adaptive process changes weights to more closely resemble inputs 2d array of neurons Set of input signals (connected to all neurons in lattice) Weighted synapses x1x1 x2x2 x3x3 xnxn... w j1 w j2 w j3 w jn j

Self-Organizing Maps SOM – Result Example ‘Poverty map’ based on 39 indicators from World Bank statistics (1992) Classifying World Poverty Helsinki University of Technology

Self-Organizing Maps SOM – Result Example ‘Poverty map’ based on 39 indicators from World Bank statistics (1992) Classifying World Poverty Helsinki University of Technology

Self-Organizing Maps SOM – Algorithm Overview 1. Randomly initialise all weights 2. Select input vector x = [x 1, x 2, x 3, …, x n ] 3. Compare x with weights w j for each neuron j to determine winner 4. Update winner so that it becomes more like x, together with the winner’s neighbours 5. Adjust parameters: learning rate & ‘neighbourhood function’ 6. Repeat from (2) until the map has converged (i.e. no noticeable changes in the weights) or pre-defined no. of training cycles have passed

Initialisation (i)Randomly initialise the weight vectors w j for all nodes j

(ii) Choose an input vector x from the training set In computer texts are shown as a frequency distribution of one word. A Text Example: Self-organizing maps (SOMs) are a data visualization technique invented by Professor Teuvo Kohonen which reduce the dimensions of data through the use of self- organizing neural networks. The problem that data visualization attempts to solve is that humans simply cannot visualize high dimensional data as is so technique are created to help us understand this high dimensional data. Input vector Region Self-organizing 2 maps 1 data4 visualization 2 technique 2 Professor 1 invented 1 Teuvo Kohonen 1 dimensions 1... Zebra0

Finding a Winner (iii) Find the best-matching neuron  (x), usually the neuron whose weight vector has smallest Euclidean distance from the input vector x The winning node is that which is in some sense ‘closest’ to the input vector ‘Euclidean distance’ is the straight line distance between the data points, if they were plotted on a (multi-dimensional) graph Euclidean distance between two vectors a and b, a = (a 1,a 2,…,a n ), b = (b 1,b 2,…b n ), is calculated as: Euclidean distance

Weight Update SOM Weight Update Equation w j (t +1) = w j (t) +  (t)  (x) (j,t) [x - w j (t)] “The weights of every node are updated at each cycle by adding Current learning rate × Degree of neighbourhood with respect to winner × Difference between current weights and input vector to the current weights” Example of  (t) Example of  (x) (j,t) L. rate No. of cycles –x-axis shows distance from winning node –y-axis shows ‘degree of neighbourhood’ (max. 1)

Example: Self-Organizing Maps The animals should be ordered by a neural networks. And the animals will be described with their attributes(size, living space). e.g. Mouse = (0/0) Size:Living space: small=0 medium=1 big=2Land=0 Water=1 Air=2 MouseLionHorseSharkDove Size small big mediumsmall big Living space Land AirWaterLand (2/0)(0/0)(0/2)(2/1)(1/0)

Example: Self-Organizing Maps This training will be very often repeated. In the best case the animals should be at close quarters ordered by similarest attribute. (0.75/0.6875) (0.1875/1.25) Dove (1.125/1.625) (1.375/0.5)(1/0.875) (1.5/0) Hourse (1.625/1) Shark (1/0.75) Lion (0.75/0) Mouse Land animals

Example: Self-Organizing Maps [Teuvo Kohonen 2001] Self-Organizing Maps; Springer; A grouping according to similarity has emerged Animal names and their attributes birds peaceful hunters is has likes to

Conclusion  Advantages SOM is Algorithm that projects high-dimensional data onto a two-dimensional map. The projection preserves the topology of the data so that similar data items will be mapped to nearby locations on the map. SOM still have many practical applications in pattern recognition, speech analysis, industrial and medical diagnostics, data mining  Disadvantages Large quantity of good quality representative training data required No generally accepted measure of ‘quality’ of a SOM e.g. Average quantization error (how well the data is classified)

Discussion topics  What is the main purpose of the SOM?  Do you know any example systems with SOM Algorithm?

References [Witten and Frank (1999)] Witten, I.H. and Frank, Eibe. Data Mining: Practical Machine Learning Tools and Techniques with Java Implementations. Morgan Kaufmann Publishers, San Francisco, CA, USA [Kohonen (1982)] Teuvo Kohonen. Self-organized formation of topologically correct feature maps. Biol. Cybernetics, volume 43, [Kohonen (1995)] Teuvo Kohonen. Self-Organizing Maps. Springer, Berlin, Germany [Vesanto (1999)] SOM-Based Data Visualization Methods, Intelligent Data Analysis, 3: [Kohonen et al (1996)] T. Kohonen, J. Hynninen, J. Kangas, and J. Laaksonen, "SOM PAK: The Self-Organizing Map program package, " Report A31, Helsinki University of Technology, Laboratory of Computer and Information Science, Jan [Vesanto et al (1999)] J. Vesanto, J. Himberg, E. Alhoniemi, J Parhankangas. Self- Organizing Map in Matlab: the SOM Toolbox. In Proceedings of the Matlab DSP Conference 1999, Espoo, Finland, pp , [Wong and Bergeron (1997)] Pak Chung Wong and R. Daniel Bergeron. 30 Years of Multidimensional Multivariate Visualization. In Gregory M. Nielson, Hans Hagan, and Heinrich Muller, editors, Scientific Visualization - Overviews, Methodologies and Techniques, pages 3-33, Los Alamitos, CA, IEEE Computer Society Press. [Honkela (1997)] T. Honkela, Self-Organizing Maps in Natural Language Processing, PhD Thesis, Helsinki, University of Technology, Espoo, Finland [SVG wiki] [Jost Schatzmann (2003)] Final Year Individual Project Report Using Self-Organizing Maps to Visualize Clusters and Trends in Multidimensional Datasets Imperial college London 19 June 2003