1. Introduction to Predictive Learning
LECTURE SET 6
Neural Network Learning
Electrical and Computer Engineering

2. OUTLINE Objectives - show examples using synthetic and real-life data
- introduce biologically inspired NN learning methods for clustering, regression and classification
- explain similarities and differences between statistical and NN methods
- show examples using synthetic and real-life data
Brief history and motivation for artificial neural networks
Sequential estimation of model parameters
Methods for supervised learning
Methods for unsupervised learning
Summary and discussion

3. Brief history and motivation for ANN
Huge interest in understanding the nature and mechanism of biological/ human learning
Biologists + psychologists do not adopt classical parametric statistical learning, because:
- parametric modeling is not biologically plausible
- biological info processing is clearly different from algorithmic models of computation
Mid 1980’s: growing interest in applying biologically inspired computational models to:
- developing computer models (of human brain)
- various engineering applications
 New field Artificial Neural Networks (~1986 – 1987)
ANN’s represent nonlinear estimators implementing the ERM approach (usually squared-loss function)

4. History and motivation (cont’d)
Relationship to the problem of inductive learning:
The same learning problem setting
Neural-style learning algorithm:
- on-line (flow through)
- simple processing
Biological terminology

5. Neural vs Algorithmic computation
Biological systems do not use principles of digital circuits
Digital Biological
Connectivity 1~10 ~10,000
Signal digital analog
Timing synchronous asynchronous
Signal propag. feedforward feedback
Redundancy no yes
Parallel proc. no yes
Learning no yes
Noise tolerance no yes

6. Neural vs Algorithmic computation
Computers excel at algorithmic tasks (well-posed mathematical problems)
Biological systems are superior to digital systems for ill-posed problems with noisy data
Example: object recognition [Hopfield, 1987]
PIGEON: ~ 10^^9 neurons, cycle time ~ 0.1 sec,
each neuron sends 2 bits to ~ 1K other neurons
 2x10^^13 bit operations per sec
OLD PC: ~ 10^^7 gates, cycle time 10^^-7, connectivity=2
 10x10^^14 bit operations per sec
Both have similar raw processing capability, but pigeons are better at recognition tasks

7. Neural terminology and artificial neurons
Some general descriptions of ANN’s:
McCulloch-Pitts neuron (1943)
Threshold (indicator) function of weighted sum of inputs

8. Goals of ANN’s
Develop models of computation inspired by biological systems
Study computational capabilities of networks of interconnected neurons
Apply these models to real-life applications
Learning in NNs = modification (adaptation) of synaptic connections (weights) in response to external inputs

9. Historical highlights of ANN
McCulloch-Pitts neuron
1949 Hebbian learning
1960’s Rosenblatt (perceptron), Widrow
60’s-70’s dominance of ‘hard’ AI
1980’s resurgence of interest (PDP group, MLP, SOM etc.)
1990’s connection to statistics/VC-theory
2000’s mature field/ fragmentation

10. OUTLINE
Objectives
Brief history and motivation for artificial neural networks
Sequential estimation of model parameters
Methods for supervised learning
Methods for unsupervised learning
Summary and Discussion

11. Sequential estimation of model parameters
Batch vs on-line (iterative) learning
- Algorithmic (statistical) approaches ~ batch
- Neural-network inspired methods ~ on-line
BUT the only difference is on the implementation level (so both types of methods should yield similar generalization)
Recall ERM inductive principle (for regression):
Assume dictionary parameterization with fixed basis fcts

12. Sequential (on-line) least squares minimization
Training pairs presented sequentially
On-line update equations for minimizing empirical risk (MSE) wrt parameters w are:
(gradient descent learning)
where the gradient is computed via the chain rule:
the learning rate is a small positive value (decreasing with k)

13. On-line least-squares minimization algorithm
Known as delta-rule (Widrow and Hoff, 1960):
Given initial parameter estimates w(0), update parameters during each presentation of k-th training sample x(k),y(k)
Step 1: forward pass computation
- estimated output
Step 2: backward pass computation
- error term (delta)

14. Neural network interpretation of delta rule
Forward pass Backward pass
Biological learning

15. Theoretical basis for on-line learning
Standard inductive learning: given training data find the model providing min of prediction risk
Stochastic Approximation guarantees minimization of risk (asymptotically):
under general conditions
on the learning rate:

16. Practical issues for on-line learning
Given finite training set (n samples):
this set is presented sequentially to a learning algorithm many times. Each presentation of n samples is called an epoch, and the process of repeated presentations is called recycling (of training data)
Learning rate schedule: initially set large, then slowly decreasing with k (iteration number). Typically ’good’ learning rate schedules are data-dependent.
Stopping conditions:
(1) monitor the gradient (i.e., stop when the gradient falls below some small threshold)
(2) early stopping can be used for complexity control

17. OUTLINE
Objectives
Brief history and motivation for artificial neural networks
Sequential estimation of model parameters
Methods for supervised learning
- MultiLayer Perceptron (MLP) networks
- Radial Basis Function (RBF) Networks
Methods for unsupervised learning
Summary and discussion

18. Multilayer Perceptrons (MLP)
Recall graphical NN
representation for
dictionary methods:
where
How to estimate parameters (weights) via ERM?

19. Learning for a single neuron (delta rule):
Forward pass Backward pass
How to implement gradient-descent learning in a network of neurons?

20. Backpropagation training
Minimization of
with respect to parameters (weights) W, V
Gradient descent optimization for
where
Careful application of gradient descent leads
leads to the backpropagation algorithm

21. Backpropagation: forward pass for training input x(k), estimate predicted output

22. Backpropagation: backward pass update the weights by propagating the error

23. Details of backpropagation
Sigmoid activation - picture?
simple derivative
 Poor behaviour for large t ~ saturation
How to avoid saturation?
- Proper initialization (small weights)
- Pre-scaling of inputs (zero mean, unit variance)
Learning rate schedule (initial, final)
Stopping rules, number of epochs
Number of hidden units

24. Regularization Effect of Backpropagation
Backpropagation ~ iterative optimization
Final model (weights) depends on:
- initial point + final point (stopping rules)
 initialization and/ or stopping rules can be used for model complexity control

25. Various forms of complexity control
MLP topology ~ number of hidden units
Constraints on parameters (weights) ~ weight decay
Type of optimization algorithm (many versions of backprop., other opt. methods)
Stopping rules
Initial conditions (initial ‘small’ weights)
Multiple factors make it difficult to control complexity; usually vary one complexity parameter while keeping all others fixed

26. Example: univariate regression
Data set: 30 samples generated using sine-squared target function with Gaussian noise (st. deviation 0.1).
MLP network
(two hidden units)
underfitting

27. Example: univariate regression
Data set: 30 samples generated using sine-squared target function with Gaussian noise (st. deviation 0.1).
MLP network
(five hidden units)
near optimal

28. Example: univariate regression
Data set: 30 samples generated using sine-squared target function with Gaussian noise (st. deviation 0.1).
MLP network
(20 hidden units)
little overfitting

29. Backpropagation for classification
Original MLP is for regression
(as shown)
For classification:
- sigmoid output unit (~ logistic regression using log-likelihood loss – see textbook)
- during training, use real-values 0/1 for class labels
- during operation, threshold the output of a trained MLP classifier at 0.5 to predict class labels

30. Classification example (Ripley’s data set)
Data set: 250 samples ~ mixture of gaussians, where Class 0 data has centers (-0.3, 0.7) and (0.4, 0.7), and Class 1 data has centers (-0.7, 0.3) and (0.3, 0.3). The variance of all gaussians is 0.03.
MLP classifier
(two hidden units)
underfitting

31. Classification Example
MLP classifier (three hidden units)
~ near optimal solution

32. Classification Example
MLP classifier (six hidden units)
some overfitting

33. MLP software MLP software widely available in public domain
Can handle multi-class problems
For example, Netlab toolbox (in Matlab) at
Many commercial products (full of marketing hype)
’Nearly 80% Accurate Market Forecasting Software Get FREE up to date predictions and see for yourself!’

34. NetTalk (Sejnowski and Rosenberg, 1987)
One of the first successful applications of backpropagation:
Goal: Learning to read (English text) aloud, i.e.
Learn Mapping: English text  phonemes
using MLP classifier network
Network inputs encode 7-letter window (the 4-th letter in the middle needs to be pronounced)
Network outputs (26 units) encode phonemes that drive a speech synthesizer
The MLP network is trained using labeled data (both individual words and unrestricted text)

35. NetTalk architecture
Input encoding: 7x29 = 203 units Output encoding: 26 units (phonemes) Hidden layer: 80 hidden units

36. Listening to NetTalk-generated speech
Listen to tape recordings illustrating NETtalk operation available on Youtube
These three recordings contain 3 different audio outputs of NETtalk:
(a) during the first 5 minutes of training, starting with weights initialized to zero.
(b) after training using the set of 10,000 words. This training set corresponds to 20 passes (epochs) over 500-word text.
(c) generated with new text input that was not part of the training set.
After listening to these recordings, answer and comment on the following questions:
- can you recognize words in the recording (a), (b) and (c)? – Explain why.
- compare the quality of outputs (b) and (c). Which one seems closer to human speech and why?
Question for discussion: Problem 6.8
- Why NETtalk uses a seven-letter window?

37. Radial Basis Function (RBF) Networks
Dictionary parameterization:
- each b.f. is (usually) local
- center and width
i.e. Gaussian:
Typically used for regression or classification

38. RBF network training RBF training (learning) ~ estimation of
(1) RBF parameters (centers, width)
(2) linear weights w’s
Non-adaptive implementation:
(1) Estimate RBF parameters via unsupervised learning (only x-values of training data) – can use SOM, GLA etc.
(2) Estimate weights w via linear least squares
Advantages:
- fast training;
- when x-samples are plenty, but (x,y) data are few
Limitations: cannot discard irrelevant inputs
the curse of dimensionalty

39. Non-adaptive RBF training algorithm
Choose the number of basis functions (centers) m.
Estimate centers using x-values of training data via unsupervised learning (SOM, GLA, clustering etc.)
Determine width parameters using heuristic:
For a given center
(a) find the distance to the closest center:
for all
(b) set the width parameter
where parameter controls degree of overlap between adjacent basis functions. Typically
4. Estimate weights w via linear least squares (minimization of the empirical risk MSE).

40. RBF network complexity control
RBF model complexity can be controlled by
The number of RBFs:
Goal: select opt number of units (RBFs)
RBF width:
Goal: select opt width parameter (for large number of RBF’s)
Penalization of large weights w’s
See toy examples next (using the number of units as the complexity parameter)

41. Example: RBF regression
Data set: 30 samples generated using sine-squared target function with Gaussian noise (st. deviation 0.1).
RBF network: automatic width selection (via x-validation)
2 RBF’s
underfitting

42. Example: RBF regression
Data set: 30 samples generated using sine-squared target function with Gaussian noise (st. deviation 0.1).
RBF network: automatic width selection
5 RBF’s
~ optimal

43. Example: RBF regression
Data set: 30 samples generated using sine-squared target function with Gaussian noise (st. deviation 0.1).
RBF network: automatic width selection
20 RBF’s
overfitting

44. RBF Classification example (Ripley’s data)
Data set: 250 samples ~ mixture of gaussians, where Class 0 data has centers (-0.3, 0.7) and (0.4, 0.7), and Class 1 data has centers (-0.7, 0.3) and (0.3, 0.3). The variance of all gaussians is 0.03.
RBF classifier
(4 units)
some underfitting

45. RBF Classification example (cont’d)
RBF classifier (9 units)
Optimal

46. RBF Classification example (cont’d)
RBF classifier (25 units)
Little overfitting

47. OUTLINE
Objectives
Brief history and motivation for artificial neural networks
Sequential estimation of model parameters
Methods for supervised learning
Methods for unsupervised learning
- clustering and vector quantization
- Self-Organizing Maps (SOM)
- Application example
Summary and discussion

48. Overview Recall from Lecture Set 2: unsupervised learning
data reduction approach
Example: Training data represented by 3 ‘centers’ H

49. Two types of problems 1. Data reduction: VQ + clustering
‘Model’ ~ m points
Vector Quantizer Q:
VQ setting: given n training samples
find the coordinates of m centers (prototypes) such that the total squared error distortion is minimized

50. Dimensionality reduction: linear nonlinear
‘Model’ ~ projection of high-dim. data onto low-dim. space.
Note: the goal is to estimate a mapping from d-dimensional input space (d=2) to low-dimensional feature space (d*=1)

51. Vector Quantization and Clustering
Two complementary goals of VQ:
1. partition the input space into disjoint regions
2. find positions of units (coordinates of prototypes)
Note: optimal partitioning into regions is according to the nearest-neighbor rule (~ the Voronoi regions)

52. Generalized Lloyd Algorithm(GLA) for VQ
Given data points , loss function L (i.e., squared loss) and initial centers
Perform the following updates upon presentation of
1. Find the nearest center to the data point (the winning unit):
2. Update the winning unit coordinates (only) via
Increment k and iterate steps (1) – (2) above
Note: - the learning rate decreases with iteration number k
- biological interpretations of steps (1)-(2) exist

53. Batch version of GLA
Given data points , loss function L (i.e., squared loss) and initial centers
Iterate the following two steps
1. Partition the data (assign sample to unit j ) using the nearest neighbor rule. Partitioning matrix Q:
2. Update unit coordinates as centroids of the data:
Note: final solution may depend on initialization (local min) – potential problem for both on-line and batch GLA

54. Numeric Example of univariate VQ
Given data: {2,4,10,12,3,20,30,11,25}, set m=2
Initialization (random): c1=3,c2=4
Iteration 1
Projection: P1={2,3} P2={4,10,12,20,30,11,25}
Expectation (averaging): c1=2.5, c2=16
Iteration 2
Projection: P1={2,3,4}, P2={10,12,20,30,11,25} Expectation(averaging): c1=3, c2=18
Iteration 3
Projection: P1={2,3,4,10},P2={12,20,30,11,25} Expectation(averaging): c1=4.75, c2=19.6
Iteration 4
Projection: P1={2,3,4,10,11,12}, P2={20,30,25} Expectation(averaging): c1=7, c2=25
Stop as the algorithm is stabilized with these values

55. GLA Example 1 Modeling doughnut distribution using 5 units
(a) initialization (b) final position (of units)

56. GLA Example 2 Modeling doughnut distribution using 3 units:
Bad initialization  poor local minimum

57. GLA Example 3 Modeling doughnut distribution using 20 units:
7 units were never moved by the GLA
 the problem of unused units (dead units)

58. Avoiding local minima with GLA
Starting with many random initializations, and then choosing the best GLA solution
Conscience mechanism: forcing ‘dead’ units to participate in competition, by keeping the frequency count (of past winnings) for each unit,
i.e. for on-line version of GLA in Step 1
Self-Organizing Map: introduce topological relationship (map), thus forcing the neighbors of the winning unit to move towards the data.

59. Clustering methods
Clustering: separating a data set into several groups (clusters) according to some measure of similarity
Goals of clustering:
interpretation (of resulting clusters)
exploratory data analysis
preprocessing for supervised learning
often the goal is not formally stated
VQ-style methods (GLA) often used for clustering, i.e. k-means or c-means
Many other clustering methods as well

60. Clustering (cont’d)
Clustering: partition a set of n objects (samples) into k disjoint groups, based on some similarity measure. Assumptions:
- similarity ~ distance metric dist (i,j)
- usually k given a priori (but not always!)
Intuitive motivation:
similar objects into one cluster
dissimilar objects into different clusters
 the goal is not formally stated
Similarity (distance) measure is critical
but usually hard to define (objectively). Distance needs to be defined for different types of input variables.

61. Self-Organizing Maps History and biological motivation
Brain changes its internal structure to reflect life experiences  interaction with environment is critical at early stages of brain development (first 1-2 years of life)
Existence of various regions (maps) in the brain
How these maps may be formed?
i.e. information-processing model leading to map formation
T. Kohonen (early 1980’s) proposed SOM

62. Goal of SOM
Dimensionality reduction: project given (high-dim.) data onto low-dimensional space (called a map)
Feature space (Z-space) is 1D or 2D and is discretized as a number of units, i.e., 10x10 map
Z-space has distance metric  ordering of units
Similarities and differences between VQ and SOM

63. Self-Organizing Map
Discretization of 2D space via 10x10 map. In this discrete
space, distance relations exist between all pairs of units. Distance relation ~ map topology

64. SOM Algorithm (flow through)
Given data points , distance metric in the input space (~ Euclidean), map topology (in z-space), initial position of units (in x-space)
Perform the following updates upon presentation of
1. Find the nearest unit to the data point (the winning unit denoted as z(k)):
2. Update all units around the winning unit via
Increment k, decrease the learning rate and the neighborhood width, and repeat steps (1) – (2) above

65. SOM example (one iteration)
Step 1:
Step 2:

66. SOM example (next iteration)
Step 1:
Step 2:
Final map

67. Hyper-parameters of SOM
SOM performance depends on parameters (~ user-defined):
Map dimension and topology (usually 1D or 2D)
Number of SOM units ~ quantization level (of z-space)
Neighborhood function ~ usually rectangular or gaussian (shape not important)
Neighborhood width decrease schedule (important), i.e. exponential decrease for Gaussian
with user defined:
Also linear decrease of neighborhood width
Learning rate schedule (important)
(also linear decrease)
Note: learning rate and neighborhood decrease should be set jointly

68. Modeling uniform distribution via SOM
(a) 300 random samples (b) 10X10 map
SOM neighborhood: Gaussian
Learning rate: linear decrease

69. Position of SOM units: (a) initial, (b) after 50 iterations, (c) after 100 iterations, (d) after 10,000 iterations

70. Batch SOM (similar to Batch GLA)
Given data points , distance metric (i.e., squared loss), map topology and initial centers
Iterate the following two steps
1. Partition the data into clusters using the minimum distance rule. This results in assignment of n samples to m clusters (units) according to assignment matrix Q
2. Update center coordinates as the weighted average of all data samples (in each cluster):
Decrease the neighborhood width, and iterate.

71. Example: effect of the final neighborhood width 90% 50%
10%

72. SOM Applications Two types of applications:
Vector Quantization
Clustering of multivariate data
Main web site:
Numerous Applications
Marketing surveys/ segmentation
Financial/ stock market data
Text data / document map – WEBSOM
Image data / picture map - PicSOM
see HUT web site

73. Practical Issues for SOM
Pre-scaling of inputs, usually to [0, 1] range. Why?
Map topology: usually 1D or 2D
Number of map units (per dimension)
Learning rate schedule (for on-line version)
Neighborhood type and schedule:
Initial size (~1), final size
Final neighborhood size + the number of units affect model complexity.

74. Modeling US states using 1D SOM (performed by Feng Cai)
Purpose: clustering of US states
Data encoding: each state described by 5 socio-economic indicators: obesity index, result of 2004 presidential elections, median income, mean NAEP, IQ score
Data scaling: each input scaled independently to [0,1] range
SOM specs: 1D map, 9 units, initial neighborhood width 1, final width 0.05

77. SOM Modeling 1 of US states

79. SOM Modeling 2 of US states - remove the voting input and apply 1D SOM:

80. SOM Modeling 2 of US states (cont’d) - remove voting input and apply 1D SOM:

81. Clustering of European Languages
Background: historical linguistics studies relatedness btwn languages based on
phonology, morphology, syntax and lexicon
Difficulty of the problem: due to evolving nature of human languages and globalization.
Hypothesis: similarity based on analysis of a small ‘stable’ word set.
See glottochronology, Swadesh list, at

82. SOM Clustering of European Languages
Modeling approach: language ~ 10 word set.
Assuming words in different languages are encoded in the same alphabet, it is possible to perform clustering using some distance measure.
Issues:
selection of a stable word set
data encoding + distance metric
Stable word set: numbers 1 to 10
Data encoding: Latin alphabet, use 3 first letters (in each word)

83. Numbers word set in 18 European languages
Each language is a feature vector encoding 10 words

84. Data Encoding Word ~ feature vector encoding 3 first letters
Alphabet ~ 26 letters + 1 symbol ‘BLANK’
vector encoding:
For example, ONE : ‘O’~14 ‘N’~15 ‘E’~05

85. Word Encoding (cont’d)
Word  27-dimensional feature vector
Encoding is insensitive to order (of 3 letters)
Encoding of 10-word set: concatenate feature vectors of all words: ‘one’ + ‘two’ + …+ ‘ten’
 word set encoded as vector of dim. [1 X 270]

86. SOM Modeling Approach 2-Dimensional SOM (Batch Algorithm)
Number of Units per dimension=4
Initial Neighborhood =1 Final Neighborhood = 0.15
Total Number of Iterations= 70

87. OUTLINE
Objectives
Brief history and motivation for artificial neural networks
Sequential estimation of model parameters
Methods for supervised learning
Methods for unsupervised learning
Summary and discussion

88. Summary and Discussion
Neural Network methods (vs statistical approaches):
- new techniques/ grad descent style methods
- simple (brute-force) computational approaches
- black-box models (e.g. MLP network)
- biological motivation
The same fundamental issues: small-sample problems, curse-of-dimensionality, non-linear optimization, complexity control
Neural network methods implement ERM or SRM approach (under predictive learning setting)
Hype and controversy