1. Machine Learning
Neural Networks (2)

2. Accuracy
Easily the most common and intuitive measure of classification performance.

3. Loss/Error Functions How do we measure performance? Regression:
L2 error
Classification:
#misclassifications
Weighted misclassification via a cost matrix
For 2-class classification:
True Positive, False Positive, True Negative, False Negative
For k-class classification:
Confusion Matrix

4. F-Measure
True Values
Positive
Negative
Hyp Values 10 50 40

5. Types of Errors False Positives
The system predicted TRUE but the value was FALSE
aka “False Alarms” or Type I error
False Negatives
The system predicted FALSE but the value was TRUE
aka “Misses” or Type II error

6. Basic Evaluation
Training data – used to identify model parameters
Testing data – used for evaluation
Optionally: Development / tuning data – used to identify model hyperparameters.
Difficult to get significance or confidence values

7. Cross validation Identify n “folds” of the available data.
Train on n-1 folds
Test on the remaining fold.
In the extreme (n=N) this is known as “leave-one-out” cross validation
n-fold cross validation (xval) gives n samples of the performance of the classifier.

8. Cross-validation visualized
Available Labeled Data
Identify n partitions
Train
Train
Train
Train
Dev
Test
Fold 1

9. Cross-validation visualized
Available Labeled Data
Identify n partitions
Test
Train
Train
Train
Train
Dev
Fold 2

10. Cross-validation visualized
Available Labeled Data
Identify n partitions
Dev
Test
Train
Fold 3

11. Cross-validation visualized
Available Labeled Data
Identify n partitions
Train
Dev
Test
Train
Fold 4

12. Cross-validation visualized
Available Labeled Data
Identify n partitions
Train
Train
Dev
Test
Train
Fold 5

13. Cross-validation visualized
Available Labeled Data
Identify n partitions
Train
Train
Train
Dev
Test
Train
Fold 6
Calculate Average Performance

14. Concepts Capacity Measure how large hypothesis class H is.
Are all functions allowed?
Overfitting
f works well on training data
Works poorly on test data
Generalization
The ability to achieve low error on new test data

15. Multi-layers Network Let the network of 3 layers Input layer
Hidden layer
Output layer
Each layer has different number of neurons
The famous example to need the multi-layer network is XOR unction
The perceptron learning rule can not be applied to multi-layer network
We use BackPropagation Algorithm in learning process

20. Feed-forward + Backpropagation
input from the features is fed forward in the network from input layer towards the output layer
Backpropagation:
Method to asses the blame of errors to weights
error rate flows backwards from the output layer to the input layer (to adjust the weight in order to minimize the output error)

21. Backprop Back-propagation training algorithm illustrated:
Backprop adjusts the weights of the NN in order to minimize the network total mean squared error.
Network activation
Error computation
Forward Step
Error propagation
Backward Step

22. Correlation Learning Hebbian Learning (1949):
“When an axon of cell A is near enough to excite a cell B and repeatedly or persistently takes place in firing it, some growth process or metabolic change takes place in one or both cells such that A’s efficiency, as one of the cells firing B, is increased.”
Weight modification rule:
wi,j = cxixj
Eventually, the connection strength will reflect the correlation between the neurons’ outputs.

23. Backpropagation Learning
the goal of the Backpropagation learning algorithm is to modify the network’s weights so that its output vector
op = (op,1, op,2, …, op,K)
is as close as possible to the desired output vector
dp = (dp,1, dp,2, …, dp,K)
for K output neurons and input patterns p = 1, …, P.
The set of input-output pairs (exemplars) {(xp, dp) | p = 1, …, P} constitutes the training set.

24. The weight change rule is Where  is the learning factor <1
Bp Algorithm
The weight change rule is
Where  is the learning factor <1
Error is the difference between actual and trained value
f’ is is the derivative of sigmoid function = f(1-f)

25. Delta Rule
Each observation contributes a variable amount to the output
The scale of the contribution depends on the input
Output errors can be blamed on the weights
A least mean square (LSM) error function can be defined (ideally it should be zero)
E = ½ (t – y)2

26. Example
For the network with one neuron in input layer and one neuron in hidden layer the following values are given
X=1, w1 =1, b1=-2, w2=1, b2 =1, =1 and t=1
Where X is the input value
W1 is the weight connect input to hidden
W2 is the weight connect hidden to output
B1 and b2 are bias
T is the training value

27. Exercises
Design a neural network to recognize the problem of
X1=[2 2] , t1=0
X=[1 -2], t2=1
X3=[-2 2], t3=0
X4=[-1 1], t4=1
Start with initial weights w=[0 0] and bias =0

28. Exercises
Perform one iteration of backprpgation to network of two layers. First layer has one neuron with weight 1 and bias –2. The transfer function in first layer is f=n2
The second layer has only one neuron with weight 1 and bias 1. The f in second layer is 1/n.
The input to the network is x=1 and t=1

29. Neural Network Construct a neural network to solve the problem X1 X2
Output
1.0 1
9.4
6.4 -1
2.5
2.1
8.0
7.7
0.5
2.2
7.9
8.4
7.0
2.8
0.8
1.2
3.0
7.8
6.1
Initialize the weights 0.75 , 0.5, and –0.6

30. Neural Network Construct a neural network to solve the XOR problem X1
Output 1
Initialize the weights –7.0 , -7.0, -5.0 and –4.0

31. Example -0.5 The transfer function is linear function. -2 1 1 1 -1 -13
0.5
-0.5

32. Example
Consider a transfer function as f(n) = n2. Perform one iteration of BackPropagation with a= 0.9 for neural network of two neurons in input layer and one neuron in output layer. The input values are X=[1 -1] and t = 8, the weight values between input and hidden layer are w11 = 1, w12 = - 2, w21 = 0.2, and w22 = 0.1. The weight between input and output layers are w1 = 2 and w2= -2. The bias in input layers are b1 = -1, and b2= 3.

33. Some of the possible solutions to this problem are:
Some variations
learning rate (). If  is too small then learning is very slow. If large, then the system's learning may never converge.
Some of the possible solutions to this problem are:
Add a momentum term to allow a large learning rate.
Use a different activation function
Use a different error function
Use an adaptive learning rate
Use a good weight initialization procedure.
Use a different minimization procedure

34. Problems with Local Minima
backpropagation
Can find its ways into local minima
One partial solution:
Random re-start: learn lots of networks
Starting with different random weight settings
Can take best network
Or can set up a “committee” of networks to categorise examples
Another partial solution: Momentum

35. Adding Momentum Gets stuck here Imagine rolling a ball down a hill
Without Momentum With Momentum
Gets stuck
here

36. Momentum in Backpropagation
For each weight
Remember what was added in the previous epoch
In the current epoch
Add on a small amount of the previous Δ
The amount is determined by
The momentum parameter, denoted α
α is taken to be between 0 and 1

37.  wij (n+1) =  (pj opi) +   wij(n)
Momentum
Weight update becomes:
 wij (n+1) =  (pj opi) +   wij(n)
The momentum parameter  is chosen between 0 and 1, typically 0.9. This allows one to use higher learning rates. The momentum term filters out high frequency oscillations on the error surface.

38. Problems with Overfitting
Plot training example error versus test example error:
Test set error is increasing!
Network is overfitting the data
Learning idiosyncrasies in data, not general principles
Big problem in Machine Learning (ANNs in particular)

39. Avoiding Overfitting
Bad idea to use training set accuracy to terminate
One alternative: Use a validation set
Hold back some of the training set during training
Like a miniature test set (not used to train weights at all)
If the validation set error stops decreasing, but the training set error continues decreasing
Then it’s likely that overfitting has started to occur, so stop
Another alternative: use a weight decay factor
Take a small amount off every weight after each epoch
Networks with smaller weights aren’t as highly fine tuned (overfit)

40. Feedback NN Recurrent Neural Networks

41. Recurrent Neural Networks
Can have arbitrary topologies
Can model systems with internal states (dynamic ones)
Delays are associated to a specific weight
Training is more difficult 1 1 1 x1 x2

42. Recurrent neural networks
Feedback as well as feedforward connections
Allow preservation of information over time
Demonstrated capacity to learn sequential behaviors

43. Recurrent neural networks
Architectures
Totally recurrent networks
Partially recurrent networks
Dynamics of recurrent networks
Continuous time dynamics
Discrete time dynamics
Associative memories
Solving optimization problems

44. Input-Output Recurrent Model
Input-Output Recurrent Model → nonlinear autoregressive with exogeneous inputs model (NARX) y(n+1) = F(y(n),...,y(n-q+1),u(n),...,u(n-q+1))
The model has a single input. It has a single output that is fed back to the input.
The present value of the model input is denoted u(n), and the corresponding value of the model output is denoted by y(n+1).

45. Recurrent multilayer perceptron (RMLP)
It has one or more hidden layers. Each computation layer of an RMLP has feedback around it.
xI(n+1) =I(xI(n),u(n))
xII(n+1) =II(xII(n),xI(n+1)), ...,
xO(n+1) =O(xO(n), xK(n))

46. The equivalence between layered, feedforward nets and recurrent nets
w w2
time=3
w w2
w3 w4
w w4
time=2
w w2
w3 w4
Assume that there is a time delay of 1 in using each connection.
The recurrent net is just a layered net that keeps reusing the same weights.
time=1
w w2
w3 w4
time=0

47. Recurrent Neural Networks : Hopfield Network
Proper when exact binary representations are possible.
Can be used as an associative memory or to solve optimization problems.
The number of classes (M) must be kept smaller than 0.15 times the number of nodes (N).

48. . . . . . Hopfield NN OUTPUTS(Valid After Convergence)
x’0
x’1
x’N-2
x’N-1
xN-2
xN-1 x0 x1
INPUTS(Applied At Time Zero)

49. Recurrent Neural Networks : Hopfield Network Algorithm
Step 1 : Assign Connection Weights.
Step 2 : Initialize with unknown input pattern.

50. Recurrent Neural Networks :Hopfield Network Algorithm
Step 3 : Iterate until convergence.
Step 4 : goto step 2. 1 -1

51. Example
Illustrate your understanding of the Recurrent back propagation Neural Networks by explicitly showing all steps of the calculations with a Sigmoidal nonlinearity and =0.8 for neural network blow. The input values are X=[1 1] and t = 8, the initial weight values are w1=1, w2=-1, w3=1, w4=1, w5 =2, and w6= -2. Show all the calculations for ONE iteration. Show the weight values at the end of the first iteration?

52. The illustrated Simple Recurrent Neural Network has two neurons
The illustrated Simple Recurrent Neural Network has two neurons. All neurons have sigmoid function
. The network ues the standard error function E =
using the initial weights [b1=-0.5, w1=2,b2=0.5 and w2=0.5] and let the input = 2,  = 1 and t = 5. Perform two iterations of recurrent back-propagation algorithm.