1. Lecture Notes for Chapter 4 Artificial Neural Networks
Data Mining
Lecture Notes for Chapter 4
Artificial Neural Networks
Introduction to Data Mining , 2nd Edition by
Tan, Steinbach, Karpatne, Kumar

2. Artificial Neural Networks (ANN)
Output Y is 1 if at least two of the three inputs are equal to 1.

3. Artificial Neural Networks (ANN)

4. Artificial Neural Networks (ANN)
Model is an assembly of inter-connected nodes and weighted links
Output node sums up each of its input value according to the weights of its links
Compare output node against some threshold t
Perceptron Model

5. General Structure of ANN
Training ANN means learning the weights of the neurons

6. Artificial Neural Networks (ANN)
Various types of neural network topology
single-layered network (perceptron) versus multi-layered network
Feed-forward versus recurrent network
Various types of activation functions (f)

7. Perceptron Single layer network Contains only input and output nodes
Activation function: f = sign(wx)
Applying model is straightforward
X1 = 1, X2 = 0, X3 =1 => y = sign(0.2) = 1

8. Perceptron Learning Rule
Initialize the weights (w0, w1, …, wd)
Repeat
For each training example (xi, yi)
Compute f(w, xi)
Update the weights:
Until stopping condition is met

9. Perceptron Learning Rule
Weight update formula:
Intuition:
Update weight based on error:
If y=f(x,w), e=0: no update needed
If y>f(x,w), e=2: weight must be increased so that f(x,w) will increase
If y<f(x,w), e=-2: weight must be decreased so that f(x,w) will decrease

10. Example of Perceptron Learning

11. Perceptron Learning Rule
Since f(w,x) is a linear combination of input variables, decision boundary is linear
For nonlinearly separable problems, perceptron learning algorithm will fail because no linear hyperplane can separate the data perfectly

12. Nonlinearly Separable Data
XOR Data

13. Multilayer Neural Network
Hidden layers
intermediary layers between input & output layers
More general activation functions (sigmoid, linear, etc)

14. Multi-layer Neural Network
Multi-layer neural network can solve any type of classification task involving nonlinear decision surfaces
XOR Data

15. Learning Multi-layer Neural Network
Can we apply perceptron learning rule to each node, including hidden nodes?
Perceptron learning rule computes error term e = y-f(w,x) and updates weights accordingly
Problem: how to determine the true value of y for hidden nodes?
Approximate error in hidden nodes by error in the output nodes
Problem:
Not clear how adjustment in the hidden nodes affect overall error
No guarantee of convergence to optimal solution

16. Gradient Descent for Multilayer NN
Weight update:
Error function:
Activation function f must be differentiable
For sigmoid function:
Stochastic gradient descent (update the weight immediately)

17. Gradient Descent for MultiLayer NN
For output neurons, weight update formula is the same as before (gradient descent for perceptron)
For hidden neurons:

18. Design Issues in ANN Number of nodes in input layer
One input node per binary/continuous attribute
k or log2 k nodes for each categorical attribute with k values
Number of nodes in output layer
One output for binary class problem
k or log2 k nodes for k-class problem
Number of nodes in hidden layer
Initial weights and biases

19. Characteristics of ANN
Multilayer ANN are universal approximators but could suffer from overfitting if the network is too large
Gradient descent may converge to local minimum
Model building can be very time consuming, but testing can be very fast
Can handle redundant attributes because weights are automatically learnt
Sensitive to noise in training data
Difficult to handle missing attributes

20. Recent Noteworthy Developments in ANN
Use in deep learning and unsupervised feature learning
Seek to automatically learn a good representation of the input from unlabeled data
Google Brain project
Learned the concept of a ‘cat’ by looking at unlabeled pictures from YouTube
One billion connection network