1. Chapter 9
Neural Network

2. Chapter Objective
Understand how feed-forward networks are used to solve estimation problems.
Know how input and output data conversions are performed for neural networks.
Understand how feed-forward neural networks learn through backpropagation.
Know how genetic learning is applied to train feed-forward neural networks.
Know how self-organizing neural networks perform unsupervised clustering.
List the strengths and weaknesses of neural networks.
Chapter 9

3. Feed-Forward Neural Network
Chapter 9

4. Feed-Forward Neural Network
Chapter 9

5. Neural Network Training: A Conceptual View
Chapter 9

6. Neural Network Training: A Conceptual View
Chapter 9

7. Neural Network Training: A Conceptual View
Chapter 9

8. Neural Network Explanation
Sensitivity analysis is a technique that has been successfully applied to gain insight into the effect individual attributes have on neural network output.
The general process consists of the following steps:
1. Divide the data into a training set and a test dataset.
2. Train the network with the training data.
Chapter 9

9. Neural Network Explanation
3. Use the test set data to create a new instance I Each attribute value for I is the average of all attribute values within the test data.
4. For each attribute:
a. Vary the attribute value within instance I and present the modification of I to the network for classification.
b. Determine the effect the variations have on the output of the neural network.
c. The relative importance of each attribute is measured by the effect of attribute variations on network output.
Chapter 9

10. General Considerations
The following is a partial list of choices that affect the performance of a neural network model:
What input attributes will be used to build the network?
How will the network output be represented?
How many hidden layers should the network contain?
How many nodes should there be in each hidden layer?
What condition will terminate network training?
Chapter 9

11. Neural Network Training: A Detailed View
Chapter 9

12. Neural Networks Advantages Criticism
prediction accuracy is generally high
robust, works when training examples contain errors
output may be discrete, real-valued, or a vector of several discrete or real-valued attributes
fast evaluation of the learned target function
Criticism
long training time
difficult to understand the learned function (weights)
not easy to incorporate domain knowledge
Chapter 9

13. - f A Neuron mk å weighted sum Input vector x output y Activation
function
weight
vector w å w0 w1 wn x0 x1 xn
The n-dimensional input vector x is mapped into variable y by means of the scalar product and a nonlinear function mapping
Chapter 9

14. Network Training The ultimate objective of training Steps
obtain a set of weights that makes almost all the tuples in the training data classified correctly
Steps
Initialize weights with random values
Feed the input tuples into the network..... one by one
For each unit
Compute the net input to the unit as a linear combination of all the inputs to the unit
Compute the output value using the activation function
Compute the error
Update the weights and the bias
Chapter 9

15. Multi-Layer Perceptron
Output vector
Output nodes
Hidden nodes
wij
Input nodes
Input vector: xi
Chapter 9

16. Network Pruning and Rule Extraction
Fully connected network will be hard to articulate
N input nodes, h hidden nodes and m output nodes lead to h(m+N) weights
Pruning: Remove some of the links without affecting classification accuracy of the network
Chapter 9

17. Network Pruning and Rule Extraction
Extracting rules from a trained network
Discretize activation values; replace individual activation value by the cluster average maintaining the network accuracy
Enumerate the output from the discretized activation values to find rules between activation value and output
Find the relationship between the input and activation value
Combine the above two to have rules relating the output to input
Chapter 9

18. Chapter Summary
A neural network is parallel computing system of several interconnected processor nodes.
The input to individual network nodes is restricted to numeric values falling in the closed interval range [0,1].
Because of this, categorical data must be transformed prior to network training.
Chapter 9

19. Chapter Summary
Developing a neural network involves first training the network to carry out the desired computations and then applying the trained network to solve new problems.
During the learning phase, training data is used to modify the connection weights between pairs of nodes so as to obtain a best result for the output node (s).
The feed-forward neural network architecture is commonly used for supervised learning.
Feed-forward neural networks contain a set of layered nodes and weighted connections between nodes in adjacent layers.
Chapter 9

20. Chapter Summary
Feed-forward neural networks are often trained using a backpropagation learning scheme.
Backpropagation learning works by making modifications in weight values starting at the output layer then moving backward through the hidden layers of the network.
Genetic learning can also be applied to train feed-forward networks.
Chapter 9

21. Chapter Summary
The self-organizing Kohonen neural network architecture is a popular model for unsupervised clustering.
A self-organizing neural network learns by having several output nodes complete for the training instances.
For each instance, the output node whose weight vectors most closely match the attribute values of the input instance is the winning node.
Chapter 9

22. Chapter Summary
As a result, the winning node has its associated input weights modified to more closely match the current training instance.
When unsupervised learning is complete, output nodes winning the most instances are saved.
After this, test data is applied and the clusters formed by the test set data are analyzed to help determine the meaning of what has been found.
Chapter 9

23. Chapter Summary
A central issue surrounding neural network is their inability to explain what has been learned.
Despite this, neural network have been successfully applied to solve problems in both the business and scientific world.
Although we have discussed the most popular neural network models, several other architectures and learning rules have been developed.
Jain, Mao, and Mohiuddin (1996) provide a good starting point for learning more about neural networks.
Chapter 9

24. Key Terms
Average member technique. An unsupervised clustering neural network explanation technique where the most typical member of each cluster is computed by finding the average value for each class attribute.
Backpropagation learning. A training method used with many feed-forward networks that works by making modifications in weight values starting at the output layer then moving backward through the hidden layer.
Delta rule. A neural network learning rule designed to minimize the sum of squared errors between computed and target network output.
Chapter 9

25. Key Terms
Epoch. One complete pass of the training data through a neural network.
Feed-forward neural network. A neural network architecture where all weights at one layer are directed toward nodes at the next network layer. Weights do not cycle back as inputs to previous layers.
Fully connected. A neural network structure where all nodes at one layer of the network are connected to all nodes in the next layer.
Kohonen network. A two-layer neural network used for unsupervised clustering.
Chapter 9

26. Key Terms
Neural network. A parallel computing system consisting of several interconnected processors.
Neurode. A neural network processor node. Several neurodes are connected to form a complete neural network structure.
Sensitivity analysis. A neural network explanation technique that allows us to determine a rank ordering for the relative importance of individual attributes.
Sigmoid function. One of several commonly used neural network evaluation functions. The sigmoid function is continuous and outputs a value between 0 or 1.
Chapter 9

27. Key Terms
Linearly separable. Two classes, A and B, are said to be linearly separable if a straight line can be drawn to separate the instances of class B.
Perceptron neural network. A simple feed-forward neural network architecture consisting of an input layer and a single output layer.
Chapter 9

28. Reference
Data Mining: Concepts and Techniques (Chapter 7 Slide for textbook), Jiawei Han and Micheline Kamber, Intelligent Database Systems Research Lab, School of Computing Science, Simon Fraser University, Canada
Chapter 9