1. Artificial Intelligence Methods
Neural Networks
Lecture 2
Rakesh K. Bissoondeeal

2. ANNs Architectures Architecture
- layers, connections, activation (transfer) functions
Layers –ANNs are usually thought as arranged in layers: single and multilayer
single layer 1 X1 X2 Y

3. ANNs Architectures Multilayer ANNs -In addition to input and output
layers
- hidden layer(s)
- allow(s) the ANN to
learn nonlinear
relationships

4. ANNs Architectures Single layer network - learning is simpler
- limited in the tasks it can do
Multilayer network
-Learning is more difficult
-can solve more complicated problems
e.g network with hidden layer can approximate any continuous function

5. ANNs Architectures Optimal number of hidden layers
-depends on the problem
e.g. function approximation
-network with 1 hidden layer can approximate any continuous function (well known and commonly used)
-2 or more can be beneficial to certain problems but number of parameters (weights) increases; problematic with small data samples

6. ANNs Architectures Optimal number of neurons (nodes)
- input nodes – input variables
- depends on analysis, theory may help
- hidden nodes – nodes in hidden layer
- too many nodes – overfitting, i.e., networks perform well within sample, but poor out-of-sample performance
- too little – difficulty to learn pattern in the data
- output nodes – output
- easiest to choose
- depends on the problem

7. ANNs Architectures Connections -Feedforward and feedback
Feedforward networks:
- no link backwards
- each node is connected to nodes in the next layer
- no links between nodes in same layer
- no links skip a layer
- simpler, proven most useful

8. ANNs Architectures Connections - Feedback networks
- feedback from output to input units
- Complex dynamics – learning is more difficult in feedback network
- E.g Hopfield network – used for associative memory
- Train network with a set of pictures, then present a piece of one pictures, network will produce the picture from which the piece of picture taken

9. ANNs Architectures Activation functions
Usually same function used for all neurons in the same layer
Linear and nonlinear activation functions
- Linear functions, usually f(x)=x

10. ANNs Architectures Activation functions
- nonlinear activation functions
- step function
- sign function
- logistic sigmoid function
- hyperbolic tangent function

11. ANNs Architecture
Nonlinear functions
sign
step

12. ANNs Architectures
Nonlinear activation function : Logistic Sigmoid

13. ANN Architectures
Nonlinear activation function: hyperbolic tangent

14. Learning (Training) How do ANNs learn?
-McCulloch-Pitt Neuron has no concept of learning
- Learning is achieved by modifying the weights in the network
- Weights are initially randomly selected
Two types of Learning
- Supervised
- Unsupervised

15. Learning (Training) Supervised learning
- Bulk of networks use supervised training
- Both inputs and outputs provided
(p1,t1), (p2,t2), . . ., (pn, tn)
- Network processes inputs and compares resulting output with desired output
- depending on the size the error (desired output-network output) the weights are adjusted using an appropriate learning algorithm

16. Learning (Training) Supervised learning
-Process repeated until the error is considered to be small enough.
Some cases error = 0
Some cases error cannot be O, avoid overtraining
-Training set, validation set, test set
-Examples, Backpropagation, Quasi-Newton, Levernberg-Marqualt

17. Learning (Training) Unsupervised learning
- The network is provided with inputs but not the desired output
- Often used for finding the patterns in the data rather than for modelling input/output mapping
- The network must itself decide what features to use to group the input data
-Not very well understood

18. Recommended Reading
Fundamentals of neural networks; Architectures, Algorithms and Applications, L. Fausett, 1994.
Artificial Intelligence: A Modern Approach, S. Russel and P. Norvig, 1995.
An Introduction to Neural Networks. 2nd Edition, Morton, IM.