ARTIFICIAL NEURAL NETWORKS

Overview EdGeneral concepts Areej:Learning and Training Wesley:Limitations and optimization of ANNs Cora:Applications and examples

Biological Inspiration

Neuron

Activation Function

Errors Is proportional to

Layers – Famous Problem

Layers – Building Blocks

Layers - Idea

Layers – Solution P1

Layers – Solution P2 3 layers“feedforward”

LEARNING AND TRAINING

Learning methods Supervised learning. Learn to predict an output when given an input. Unsupervised learning. Discover a good internal representation of the input.

Types of Artificial Neural Networks Neural network topologies can vary widely, resulting in differences in architecture, function, and behaviour. Feed- forward ANN. Recurrent ANN.

Feed- forward Neural Networks It is the first and the simplest type of ANNs. This type of organisation is also referred to as bottom-up or top-down. It allows signals to travel in one direction only. From input to output through the hidden layers if there were any. There are no feedback (loops or circles). The output of any layer does not affect that same layer.

Teaching process of multilayer ANN Image source [2]

Backpropagation algorithm Image source [2]

Backpropagation algorithm Image source [2]

Backpropagation algorithm Image source [2]

Backpropagation algorithm Image source [2]

Backpropagation algorithm Image source [2]

Backpropagation algorithm Image source [2]

Backpropagation algorithm Image source [2]

Backpropagation algorithm Image source [2]

Backpropagation algorithm Image source [2]

Backpropagation algorithm Image source [2]

Backpropagation algorithm Image source [2]

Backpropagation algorithm Image source [2]

Recurrent Neural Networks It can be called as feed- back or interactive networks. It have signals traveling in both directions (loops and circles). Feedback networks are powerful and can get extremely complicated. Computations derived from earlier input are fed back into the network, which gives them a kind of memory.

LIMITATIONS & OPTIMISATIONS

Overfitting When performance on training data is high but performance on unseen data is poor. Effectively memorizing training data without learning generalized rules.

Split-sample testing Available test data is split into training set, validation set, and test set. Final weights are those used at stopping point. Image source: [1]

Regularization Reduces overfitting by adjusting the training process. Lx regularization adds a regularization term to the cost function. L2 regularization or weight decay: λ is the regularization parameter. Tends towards smaller overall weights. Less sensitive to noise.

Regularization (cont’) L1 regularization: Tends to maintain most useful weights and shrink others.

Regularization (cont’) Dropout involves training with only a subset of neurons active. For each training run, half of the neurons are randomly chosen to be active, the others unused. Final network is less dependent on any particular input or neuron. More testing data is better for generalization, but expensive. Training data can be artificially increased by adjusting existing data.

Learning slowdown Using Mean-Squared-Error for the cost function can result in very slow learning when the derivative of the output is small. Learning can be initially slow if weight initialisation is far from correct.

Learning slowdown (cont’) The Cross-Entropy cost function eliminates the output derivative from the weight gradients. Weights learn solely based on output value and not slope of output when using sigmoid functions.

Learning slowdown (cont’) Alternative cost function is log-likelihood function combined with softmax output layer. Output represents a probability distribution. Useful for classifiers.

Normalization Scales data so that each input lies in same range, reducing influence of one input feature over another. Speeds up training when the process starts with all input features on same scale. If applied to input in training set then must subsequently be applied to all future input.

Normalization (cont’) ●Z-Score or Gaussian normalization scales by mean (μ) and standard deviation (σ). Min-Max normalization scales input to lie in a fixed range. Sigmoidal normalization reduces the effects of outliers in the input.

Vanishing Gradient Problem Increasing number of hidden layers reduces required complexity of network but makes training harder. Vanishing gradient problem is where the gradient of weights in of early layers are small in multi-layer networks compared to later layers and thus learn slower. Alternative is exploding gradient problem. Unstable gradient problem can be avoided with pre- training. Layers are trained one at a time to produce a compressed form of the input.

APPLICATIONS & EXAMPLES

Application Areas 1. Function approximation 2. Pattern recognition 3.Database mining Problems 1. Car insurance policy – claim frequency 2. Door security -- Fingerprint 3. Medical data mining -- Diagnosing lung cancer

Structure Image source:[1]

Image source:[8] Car insurance policy –claim frequency

Door security--fingerprint Image source:[9]

Image source:[10]

References [1] L. L. Priddy and P.E. Keller (2005). Artificial Neural Networks: An introduction. Vol. 68: SPIE press. [2] M. Bernacki. (2005) Principles of training multi-layer neural network using backpropagation. Available: [3] S. Haykin and N. Network, "A comprehensive foundation," Neural Networks, vol. 2, [4] M. James. (2012), The Triumph of Deep Learning. Available: intelligence/5206-the-triumph-of-deep-learning.html [5] J. McCaffrey. (2014), How To Standardize Data for Neural Networks. Visual Studio Magazine. Available: visualstudiomagazine.com/articles/2014/01/01/how-to-standardize-data-for-neural-networks.aspx [6] M. Nielsen. (2015). Neural Networks and Deep Learning. Available: neuralnetworksanddeeplearning.com/ [7] W. Wu, H. Maier, G. Dandy, and R. May, "Exploring the impact of data splitting methods on artificial neural network models," in International Conference on Hydroinformatics (10th: 2012: Hamburg, Germany), [8] [9] [10]

Thank you