1 Artificial Neural Networks Sanun Srisuk EECP0720 Expert Systems – Artificial Neural Networks

2 Introduction Artificial neural networks (ANNs) provide a general, practical method for learning real-valued, discrete- valued, and vector-valued functions from examples. Algorithms such as BACKPROPAGATION use gradient descent to tune network parameters to best fit a training set of input-output pairs. ANN learning is robust to errors in the training data and has been successfully applied to problems such as face recognition/detection, speech recognition, and learning robot control strategies. EECP0720 Expert Systems – Artificial Neural Networks

3 Autonomous Vehicle Steering EECP0720 Expert Systems – Artificial Neural Networks

4 Characteristics of ANNs Instances are represented by many attribute-value pairs. The target function output may be discrete-valued, real- valued, or a vector of several real- or discrete-valued attributes. The training examples may contain errors. Long training times are acceptable. Fast evaluation of the learned target function may be required. The ability of humans to understand the learned target function is not important. EECP0720 Expert Systems – Artificial Neural Networks

5 One type of ANN system is based on a unit called a perceptron. The perceptron function can sometimes be written as The space H of candidate hypotheses considered in perceptron learning is the set of all possible real-valued weight vectors. Perceptrons EECP0720 Expert Systems – Artificial Neural Networks

6 Representational Power of Perceptrons EECP0720 Expert Systems – Artificial Neural Networks

7 Decision surface EECP0720 Expert Systems – Artificial Neural Networks linear decision surfacenonlinear decision surface Programming Example of Decision Surface

8 The Perceptron Training Rule One way to learn an acceptable weight vector is to begin with random weights, then iteratively apply the perceptron to each training example, modifying the perceptron weights whenever it misclassifies an example. This process is repeated, iterating through the training examples as many times as needed until the perceptron classifies all training examples correctly. Weights are modified at each step according to the perceptron training rule, which revises the weight associated with input according to the rule EECP0720 Expert Systems – Artificial Neural Networks

9 Gradient Descent and Delta Rule The delta training rule is best understood by considering the task of training an unthresholded perceptron; that is, a linear unit for which the output o is given by In order to derive a weight learning rule for linear units, let us begin by specifying a measure for the training error of a hypothesis (weight vector), relative to the training examples. EECP0720 Expert Systems – Artificial Neural Networks

10 Visualizing the Hypothesis Space EECP0720 Expert Systems – Artificial Neural Networks minimum error initial weight vector by random

11 Derivation of the Gradient Descent Rule The vector derivative is called the gradient of E with respect to, written The gradient specifies the direction that produces the steepest increase in E. The negative of this vector therefore gives the direction of steepest decrease. The training rule for gradient descent is EECP0720 Expert Systems – Artificial Neural Networks

12 Derivation of the Gradient Descent Rule (cont.) The negative sign is presented because we want to move the weight vector in the direction that decreases E. This training rule can also written in its component form which makes it clear that steepest descent is achieved by altering each component of in proportion to. EECP0720 Expert Systems – Artificial Neural Networks

13 The vector of derivatives that form the gradient can be obtained by differentiating E EECP0720 Expert Systems – Artificial Neural Networks Derivation of the Gradient Descent Rule (cont.) The weight update rule for standard gradient descent can be summarized as

14 Stochastic Approximation to Gradient Descent EECP0720 Expert Systems – Artificial Neural Networks

15 Summary of Perceptron Perceptron training rule guaranteed to succeed if training examples are linearly separable sufficiently small learning rate Linear unit training rule uses gradient descent guaranteed to converge to hypothesis with minimum squared error given sufficiently small learning rate even when training data contains noise EECP0720 Expert Systems – Artificial Neural Networks

16 BACKPROPAGATION Algorithm EECP0720 Expert Systems – Artificial Neural Networks

17 Error Function The Backpropagation algorithm learns the weights for a multilayer network, given a network with a fixed set of units and interconnections. It employs gradient descent to attempt to minimize the squared error between the network output values and the target values for those outputs. We begin by redefining E to sum the errors over all of the network output units where outputs is the set of output units in the network, and t kd and o kd are the target and output values associated with the k th output unit and training example d. EECP0720 Expert Systems – Artificial Neural Networks

18 Architecture of Backpropagation EECP0720 Expert Systems – Artificial Neural Networks

19 Backpropagation Learning Algorithm EECP0720 Expert Systems – Artificial Neural Networks

20 Backpropagation Learning Algorithm (cont.) EECP0720 Expert Systems – Artificial Neural Networks

21 Backpropagation Learning Algorithm (cont.) EECP0720 Expert Systems – Artificial Neural Networks

22 Backpropagation Learning Algorithm (cont.) EECP0720 Expert Systems – Artificial Neural Networks

23 Backpropagation Learning Algorithm (cont.) EECP0720 Expert Systems – Artificial Neural Networks

24 Face Detection using Neural Networks EECP0720 Expert Systems – Artificial Neural Networks Neural Network Face Database Non-Face Database Training Process Output=1, for face database Output=0, for non-face database Face or Non- Face? Testing Process

25 End of Presentation EECP0720 Expert Systems – Artificial Neural Networks

26 Derivation of Backpropagation EECP0720 Expert Systems – Artificial Neural Networks

27 Derivation of Backpropagation (cont.) EECP0720 Expert Systems – Artificial Neural Networks