Neural Networks: Backpropagation algorithm Data Mining and Semantic Web University of Belgrade School of Electrical Engineering Chair of Computer Engineering and Information Theory Miroslav Tišma

You see this: But the camera sees this: What is this? Miroslav Tišma 2/21

Computer Vision: Car detection Testing: What is this? Not a car Cars Miroslav Tišma 3/21

pixel 1 pixel 2 Raw image Cars “Non”-Cars 50 x 50 pixel images→ 2500 pixels (7500 if RGB) pixel 1 intensity pixel 2 intensity pixel 2500 intensity Quadratic features ( ): ≈3 million features Learning Algorithm pixel 1 pixel Miroslav Tišma 4/21

Neural Networks Origins: Algorithms that try to mimic the brain Was very widely used in 80s and early 90s; popularity diminished in late 90s. Recent resurgence: State-of-the-art technique for many applications Miroslav Tišma 5/21

Neurons in the brain Dendr(I)tes Ax(O)n Miroslav Tišma 6/21

Neuron model: Logistic unit Sigmoid (logistic) activation function. “output” “input wires” “weights” - parameters Miroslav Tišma 7/21

Neural Network Layer 3Layer 1Layer 2 “bias unit” “output layer” “hidden layer”“input layer” Miroslav Tišma 8/21

Neural Network “activation” of unit in layer matrix of weights controlling function mapping from layer to layer If network has units in layer, units in layer, then will be of dimension Miroslav Tišma 9/21

Simple example: AND Miroslav Tišma 10/21

Example: OR function Miroslav Tišma 11/21

Multiple output units: One-vs-all. Pedestrian Car MotorcycleTruck Want, when pedestrian Miroslav Tišma 12/21 when car when motorcycle, etc.,

Neural Network (Classification) Binary classification 1 output unit Layer 1Layer 2Layer 3Layer 4 Multi-class classification (K classes) K output units total no. of layers in network no. of units (not counting bias unit) in layer pedestrian car motorcycle truck E.g.,,, Miroslav Tišma 13/21

Cost function Logistic regression: Miroslav Tišma 14/21 Neural network:

Gradient computation Need code to compute: Miroslav Tišma 15/21 Our goal is to minimize the cost function

Given one training example (, ): Forward propagation: Layer 1Layer 2Layer 3Layer Miroslav Tišma 16/21 Backpropagation algorithm

Intuition: “error” of node in layer. Layer 1Layer 2Layer 3Layer 4 For each output unit (layer L = 4) element-wise multiplication operator Miroslav Tišma 17/21

Backpropagation algorithm Training set Set (for all ). For Set Perform forward propagation to compute for Using, compute Compute Miroslav Tišma 18/21

Advantages: -Relatively simple implementation -Standard method and generally wokrs well -Many practical applications: * handwriting recognition, autonomous driving car Disadvantages: -Slow and inefficient -Can get stuck in local minima resulting in sub-optimal solutions Miroslav Tišma 19/21

Literature: Miroslav Tišma 20/21

Miroslav Tišma 21/21 Thank you for your attention!