Reading for Next Week Textbook, Section 9, pp. 52-56 A User’s Guide to Support Vector Machines (linked from course website)

Slides:



Advertisements
Similar presentations
Artificial Neural Networks
Advertisements

Beyond Linear Separability
Slides from: Doug Gray, David Poole
1 Machine Learning: Lecture 4 Artificial Neural Networks (Based on Chapter 4 of Mitchell T.., Machine Learning, 1997)
Artificial Intelligence 13. Multi-Layer ANNs Course V231 Department of Computing Imperial College © Simon Colton.
For Wednesday Read chapter 19, sections 1-3 No homework.
CSCI 347 / CS 4206: Data Mining Module 07: Implementations Topic 03: Linear Models.
CS Perceptrons1. 2 Basic Neuron CS Perceptrons3 Expanded Neuron.
Classification Neural Networks 1
Machine Learning Neural Networks
Overview over different methods – Supervised Learning
Artificial Neural Networks
Neural Networks.
Lecture 14 – Neural Networks
Artificial Neural Networks ML Paul Scheible.
Biological neuron artificial neuron.
Artificial Neural Networks
October 14, 2010Neural Networks Lecture 12: Backpropagation Examples 1 Example I: Predicting the Weather We decide (or experimentally determine) to use.
Artificial Neural Networks
LOGO Classification III Lecturer: Dr. Bo Yuan
CS 4700: Foundations of Artificial Intelligence
CS 484 – Artificial Intelligence
CSC 4510 – Machine Learning Dr. Mary-Angela Papalaskari Department of Computing Sciences Villanova University Course website:
131 Feed-Forward Artificial Neural Networks MEDINFO 2004, T02: Machine Learning Methods for Decision Support and Discovery Constantin F. Aliferis & Ioannis.
Artificial Neural Networks
December 5, 2012Introduction to Artificial Intelligence Lecture 20: Neural Network Application Design III 1 Example I: Predicting the Weather Since the.
Computer Science and Engineering
1 Artificial Neural Networks Sanun Srisuk EECP0720 Expert Systems – Artificial Neural Networks.
Neural Networks Ellen Walker Hiram College. Connectionist Architectures Characterized by (Rich & Knight) –Large number of very simple neuron-like processing.
CS464 Introduction to Machine Learning1 Artificial N eural N etworks Artificial neural networks (ANNs) provide a general, practical method for learning.
Machine Learning Chapter 4. Artificial Neural Networks
11 CSE 4705 Artificial Intelligence Jinbo Bi Department of Computer Science & Engineering
Machine Learning Dr. Shazzad Hosain Department of EECS North South Universtiy
Classification / Regression Neural Networks 2
LINEAR CLASSIFICATION. Biological inspirations  Some numbers…  The human brain contains about 10 billion nerve cells ( neurons )  Each neuron is connected.
CS 478 – Tools for Machine Learning and Data Mining Backpropagation.
November 26, 2013Computer Vision Lecture 15: Object Recognition III 1 Backpropagation Network Structure Perceptrons (and many other classifiers) can only.
Neural Networks and Machine Learning Applications CSC 563 Prof. Mohamed Batouche Computer Science Department CCIS – King Saud University Riyadh, Saudi.
Linear Discrimination Reading: Chapter 2 of textbook.
Linear Classification with Perceptrons
Neural Networks and Backpropagation Sebastian Thrun , Fall 2000.
CSSE463: Image Recognition Day 14 Lab due Weds, 3:25. Lab due Weds, 3:25. My solutions assume that you don't threshold the shapes.ppt image. My solutions.
Back-Propagation Algorithm AN INTRODUCTION TO LEARNING INTERNAL REPRESENTATIONS BY ERROR PROPAGATION Presented by: Kunal Parmar UHID:
Neural Networks - Berrin Yanıkoğlu1 Applications and Examples From Mitchell Chp. 4.
Artificial Neural Networks (Cont.) Chapter 4 Perceptron Gradient Descent Multilayer Networks Backpropagation Algorithm 1.
Neural Networks Teacher: Elena Marchiori R4.47 Assistant: Kees Jong S2.22
Artificial Neural Network
EEE502 Pattern Recognition
Multilayer Neural Networks (sometimes called “Multilayer Perceptrons” or MLPs)
Artificial Neural Network. Introduction Robust approach to approximating real-valued, discrete-valued, and vector-valued target functions Backpropagation.
Artificial Neural Network. Introduction Robust approach to approximating real-valued, discrete-valued, and vector-valued target functions Backpropagation.
129 Feed-Forward Artificial Neural Networks AMIA 2003, Machine Learning Tutorial Constantin F. Aliferis & Ioannis Tsamardinos Discovery Systems Laboratory.
Learning: Neural Networks Artificial Intelligence CMSC February 3, 2005.
Learning with Neural Networks Artificial Intelligence CMSC February 19, 2002.
CSE343/543 Machine Learning Mayank Vatsa Lecture slides are prepared using several teaching resources and no authorship is claimed for any slides.
Machine Learning Supervised Learning Classification and Regression
Fall 2004 Backpropagation CS478 - Machine Learning.
Linear separability Hyperplane In 2D: Feature 1 Feature 2 A perceptron can separate data that is linearly separable.
Linear separability Hyperplane In 2D: Feature 1 Feature 2 A perceptron can separate data that is linearly separable.
Classification with Perceptrons Reading:
Artificial Neural Networks
Artificial Neural Networks
Classification / Regression Neural Networks 2
Machine Learning Today: Reading: Maria Florina Balcan
CSC 578 Neural Networks and Deep Learning
Classification Neural Networks 1
Artificial Neural Networks
Lecture Notes for Chapter 4 Artificial Neural Networks
Seminar on Machine Learning Rada Mihalcea
Presentation transcript:

Reading for Next Week Textbook, Section 9, pp A User’s Guide to Support Vector Machines (linked from course website)

Results of classifier ThresholdAccuracyTPRFPR ∞ Creating a ROC Curve (If the score is equal to or greater than the threshold, the classifier assigns “positive”, else it assigns “negative”.)

Why is it desirable to maximize the area under the ROC curve?

Multi-Layer Perceptrons (Neural Networks)

biological neuronartificial neuron

A two-layer neural network Input layer (activations represent feature vector for one training example) Hidden layer (“internal representation”) Output layer (activation represents classification) Weighted connections

Example: ALVINN (Pomerleau, 1993) ALVINN learns to drive an autonomous vehicle at normal speeds on public highways. Input: 30 x 32 grid of pixel intensities from camera

Each output unit correspond to a particular steering direction. The most highly activated one gives the direction to steer.

What kinds of problems are suitable for neural networks? Have sufficient training data Long training times are acceptable Not necessary for humans to understand learned target function or hypothesis

Advantages of neural networks Designed to be parallelized Robust on noisy training data Fast to evaluate new examples

Multi-layer Perceptrons (MLPs) Single-layer perceptrons can only represent linear decision surfaces. Multi-layer perceptrons can represent non-linear decision surfaces

Learning in Multilayer Neural Networks: Gradient descent in weight space From T. M. Mitchell, Machine Learning Error w0w0 w1w1

Differentiable Threshold Units In general, in order to represent non-linear functions, need non-linear output function at each unit. In order to do gradient descent on weights, need differentiable error function (and thus differentiable output function).

Sigmoid activation function: To do gradient descent in multi-layer networks, need to generalize perceptron learning rule for non-linear activation functions.

Why use sigmoid activation function? Note that sigmoid activation function is non-linear, differentiable, and approximates a sgn function. The derivative of the sigmoid activation function is easily expressed in terms of its output: This is useful in deriving the back-propagation algorithm.

Training multi-layer perceptrons Assume two-layer networks: Repeat for a given number of “epochs” or until accuracy on training or test data is acceptable: For each training example: 1.Present input to the input layer. 2.Forward propagate the activations times the weights to each node in the hidden layer. 3.At each output unit, determine the error Err 4.Run the back-propagation algorithm to update all weights in the network.

Forward propagation example h1h1 h2h2 x1x1 x2x2 o What is o if input x = (1, -1)?

Calculate error terms: For each output unit k, calculate error term  k : For each hidden unit h, calculate error term  h : Backpropagation (Derivation of formulas is in course readings)

Update the weights: For each network weight w ji : where

Will this algorithm converge on optimal weights? Not always, because of local minima in error surface. “ In many practical applications the problem of local minima has not been found to be as severe as one might fear. ” Especially true with large networks, since many dimensions give many “ escape routes ”. But “ no methods are known to predict with certainty when local minima will cause difficulties. ”

Multi-layer networks can do everything (in principle) Universal approximation theorem: One layer of hidden units suffices to approximate any function with finitely many discontinuities to arbitrary precision, if the activation functions of the hidden units are non-linear. In most applications, two-layer network with sigmoid activation function used.

Heuristics for avoiding local minima in weight space Use stochastic gradient descent rather than true gradient descent. There is a different “ error surface ” for each training example. This adds “ noise ” to search process which can prevent it from getting stuck in local optima. Train multiple networks using the same data, but start each one with different random weights. If different weight vectors are found after training, select best one using separate validation set, or use ensemble of networks to vote on correct classification. Add “ momentum ” term to weight update rule. Simulated annealing Boosting

Learning rate and momentum Recall that stochastic gradient descent approaches “ true ” gradient descent as learning rate  0. For practical purposes: choose  large enough so that learning is efficient, but small enough that a good approximation to true gradient descent is maintained, and oscillations are not reached. To avoid oscillations at large , introduce momentum, in which change in weight is dependent on past weight change: where n is the iteration through the main loop of backpropagation The idea is to keep weight changes moving in the same direction.

Hidden-layer representation in multi-layer perceptrons “ Auto-associator network ” InputHidden Output activations                 Results after network was trained for 5000 epochs. What is the encoding? bias unit +1

One advantage of neural networks is that “ high-level feature extraction ” can be done automatically!

How the auto-associator network was trained Initial weights set to random values in (-0.1, 0.1). Learning rate  = 0.3 Momentum  = epochs of training with back-propagation

Regularization “ Regularization ” means to modify the error function to constrain the network in various ways. Examples: –Add a penalty term for the magnitude of the weight vector (to decrease overfitting):

Other topics not covered here Other methods for training the weights Recurrent networks Dynamically modifying network structure

Next set of slides will be covered only if we have time.

Example: Face recognition (From T. M. Mitchell, Machine Learning, Chapter 4) Task: Classify camera images of various people in various poses. Data: Photos, varying: –Facial expression: happy, sad, angry, neutral –Direction person is facing: left, right, straight ahead, up –Wearing sunglasses?: yes, no Within these, variation in background, clothes, position of face for a given person.

an2i_right_sad_sunglasses_4 an2i_left_angry_open_4 glickman_left_angry_open_4

Design Choices Input encoding Output encoding Network topology Learning rate Momentum

Preprocessing of photo: –Create 30x32 coarse resolution version of 120x128 image –This makes size of neural network more manageable Input to neural network: –Photo is encoded as 30x32 = 960 pixel intensity values, scaled to be in [0,1] –One input unit per pixel Output units: –Encode classification of input photo

Possible target functions for neural network: –Direction person is facing –Identity of person –Gender of person –Facial expression –etc. As an example, consider target of “ direction person is facing ”.

Network architecture... Input layer: 960 units Output layer: 4 units Classification result is most highly activated output unit. Hidden layer: 3 units left right up straight Network is fully connected

Target function Target function is: –Output unit should have activation 0.9 if it corresponds to correct classification –Otherwise output unit should have activation 0.1 Use these values instead of 1 and 0, since sigmoid units can ’ t produce 1 and 0 activation.

Other parameters Learning rate  = 0.3 Momentum  = 0.3 If these are set too high, training fails to converge on network with acceptable error over training set. If these are set too low, training takes much longer.

Training For maximum of M epochs: –For each training example Input photo to network Propagate activations to output units Determine error in output units Adjust weights using back-propagation algorithm –Test accuracy of network on validation set. If accuracy is acceptable, stop algorithm. Demo of code

Understanding weight values After training: –Weights from input to hidden layer: high positive in certain facial regions –“ Right ” output unit has strong positive from second hidden unit, strong negative from third hidden unit. Second hidden unit has positive weights on right side of face (aligns with bright skin of person turned to right) and negative weights on top of head (aligns with dark hair of person turned to right) Third hidden unit has negative weights on right side of face, so will output value close to zero for person turned to right.

After 100 epochs of training. (From T. M. Mitchell, Machine Learning) Weights from each pixel to hidden units 1, 2, 3 (white = high, black = low) Weights from each hidden unit to four output units Bias unit

Hidden unit 2 has high positive weights from right side of face. If person is looking to the right, this will cause unit 2 to have high activation. Output unit right has high positive weight from hidden unit 2. Weights from each pixel to hidden units 1, 2, 3 (white = high, black = low) Weights from each hidden unit to four output units Bias unit