Machine Learning Artificial Neural Networks MPλ ∀ Stergiou Theodoros 1.

Slides:



Advertisements
Similar presentations
Artificial Neural Networks
Advertisements

Slides from: Doug Gray, David Poole
Learning in Neural and Belief Networks - Feed Forward Neural Network 2001 년 3 월 28 일 안순길.
Navneet Goyal, BITS-Pilani Perceptrons. Labeled data is called Linearly Separable Data (LSD) if there is a linear decision boundary separating the classes.
Reading for Next Week Textbook, Section 9, pp A User’s Guide to Support Vector Machines (linked from course website)
Machine Learning: Connectionist McCulloch-Pitts Neuron Perceptrons Multilayer Networks Support Vector Machines Feedback Networks Hopfield Networks.
Machine Learning Neural Networks
Artificial Neural Networks ECE 398BD Instructor: Shobha Vasudevan.
Neural NetworksNN 11 Neural Networks Teacher: Elena Marchiori R4.47 Assistant: Kees Jong S2.22
Decision Support Systems
Artificial Neural Networks ML Paul Scheible.
Connectionist models. Connectionist Models Motivated by Brain rather than Mind –A large number of very simple processing elements –A large number of weighted.
Artificial Intelligence Statistical learning methods Chapter 20, AIMA (only ANNs & SVMs)
1 Chapter 11 Neural Networks. 2 Chapter 11 Contents (1) l Biological Neurons l Artificial Neurons l Perceptrons l Multilayer Neural Networks l Backpropagation.
Neural Networks Marco Loog.
Artificial Neural Networks
Image Compression Using Neural Networks Vishal Agrawal (Y6541) Nandan Dubey (Y6279)
Artificial Neural Networks
Biointelligence Laboratory, Seoul National University
Machine Learning1 Machine Learning: Summary Greg Grudic CSCI-4830.
Classification Part 3: Artificial Neural Networks
Presentation on Neural Networks.. Basics Of Neural Networks Neural networks refers to a connectionist model that simulates the biophysical information.
Artificial Neural Networks (ANN). Output Y is 1 if at least two of the three inputs are equal to 1.
Artificial Neural Nets and AI Connectionism Sub symbolic reasoning.
DIGITAL IMAGE PROCESSING Dr J. Shanbehzadeh M. Hosseinajad ( J.Shanbehzadeh M. Hosseinajad)
Multi-Layer Perceptrons Michael J. Watts
Introduction to Neural Networks Debrup Chakraborty Pattern Recognition and Machine Learning 2006.
Explorations in Neural Networks Tianhui Cai Period 3.
Machine Learning Dr. Shazzad Hosain Department of EECS North South Universtiy
NEURAL NETWORKS FOR DATA MINING
LINEAR CLASSIFICATION. Biological inspirations  Some numbers…  The human brain contains about 10 billion nerve cells ( neurons )  Each neuron is connected.
1 Chapter 11 Neural Networks. 2 Chapter 11 Contents (1) l Biological Neurons l Artificial Neurons l Perceptrons l Multilayer Neural Networks l Backpropagation.
ARTIFICIAL NEURAL NETWORKS. Overview EdGeneral concepts Areej:Learning and Training Wesley:Limitations and optimization of ANNs Cora:Applications and.
Building high-level features using large-scale unsupervised learning Anh Nguyen, Bay-yuan Hsu CS290D – Data Mining (Spring 2014) University of California,
1 Introduction to Neural Networks And Their Applications.
Non-Bayes classifiers. Linear discriminants, neural networks.
CS-424 Gregory Dudek Today’s Lecture Neural networks –Training Backpropagation of error (backprop) –Example –Radial basis functions.
Back-Propagation Algorithm AN INTRODUCTION TO LEARNING INTERNAL REPRESENTATIONS BY ERROR PROPAGATION Presented by: Kunal Parmar UHID:
Neural Networks Teacher: Elena Marchiori R4.47 Assistant: Kees Jong S2.22
EEE502 Pattern Recognition
Each neuron has a threshold value Each neuron has weighted inputs from other neurons The input signals form a weighted sum If the activation level exceeds.
Neural Networks 2nd Edition Simon Haykin
Perceptrons Michael J. Watts
Chapter 6 Neural Network.
Bab 5 Classification: Alternative Techniques Part 4 Artificial Neural Networks Based Classifer.
Neural Networks Lecture 11: Learning in recurrent networks Geoffrey Hinton.
CSC2535: Computation in Neural Networks Lecture 8: Hopfield nets Geoffrey Hinton.
Kim HS Introduction considering that the amount of MRI data to analyze in present-day clinical trials is often on the order of hundreds or.
Neural networks (2) Reminder Avoiding overfitting Deep neural network Brief summary of supervised learning methods.
語音訊號處理之初步實驗 NTU Speech Lab 指導教授: 李琳山 助教: 熊信寬
Supervised Learning – Network is presented with the input and the desired output. – Uses a set of inputs for which the desired outputs results / classes.
Business Intelligence and Decision Support Systems (9 th Ed., Prentice Hall) Chapter 6: Artificial Neural Networks for Data Mining.
CSE343/543 Machine Learning Mayank Vatsa Lecture slides are prepared using several teaching resources and no authorship is claimed for any slides.
Machine Learning Supervised Learning Classification and Regression
Neural networks and support vector machines
Deep Learning Amin Sobhani.
Artificial neural networks
Real Neurons Cell structures Cell body Dendrites Axon
with Daniel L. Silver, Ph.D. Christian Frey, BBA April 11-12, 2017
CSE P573 Applications of Artificial Intelligence Neural Networks
Chapter 3. Artificial Neural Networks - Introduction -
Artificial Intelligence Methods
network of simple neuron-like computing elements
Artificial Neural Networks
Lecture Notes for Chapter 4 Artificial Neural Networks
Deep Learning Authors: Yann LeCun, Yoshua Bengio, Geoffrey Hinton
EE 193/Comp 150 Computing with Biological Parts
CSC 578 Neural Networks and Deep Learning
Outline Announcement Neural networks Perceptrons - continued
Presentation transcript:

Machine Learning Artificial Neural Networks MPλ ∀ Stergiou Theodoros 1

What are ANNs "...a computing system made up of a number of simple, highly interconnected processing elements, which process information by their dynamic state response to external inputs.” M. Caudill 1989 Computational model based on the structure and functions of biological Neural Networks They are considered nonlinear statistical data modeling tools where the complex relationships between inputs and outputs are modeled or patterns are found 2

Example of ANN Digit recognition ANN, can be used in post office for automated recognition of zip codes. The neural network is trained by thousands of pictures of digits The network is presented a digit that has not been used in the training and it is classified. Best commercial neural networks get over 99% accuracy 3

What is a NN? Perceptron x₁, x₂, x₃ inputs w₁, w₂, w₃ weights b bias Wanna go for a cup of coffee? 4

Perceptron NN As a single perceptron can make a decision based on some inputs and some weights, a combination of perceptrons can make more complicated decisions 5

Perceptron NN-Why is it bad? When we train a NN we repeatedly adjust the weights so that the output becomes the desired (supervised learning techniques), a process called backpropagation. A small change in weights can cause the output to completely flip state changing drastically the outcome of the NN. 6

Sigmoid Neurons Similar with perceptrons, but modified so that a small change in the weights, can cause a small change in the output. The output is a value from 0 to 1, instead of 0 or 1. Sigmoid function σ(z) 7

Digit recognition example(1) Problem: Create a program that classifies numbers from 0 to 9 Inputs: Numbers from 0 to 9, as 28x28 pixel images = 784 Inputs with values of 0 to 1, representing the gray scale of each pixel. Second Layer (Hidden Layer): 15 neurons Output Layer: 10 neurons. If the first neuron fires, then the input has been identified as a 0, if the second, then as a 1 and so on… 8

Digit recognition example(2) 9

Digit recognition example(3) Which pixel from the input digit was important for the output? After the correct or incorrect classification of the digit, the weights of the neural network are slightly changed through backpropagation. Thousands of digits pass through the ANN, each changing the weights accordingly. 10

Digit recognition example(4) The core of the system is the weight vector that was created by backpropagation. After the training the weight vector is what will classify the new inputs. The initial weights could even be completely random this way. 11

Digit recognition example(5) Input 50,000 predetermined digit images 28x28 from 250 test subjects as training set Input 10,000 predetermined digit images from the same 250 test sujects as validation set Input another 10,000 from 250 different people as test set Win – ~95% classification accuracy 12

Important Aspects of ANNs(1) Cost Function The cost function quantifies how well we are achieving the weight optimization x is the input y(x) is the vector of desired output w denotes the collection of all weights in the network b are all the biases n is the total number of training inputs a is the vector of outputs from the network 13

Important Aspects of ANNs(2) Gradient Descent In order to minimize the Cost Function we use an algorithm known as Gradient Descent. The algorithm searches for a global minima A simple visualization for a 2D minimization 14

Computational Complexity of a NN Feed forward networks don’t have a stable state of memory capacity. Instead they have input-output relations. Complexity increases very fast with the depth of the network Enabling successful training we must consider: the activation function, overspecification (larger NN than the minimum required), regularization of weights 15

Problems of NN Overtrain Bad training set Curse of dimensionality: Each additional input unit in a network adds another dimension to the space in which the data cases reside Inter-dependency of variables: It would be extremely useful if each candidate input variable could be independently assessed for usefulness 16

Variations of Neural Networks By changing the Sigmoid activation function to another, or optimizing the Cost Function or the Gradient Descent and other techniques we can see a variety of different NN, optimized for different tasks. Different type of Neural Networks: Radial Basis function, self organizing, Learning Vector Quantization, Recurrent (Fully, Simple, Echo state, Long short term memory, bi-directional, hierarchical, stochastic), Modular… 17

Sources Information Theory, Complexity and Neural Networks, Y. Abu- Mostafa, 1989, IEEE, Communications Magazine On the computational efficiency of Training Neural Networks, R. Livni et al, 2014, Advances of Neural Information Processing Systems Neural Networks and Deep Learning, M. Nielsen, 2015, available online Machine Learning online courses by Andrew Ng, Stanford University, Neural Networks online courses by Geoffrey Hinton, University of Toronto, network-ann network-ann 18

Thank you for your time 19