Randomness in Neural Networks

Slides:

Advertisements

Similar presentations

Slides from: Doug Gray, David Poole

Advertisements

Support Vector Machines

Pattern Recognition and Machine Learning: Kernel Methods.

Machine learning continued Image source:

Artificial Neural Networks - Introduction -

Machine Learning Neural Networks

Image classification Given the bag-of-features representations of images from different classes, how do we learn a model for distinguishing them?

RBF Neural Networks x x1 Examples inside circles 1 and 2 are of class +, examples outside both circles are of class – What NN does.

Goals of Adaptive Signal Processing Design algorithms that learn from training data Algorithms must have good properties: attain good solutions, simple.

Machine Learning Motivation for machine learning How to set up a problem How to design a learner Introduce one class of learners (ANN) –Perceptrons –Feed-forward.

Arizona State University DMML Kernel Methods – Gaussian Processes Presented by Shankar Bhargav.

Hazırlayan NEURAL NETWORKS Radial Basis Function Networks I PROF. DR. YUSUF OYSAL.

CS Instance Based Learning1 Instance Based Learning.

Aula 4 Radial Basis Function Networks

SOMTIME: AN ARTIFICIAL NEURAL NETWORK FOR TOPOLOGICAL AND TEMPORAL CORRELATION FOR SPATIOTEMPORAL PATTERN LEARNING.

Radial Basis Function (RBF) Networks

Last lecture summary.

Radial-Basis Function Networks

Hazırlayan NEURAL NETWORKS Radial Basis Function Networks II PROF. DR. YUSUF OYSAL.

Radial Basis Function Networks

A Genetic Algorithms Approach to Feature Subset Selection Problem by Hasan Doğu TAŞKIRAN CS 550 – Machine Learning Workshop Department of Computer Engineering.

Integrating Neural Network and Genetic Algorithm to Solve Function Approximation Combined with Optimization Problem Term presentation for CSC7333 Machine.

NEURAL NETWORKS FOR DATA MINING

Classification / Regression Neural Networks 2

ARTIFICIAL NEURAL NETWORKS. Overview EdGeneral concepts Areej:Learning and Training Wesley:Limitations and optimization of ANNs Cora:Applications and.

Classifiers Given a feature representation for images, how do we learn a model for distinguishing features from different classes? Zebra Non-zebra Decision.

Neural and Evolutionary Computing - Lecture 9 1 Evolutionary Neural Networks Design  Motivation  Evolutionary training  Evolutionary design of the architecture.

Introduction to Neural Networks Introduction to Neural Networks Applied to OCR and Speech Recognition An actual neuron A crude model of a neuron Computational.

Neural Networks Presented by M. Abbasi Course lecturer: Dr.Tohidkhah.

Data Mining and Decision Support

Ghent University Pattern recognition with CNNs as reservoirs David Verstraeten 1 – Samuel Xavier de Souza 2 – Benjamin Schrauwen 1 Johan Suykens 2 – Dirk.

Learning Kernel Classifiers 1. Introduction Summarized by In-Hee Lee.

Ghent University An overview of Reservoir Computing: theory, applications and implementations Benjamin Schrauwen David Verstraeten and Jan Van Campenhout.

Deep Learning Overview Sources: workshop-tutorial-final.pdf

Machine Learning Artificial Neural Networks MPλ ∀ Stergiou Theodoros 1.

CSE343/543 Machine Learning Mayank Vatsa Lecture slides are prepared using several teaching resources and no authorship is claimed for any slides.

Combining Models Foundations of Algorithms and Machine Learning (CS60020), IIT KGP, 2017: Indrajit Bhattacharya.

Today’s Lecture Neural networks Training

Machine Learning Supervised Learning Classification and Regression

Neural networks and support vector machines

Big data classification using neural network

CS 9633 Machine Learning Support Vector Machines

Unsupervised Learning of Video Representations using LSTMs

Predictive Learning from Data

Convolutional Neural Network

CS 388: Natural Language Processing: LSTM Recurrent Neural Networks

Deep Feedforward Networks

Deep Learning Amin Sobhani.

LECTURE 09: BAYESIAN ESTIMATION (Cont.)

Recurrent Neural Networks for Natural Language Processing

Radial Basis Function G.Anuradha.

Intelligent Information System Lab

LECTURE 28: NEURAL NETWORKS

Neural networks (3) Regularization Autoencoder

RECURRENT NEURAL NETWORKS FOR VOICE ACTIVITY DETECTION

Convolutional Networks

Classification / Regression Neural Networks 2

CSC 578 Neural Networks and Deep Learning

A critical review of RNN for sequence learning Zachary C

Grid Long Short-Term Memory

Neuro-Computing Lecture 4 Radial Basis Function Network

network of simple neuron-like computing elements

Neural Network - 2 Mayank Vatsa

An Introduction To The Backpropagation Algorithm

Neural Networks Geoff Hulten.

LECTURE 28: NEURAL NETWORKS

Introduction to Radial Basis Function Networks

Neural networks (3) Regularization Autoencoder

Deep Learning Authors: Yann LeCun, Yoshua Bengio, Geoffrey Hinton

Automatic Handwriting Generation

Presentation transcript:

Randomness in Neural Networks Presented by Paul Melman

Neural networks Made up of many connected units, each one a non-linear classifier Train them by adjusting weights based on error, with Backprop or other algorithms Deep networks with many layers can produce high classification accuracy, but have very long training times Smaller networks are quick to train, but do not form rich representations of training data

Random weights Assign a subset of the weights randomly instead of training Three families of random weight models: Feedforward networks with random weights (RW-FNN) Recurrent networks with random weights (i.e. reservoir computing) Randomized kernel approximations

Basic premise Random weights are used to define a feature map to transform the input into a high dimensional space Resulting optimization problem is linear least-squares “Randomization is… cheaper than optimization” – Rahimi & Recht

RW-FNN basic architecture Dashed lines are fixed connections; solid lines are trainable

RW-FNN architecture cont. B is typically much larger than the number of input dimensions Weights wm are drawn from predefines probability distribution (potentially by an order of magnitude)

RW-FNNs cont. Additive methods: RBF methods: Each function chosen as radial basis function

Kernel approximation Random sampling can be used for kernel approximation Kernel methods are often expensive in terms of time and memory, random methods reduce these costs Sample randomly from the kernel matrix Design stochastic approximation of kernel function

Recurrent networks Dynamic data with a temporal component is difficult for a feedforward network to learn Recurrent neural networks (RNNs) have connections going in reverse, allowing for temporal processing Units get information about prior states of other units in the network via these connections

Reservoir computing Recurrent layer of fixed, randomly generated nonlinearities

Reservoir computing cont. RC architectures are very successful in tasks that require relatively short memory processing, including: Grammatical inference Stock price prediction Speech recognition Robotic control Acoustic modeling

Echo state property Reservoirs with random weights can be unstable; they may oscillate or behave chaotically The effects of any given input state must vanish over time, so that it does not persist indefinitely, or worse, become amplified Having the reservoir “forget” prior states after a certain number of epochs prevents these problems

New techniques Lateral inhibition Biologically inspired process by which the activity of one unit inhibits adjacent units Can be implemented by having multiple smaller reservoirs which inhibit adjacent reservoirs Intrinsic plasticity Add adaptable parameters to nonlinearity function of reservoir

References Scardapane, Simone, and Dianhui Wang. "Randomness in neural networks: an overview." Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery 7.2 (2017).