ECE 539 Final Project Mark Slosarek

Slides:

Advertisements

Similar presentations

Zhen Lu CPACT University of Newcastle MDC Technology Reduced Hessian Sequential Quadratic Programming(SQP)

Advertisements

NEURAL NETWORKS Perceptron

Fast Algorithms For Hierarchical Range Histogram Constructions

CSCI 347 / CS 4206: Data Mining Module 07: Implementations Topic 03: Linear Models.

Brain Damage: Algorithms for Network Pruning Andrew Yip HMC Fall 2003.

Presenter: Yufan Liu November 17th,

Prénom Nom Document Analysis: Artificial Neural Networks Prof. Rolf Ingold, University of Fribourg Master course, spring semester 2008.

Prénom Nom Document Analysis: Linear Discrimination Prof. Rolf Ingold, University of Fribourg Master course, spring semester 2008.

September 30, 2010Neural Networks Lecture 8: Backpropagation Learning 1 Sigmoidal Neurons In backpropagation networks, we typically choose  = 1 and 

The Widrow-Hoff Algorithm (Primal Form) Repeat: Until convergence criterion satisfied return: Given a training set and learning rate Initial:  Minimize.

Prénom Nom Document Analysis: Artificial Neural Networks Prof. Rolf Ingold, University of Fribourg Master course, spring semester 2008.

KNN, LVQ, SOM. Instance Based Learning K-Nearest Neighbor Algorithm (LVQ) Learning Vector Quantization (SOM) Self Organizing Maps.

Improved BP algorithms ( first order gradient method) 1.BP with momentum 2.Delta- bar- delta 3.Decoupled momentum 4.RProp 5.Adaptive BP 6.Trinary BP 7.BP.

12 1 Variations on Backpropagation Variations Heuristic Modifications –Momentum –Variable Learning Rate Standard Numerical Optimization –Conjugate.

October 28, 2010Neural Networks Lecture 13: Adaptive Networks 1 Adaptive Networks As you know, there is no equation that would tell you the ideal number.

Last lecture summary.

Last lecture summary.

Traffic Sign Recognition Using Artificial Neural Network Radi Bekker

# 1 # 2 # 3 Multiply # 4 Multiply: Multiply # 5 # 6 Solve.

Implementation of Nonlinear Conjugate Gradient Method for MLP Matt Peterson ECE 539 December 10, 2001.

Neural Networks Lecture 8: Two simple learning algorithms

CSC 4510 – Machine Learning Dr. Mary-Angela Papalaskari Department of Computing Sciences Villanova University Course website:

Biointelligence Laboratory, Seoul National University

Soft Computing Lecture 18 Foundations of genetic algorithms (GA). Using of GA.

Lecture 3 Introduction to Neural Networks and Fuzzy Logic President UniversityErwin SitompulNNFL 3/1 Dr.-Ing. Erwin Sitompul President University

Radial Basis Function Networks:

Artificial Neural Networks. The Brain How do brains work? How do human brains differ from that of other animals? Can we base models of artificial intelligence.

Last lecture summary. biologically motivated synapses Neuron accumulates (Σ) positive/negative stimuli from other neurons. Then Σ is processed further.

Research Vignette: The TransCom3 Time-Dependent Global CO 2 Flux Inversion … and More David F. Baker NCAR 12 July 2007 David F. Baker NCAR 12 July 2007.

Soft Computing Lecture 8 Using of perceptron for image recognition and forecasting.

Non-Bayes classifiers. Linear discriminants, neural networks.

ELeaRNT: Evolutionary Learning of Rich Neural Network Topologies Authors: Slobodan Miletic 3078/2010 Nikola Jovanovic 3077/2010

CSE & CSE6002E - Soft Computing Winter Semester, 2011 Neural Networks Videos Brief Review The Next Generation Neural Networks - Geoff Hinton.

Intro. ANN & Fuzzy Systems Lecture 14. MLP (VI): Model Selection.

ADALINE (ADAptive LInear NEuron) Network and

Feature Selection in k-Median Clustering Olvi Mangasarian and Edward Wild University of Wisconsin - Madison.

Advanced Computer Graphics CSE 190 [Spring 2015], Lecture 9 Ravi Ramamoorthi

Linear Prediction Correlation can be used to make predictions – Values on X can be used to predict values on Y – Stronger relationships between X and Y.

Implementing Local Relative Sensitivity Pruning Paul Victorey.

Variations on Backpropagation.

Hand-written character recognition

ELG5377 Adaptive Signal Processing Lecture 15: Recursive Least Squares (RLS) Algorithm.

Mesh Resampling Wolfgang Knoll, Reinhard Russ, Cornelia Hasil 1 Institute of Computer Graphics and Algorithms Vienna University of Technology.

The Unscented Kalman Filter for Nonlinear Estimation Young Ki Baik.

Numerical Analysis – Data Fitting Hanyang University Jong-Il Park.

Overfitting, Bias/Variance tradeoff. 2 Content of the presentation Bias and variance definitions Parameters that influence bias and variance Bias and.

Introduction to Algorithms: Brute-Force Algorithms.

Evolutionary Computation Evolving Neural Network Topologies.

ECE 3301 General Electrical Engineering

Neural Networks Winter-Spring 2014

MATLAB Implementation of the Optimal Brain Surgeon (OBS) Algorithm

A Simple Artificial Neuron

Derivation of a Learning Rule for Perceptrons

Adaptation Behavior of Pipelined Adaptive Filters

MLP Based Feedback System for Gas Valve Control in a Madison Symmetric Torus Andrew Seltzman Dec 14, 2010.

Java Implementation of Optimal Brain Surgeon

Going Backwards In The Procedure and Recapitulation of System Identification By Ali Pekcan 65570B.

Generalization ..

Optimal Brain Surgeon Algorithm

Forward & Backward selection in hybrid network

Completing the Square.

Variations on Backpropagation.

The loss function, the normal equation,

Mathematical Foundations of BME Reza Shadmehr

2. Matrix-Vector Formulation of Backpropagation Learning

Neuro-Computing Lecture 2 Single-Layer Perceptrons

Variations on Backpropagation.

Neural Network Training

August 8, 2006 Danny Budik, Itamar Elhanany Machine Intelligence Lab

Presentation transcript:

ECE 539 Final Project Mark Slosarek Optimal Brain Surgeon ECE 539 Final Project Mark Slosarek

Background Optimal Brain Surgeon Algorithm (OBS) is a pruning algorithm Reduces weights to reduce overall complexity of network

Benefits A pruned network has several benefits Quicker Calculations Less Storage Space required A pruned network should not have significantly more error than an non-pruned network

Steps to Perform OBS Train the given MLP to minimize mean-square error Calculate the cost of the equation Compute the inverse Hessian Find the smallest Saliency If Saliency is much smaller than mean-square, delete that weight and repeat for next weight, other go to next step Update all weights

Tests Performed and Results To test the effectiveness of the OSB, I used the wine data from the homework The pruned network contained approximately 60% fewer weights Results from both networks, however results of pruning were very inconclusive The results were not very consistant

Conclusion A pruned network can save much space and time My algorithm is not perfect, and could be recoded to be more efficient Results not noticeable in simple networks, but in real world problems, could be results could be great