cs638/838 - Spring 2017 (Shavlik©), Week 7

Slides:

Advertisements

Similar presentations

ImageNet Classification with Deep Convolutional Neural Networks

Advertisements

Statistical Classification Rong Jin. Classification Problems X Input Y Output ? Given input X={x 1, x 2, …, x m } Predict the class label y  Y Y = {-1,1},

Taking the Kitchen Sink Seriously: An Ensemble Approach to Word Sense Disambiguation from Christopher Manning et al.

Face Detection using the Viola-Jones Method

Cascade Correlation Architecture and Learning Algorithm for Neural Networks.

Today’s Topics Chapter 2 in One Slide Chapter 18: Machine Learning (ML) Creating an ML Dataset –“Fixed-length feature vectors” –Relational/graph-based.

NEURAL NETWORKS FOR DATA MINING

Today’s Topics HW0 due 11:55pm tonight and no later than next Tuesday HW1 out on class home page; discussion page in MoodleHW1discussion page Please do.

Today’s Topics Read –For exam: Chapter 13 of textbook –Not on exam: Sections & Genetic Algorithms (GAs) –Mutation –Crossover –Fitness-proportional.

Today’s Topics Read Chapter 3 & Section 4.1 (Skim Section 3.6 and rest of Chapter 4), Sections 5.1, 5.2, 5.3, 5,7, 5.8, & 5.9 (skim rest of Chapter 5)

Today’s Topics Learning Decision Trees (Chapter 18) –We’ll use d-trees to introduce/motivate many general issues in ML (eg, overfitting reduction) “Forests”

Today’s Topics Playing Deterministic (no Dice, etc) Games –Mini-max –  -  pruning –ML and games? 1997: Computer Chess Player (IBM’s Deep Blue) Beat Human.

Today’s Topics 11/10/15CS Fall 2015 (Shavlik©), Lecture 22, Week 101 Support Vector Machines (SVMs) Three Key Ideas –Max Margins –Allowing Misclassified.

Today’s Topics Midterm class mean: 83.5 HW3 Due Thursday and HW4 Out Thursday Turn in Your BN Nannon Player (in Separate, ‘Dummy’ Assignment) until a Week.

Learning to Detect Faces A Large-Scale Application of Machine Learning (This material is not in the text: for further information see the paper by P.

Today’s Topics Read: Chapters 7, 8, and 9 on Logical Representation and Reasoning HW3 due at 11:55pm THURS (ditto for your Nannon Tourney Entry) Recipe.

Neural Networks Vladimir Pleskonjić 3188/ /20 Vladimir Pleskonjić General Feedforward neural networks Inputs are numeric features Outputs are in.

Today’s Topics Remember: no discussing exam until next Tues! ok to stop by Thurs 5:45-7:15pm for HW3 help More BN Practice (from Fall 2014 CS 540 Final)

Today’s Topics Artificial Neural Networks (ANNs) Perceptrons (1950s) Hidden Units and Backpropagation (1980s) Deep Neural Networks (2010s) ??? (2040s [note.

Deep Learning Overview Sources: workshop-tutorial-final.pdf

Today’s Topics 11/10/15CS Fall 2015 (Shavlik©), Lecture 21, Week 101 More on DEEP ANNs –Convolution –Max Pooling –Drop Out Final ANN Wrapup FYI:

Today’s Lecture Neural networks Training

Convolutional Sequence to Sequence Learning

Presented by Yuting Liu

cs638/838 - Spring 2017 (Shavlik©), Week 11

CS 636/838: BUILDING Deep Neural Networks

CS Fall 2016 (Shavlik©), Lecture 5

Deep Learning Amin Sobhani.

Neural Networks for Quantum Simulation

Artificial Neural Networks

cs638/838 - Spring 2017 (Shavlik©), Week 10

Data Mining, Neural Network and Genetic Programming

Why it is Called Tensor Flow Parallelism in ANNs Project Ideas and Discussion Glenn Fung Presents Batch Renormalizating Paper.

Computer Science and Engineering, Seoul National University

Convolutional Neural Fabrics by Shreyas Saxena, Jakob Verbeek

CS Fall 2016 (Shavlik©), Lecture 11, Week 6

CS Fall 2016 (© Jude Shavlik), Lecture 4

cs540 - Fall 2015 (Shavlik©), Lecture 25, Week 14

Neural Networks CS 446 Machine Learning.

Classification with Perceptrons Reading:

Intelligent Information System Lab

CS Fall 2016 (Shavlik©), Lecture 12, Week 6

Supervised Training of Deep Networks

Neural Networks 2 CS446 Machine Learning.

A brief introduction to neural network

CS540 - Fall 2016 (Shavlik©), Lecture 17, Week 10

CS Fall 2016 (Shavlik©), Lecture 8, Week 5

ANN Design and Training

Gradient Checks for ANN

CS540 - Fall 2016(Shavlik©), Lecture 16, Week 9

CS 4501: Introduction to Computer Vision Training Neural Networks II

cs540 - Fall 2016 (Shavlik©), Lecture 20, Week 11

cs540 - Fall 2016 (Shavlik©), Lecture 18, Week 10

with Daniel L. Silver, Ph.D. Christian Frey, BBA April 11-12, 2017

CS Fall 2016 (Shavlik©), Lecture 2

Chap. 7 Regularization for Deep Learning (7.8~7.12 )

Concept Learning Algorithms

CSC 578 Neural Networks and Deep Learning

Neural Networks Geoff Hulten.

On Convolutional Neural Network

CS Fall 2016 (Shavlik©), Lecture 12, Week 6

cs638/838 - Spring 2017 (Shavlik©), Week 7

RCNN, Fast-RCNN, Faster-RCNN

Deep Learning Libraries

VERY DEEP CONVOLUTIONAL NETWORKS FOR LARGE-SCALE IMAGE RECOGNITION

EE 193/Comp 150 Computing with Biological Parts

CSC 578 Neural Networks and Deep Learning

Principles of Back-Propagation

An introduction to neural network and machine learning

Evaluation David Kauchak CS 158 – Fall 2019.

Presentation transcript:

cs638/838 - Spring 2017 (Shavlik©), Week 7 CS 540 Fall 2015 (Shavlik) 11/15/2018 Today’s Topics Some Lab 3 Tips and Discussion Still Need a Lab 3 Partner? If Still Working on Lab 2, Complete ASAP! Some Advanced Weight-Learning Topics (by Yujia Bao) Interpreting ANN Outputs as Probability Distributions and the Cross-Entropy Error Function (might be covered next week) Next week: An Overview of Some Deep ML Projects at American Family 2/28/17 cs638/838 - Spring 2017 (Shavlik©), Week 7

Back to Deep ANNs - Convolution & Max Pooling (Repeat) CS 540 Fall 2015 (Shavlik) 11/15/2018 Back to Deep ANNs - Convolution & Max Pooling (Repeat) C = Convolution, MP = Max Pooling (ie, a CHANGE OF REP) My implementation (no dropout yet) of the above topology takes about 1-2 mins per epoch on the provided TRAIN/TUNE/TEST set (I measure TRAIN, TUNE and TEST accuracy after each epoch) 2/28/17 cs638/838 - Spring 2017 (Shavlik©), Week 7

Details: Image to First CONV There are MULTIPLE, INDEPENDENT plates (so each can learn a different ‘derived feature’) Within a plate, there is only one set of weights, shared for all ‘starting positions’ Image Convolution 2/28/17 cs638/838 - Spring 2017 (Shavlik©), Week 7

Details: Image to First CONV There are MULTIPLE, INDEPENDENT plates (so each can learn a different ‘derived feature’) Within a plate, there is only one set of weights, shared for all ‘starting positions’ Note that the ‘sliding windows’ OVERLAP (a design choice) Image Convolution 2/28/17 cs638/838 - Spring 2017 (Shavlik©), Week 7

Details: Image to First CONV There are MULTIPLE, INDEPENDENT plates (so each can learn a different ‘derived feature’) Within a plate, there is only one set of weights, shared for all ‘starting positions’ Image Convolution 2/28/17 cs638/838 - Spring 2017 (Shavlik©), Week 7

Details: Image to First CONV There are MULTIPLE, INDEPENDENT plates (so each can learn a different ‘derived feature’) Within a plate, there is only one set of weights, shared for all ‘starting positions’ Image Convolution 2/28/17 cs638/838 - Spring 2017 (Shavlik©), Week 7

CONV Layer Weight Sharing The SAME WEIGHTs are used for all ‘sliding window’ locations per plate You will only backprop through CONV HUs that are the MAX in some POOL window You might want to scale (ie, divide) η by the number of cells in the following POOL layer (ie, the number of MAX’es computed) 2/28/17 cs638/838 - Spring 2017 (Shavlik©), Week 7

cs638/838 - Spring 2017 (Shavlik©), Week 7 From CS 540 textbook, by Russel and Norvig, 3rd ed 2/28/17 cs638/838 - Spring 2017 (Shavlik©), Week 7

Details: Nth CONV to Nth POOL I decided to have one POOL per CONV POOLs compute MAX via code (no wgts) In general, could have several POOLs of different sizes per CONV MAX Convolution Pooling 2/28/17 cs638/838 - Spring 2017 (Shavlik©), Week 7

Details: Nth CONV to Nth POOL I decided to have one POOL per CONV POOLs compute MAX via code (no wgts) In general, could have several POOLs of different sizes per CONV Note that the ‘sliding windows’ DO NOT OVERLAP (a design choice) MAX Convolution Pooling 2/28/17 cs638/838 - Spring 2017 (Shavlik©), Week 7

Details: Nth POOLs to Nth CONV We want a CONV layer to look at ALL nodes at previous POOL layer, but only look at the same window This allows learning combinations of derived features … Pooling Convolution 2/28/17 cs638/838 - Spring 2017 (Shavlik©), Week 7

Details: Nth POOLs to Nth CONV We want a CONV layer to look at ALL nodes at previous POOL layer, but only look at the same window This allows learning combinations of derived features … Pooling Convolution 2/28/17 cs638/838 - Spring 2017 (Shavlik©), Week 7

Details: Nth CONV to 1 HU Layer We want the flat layer of HUs to look all CONV/POOL nodes at final CONV/POOL layer. Initially Lab 3 said to use the following, but ok to follow Slide 2’s drawing and have a 3rd CONV layer (I did this) CONV-POOL-CONV-POOL-FLAT-OUTPUT Connect each HU to every node at previous layer . Final ‘Flat’ HU Layer … Convolution or Pooling 2/28/17 cs638/838 - Spring 2017 (Shavlik©), Week 7

BP’ing Through POOL Nodes My implementation only BPs through the CONV nodes that are the MAX for some POOL window (all others have deviation=0) There are no learnable weights (and biases) between CONV and POOL layers (where needed I assume wgt=1, but never change it – ie, the MAX calc is not done via wgt’ed sums and an activation function; it is just Java code) POOL nodes compute ‘deviations’ (ie, the intermediate calc in the BP algorithm), but don’t change wgts and biases If POOL windows do not overlap (and only 1 POOL plate per CONV plate), a CONV node only sums the deviations from at most one POOL node in the next layer 2/28/17 cs638/838 - Spring 2017 (Shavlik©), Week 7

cs638/838 - Spring 2017 (Shavlik©), Week 7 Pictorially Weighted sum of deviations Use deviation at POOL node … … Convolution Pooling Convolution 2/28/17 cs638/838 - Spring 2017 (Shavlik©), Week 7