Presentation is loading. Please wait.

Presentation is loading. Please wait.

A practical guide to learning Autoencoders

Published bySharon Lane Modified over 6 years ago

Similar presentations

Presentation on theme: "A practical guide to learning Autoencoders"— Presentation transcript:

1 A practical guide to learning Autoencoders
Dr. Debdoot Sheet Assistant Professor, Department of Electrical Engineering Principal Investigator, Kharagpur Learning, Imaging and Visualization Group Indian Institute of Technology Kharagpur

2 Autoencoders (Practical Guide and Demystifying) / Debdoot Sheet
Contents Introductory concepts Simple neuron Neural network formulation Learning with error backpropagation Gradient checking and optimization Unsupervised pre-training Autoencoder Denoising autoencoders Sparsity in autoencoders Stacked autoencoder Supervised refinement Stacked denoising autoencoders (SDAE) – Deep neural network (DNN) 11 May 2016 Autoencoders (Practical Guide and Demystifying) / Debdoot Sheet

3 Introductory concepts
11 May 2016 Autoencoders (Practical Guide and Demystifying) / Debdoot Sheet

4 Autoencoders (Practical Guide and Demystifying) / Debdoot Sheet
Simple Neuron Model 11 May 2016 Autoencoders (Practical Guide and Demystifying) / Debdoot Sheet

5 Neural Network Formulation
11 May 2016 Autoencoders (Practical Guide and Demystifying) / Debdoot Sheet

6 Autoencoders (Practical Guide and Demystifying) / Debdoot Sheet
Error in Prediction 11 May 2016 Autoencoders (Practical Guide and Demystifying) / Debdoot Sheet

7 Error Backpropagation
11 May 2016 Autoencoders (Practical Guide and Demystifying) / Debdoot Sheet

8 Gradient Descent Learning
Cost-function Epochs 11 May 2016 Autoencoders (Practical Guide and Demystifying) / Debdoot Sheet

9 Understanding Gradient Descent
Some random initial value Cost-function Epochs 11 May 2016 Autoencoders (Practical Guide and Demystifying) / Debdoot Sheet

10 Unsupervised pre-training
11 May 2016 Autoencoders (Practical Guide and Demystifying) / Debdoot Sheet

11 Autoencoders (Practical Guide and Demystifying) / Debdoot Sheet
11 May 2016 Autoencoders (Practical Guide and Demystifying) / Debdoot Sheet

12 Denoising Autoencoder
11 May 2016 Autoencoders (Practical Guide and Demystifying) / Debdoot Sheet

13 Sparsity in Autoencoder
11 May 2016 Autoencoders (Practical Guide and Demystifying) / Debdoot Sheet

14 Autoencoders (Practical Guide and Demystifying) / Debdoot Sheet
Stacked Autoencoders 11 May 2016 Autoencoders (Practical Guide and Demystifying) / Debdoot Sheet

15 Supervised Refinement
11 May 2016 Autoencoders (Practical Guide and Demystifying) / Debdoot Sheet

16 Deep Neural Network Denoising Auto Encoder Denoising Auto Encoder
Logistic Reg. 11 May 2016 Autoencoders (Practical Guide and Demystifying) / Debdoot Sheet

17 Autoencoders (Practical Guide and Demystifying) / Debdoot Sheet
Weight Refinement 11 May 2016 Autoencoders (Practical Guide and Demystifying) / Debdoot Sheet

18 Autoencoders (Practical Guide and Demystifying) / Debdoot Sheet
Practical exampleS 11 May 2016 Autoencoders (Practical Guide and Demystifying) / Debdoot Sheet

19 Demystifying Autoencoders
Dr. Debdoot Sheet Assistant Professor, Department of Electrical Engineering Principal Investigator, Kharagpur Learning, Imaging and Visualization Group Indian Institute of Technology Kharagpur

20 Autoencoders (Practical Guide and Demystifying) / Debdoot Sheet
Contents Handwritten Digits Recognition Organ Detection in 4D MRI OCT Tissue Characterization Retinal Vessel Segmentation 11 May 2016 Autoencoders (Practical Guide and Demystifying) / Debdoot Sheet

21 Handwritten Digit Recognition
Vincent, Pascal, et al. "Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion." The Journal of Machine Learning Research 11 (2010): Handwritten Digit Recognition 11 May 2016 Autoencoders (Practical Guide and Demystifying) / Debdoot Sheet

22 Autoencoders (Practical Guide and Demystifying) / Debdoot Sheet
About the Challenge MNIST Handwritten digit recognition challenge Train set 60,000 samples Test set 10,000 samples Patch size 28 x 28 pixels uint8 gray valued Binary images 11 May 2016 Autoencoders (Practical Guide and Demystifying) / Debdoot Sheet

23 Autoencoders (Practical Guide and Demystifying) / Debdoot Sheet
Interpreting Weights 0% Noise added 25% Noise added 11 May 2016 Autoencoders (Practical Guide and Demystifying) / Debdoot Sheet

24 Dynamics of Weights with Noise
Neuron A Neuron B 0% 10% 20% 50% 11 May 2016 Autoencoders (Practical Guide and Demystifying) / Debdoot Sheet

25 Appearance Model Variations
11 May 2016 Autoencoders (Practical Guide and Demystifying) / Debdoot Sheet

26 Autoencoders (Practical Guide and Demystifying) / Debdoot Sheet
Performance Metrics 11 May 2016 Autoencoders (Practical Guide and Demystifying) / Debdoot Sheet

27 Organ Detection in 4D MRI
Shin, Hoo-Chang, et al. "Stacked autoencoders for unsupervised feature learning and multiple organ detection in a pilot study using 4D patient data.“ IEEE Trans. Pattern Analysis and Machine Intelligence, 35.8 (2013): Organ Detection in 4D MRI 11 May 2016 Autoencoders (Practical Guide and Demystifying) / Debdoot Sheet

28 Autoencoders (Practical Guide and Demystifying) / Debdoot Sheet
About the Challenge Dataset A: 256 x 256 images, 7-12 contiguous coronal slices, 40 temporal points, Patients # 46 Dataset B: 256 x 247 images, 14 contiguous coronal slices, 40 temporal points, Patients # 3 Dataset C: 209 x 256 images, 12 contiguous coronal slices, 40 temporal points, Patients # 29 11 May 2016 Autoencoders (Practical Guide and Demystifying) / Debdoot Sheet

29 Autoencoders (Practical Guide and Demystifying) / Debdoot Sheet
The Autoencoder Model 11 May 2016 Autoencoders (Practical Guide and Demystifying) / Debdoot Sheet

30 Learnt Representations
Input image Weights in first layer Class predictions 11 May 2016 Autoencoders (Practical Guide and Demystifying) / Debdoot Sheet

31 OCT Tissue Characterization
Sheet, Debdoot, et al. "Deep learning of tissue specific speckle representations in optical coherence tomography and deeper exploration for in situ histology.“ IEEE 12th International Symposium on Biomedical Imaging (ISBI), 2015. OCT Tissue Characterization 11 May 2016 Autoencoders (Practical Guide and Demystifying) / Debdoot Sheet

32 Autoencoders (Practical Guide and Demystifying) / Debdoot Sheet
About the Challenge Data Collection School of Medical Science and Technology, Indian Institute of Technology Kharagpur 1300 nm (HPBW 100 nm) Swept Source OCT System OCS 1300 SS, ThorLabs, NJ, USA 8 bit bitmap images Histology for ground truth HE stained Samples Mus musculus (small mice) 16 healthy skin 2 wounds on skin DNN architecture Patch size – 36 × 36 px DAE1 – 400 nodes DAE2 – 100 nodes Target – Logistic Reg. 5 outputs Sparsity – 20% Mini-batch training In situ Histology Performance Epithelium – 96% Papillary dermis – 93% Dermis – 99% Adipose tissue – 98% 11 May 2016 Autoencoders (Practical Guide and Demystifying) / Debdoot Sheet

33 Unfurling the Deep Network
11 May 2016 Autoencoders (Practical Guide and Demystifying) / Debdoot Sheet

34 Learning of Representations
Sparsity of representations learned by DAE2 Representation of speckle appearance models learned by DAE1 11 May 2016 Autoencoders (Practical Guide and Demystifying) / Debdoot Sheet

35 Retinal Vessel Segmentation
Maji, Debapriya; Santara, Anirban, et al. "Deep Neural Network and Random Forest Hybrid Architecture for Learning to Detect Retinal Vessels in Fundus Images.“ 34th Annual Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), 2015. Retinal Vessel Segmentation 11 May 2016 Autoencoders (Practical Guide and Demystifying) / Debdoot Sheet

36 Autoencoders (Practical Guide and Demystifying) / Debdoot Sheet
About the Challenge Images Color Fundus 563 x 582 pixels Train set 20 images Test set 11 May 2016 Autoencoders (Practical Guide and Demystifying) / Debdoot Sheet

37 Autoencoders (Practical Guide and Demystifying) / Debdoot Sheet
About the Hybrid Model 11 May 2016 Autoencoders (Practical Guide and Demystifying) / Debdoot Sheet

38 Train Patterns and Weights
DNN (SAE) weights Layer 1 Layer 2 11 May 2016 Autoencoders (Practical Guide and Demystifying) / Debdoot Sheet

39 Autoencoders (Practical Guide and Demystifying) / Debdoot Sheet
Performances 11 May 2016 Autoencoders (Practical Guide and Demystifying) / Debdoot Sheet

40 Autoencoders (Practical Guide and Demystifying) / Debdoot Sheet
Take Home Messages LeCunn, Yan, et al. “Deep Learning.” Nature, 521 (2015): Bengio, Yoshua, Aaron Courville, and Pierre Vincent. "Representation learning: A review and new perspectives." IEEE Trans. Pattern Analysis and Machine Intelligence, 35.8 (2013): Vincent, Pascal, et al. "Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion." The Journal of Machine Learning Research, 11 (2010): 11 May 2016 Autoencoders (Practical Guide and Demystifying) / Debdoot Sheet

Download ppt "A practical guide to learning Autoencoders"

Similar presentations

Greedy Layer-Wise Training of Deep Networks

Greedy Layer-Wise Training of Deep Networks
Advanced topics.

Advanced topics.
Stacking RBMs and Auto-encoders for Deep Architectures References:[Bengio, 2009], [Vincent et al., 2008] 2011/03/03 강병곤.

Stacking RBMs and Auto-encoders for Deep Architectures References:[Bengio, 2009], [Vincent et al., 2008] 2011/03/03 강병곤.
Deep Learning.

Deep Learning.
An introduction to: Deep Learning aka or related to Deep Neural Networks Deep Structural Learning Deep Belief Networks etc,

An introduction to: Deep Learning aka or related to Deep Neural Networks Deep Structural Learning Deep Belief Networks etc,
Supervised and Unsupervised learning and application to Neuroscience Cours CA6b-4.

Supervised and Unsupervised learning and application to Neuroscience Cours CA6b-4.
How to do backpropagation in a brain

How to do backpropagation in a brain
Unsupervised Learning With Neural Nets Deep Learning and Neural Nets Spring 2015.

Unsupervised Learning With Neural Nets Deep Learning and Neural Nets Spring 2015.
Deep Belief Networks for Spam Filtering

Deep Belief Networks for Spam Filtering
Autoencoders Mostafa Heidarpour

Autoencoders Mostafa Heidarpour
Handwritten Character Recognition Using Block wise Segmentation Technique (BST) in Neural Network 47th Annual Convention of the Computer Society of India.

Handwritten Character Recognition Using Block wise Segmentation Technique (BST) in Neural Network 47th Annual Convention of the Computer Society of India.
LLNL-PRES-671957 This work was performed under the auspices of the U.S. Department of Energy by Lawrence Livermore National Laboratory under Contract DE-AC52-07NA27344.

LLNL-PRES This work was performed under the auspices of the U.S. Department of Energy by Lawrence Livermore National Laboratory under Contract DE-AC52-07NA27344.
Neural Networks Chapter 6 Joost N. Kok Universiteit Leiden.

Neural Networks Chapter 6 Joost N. Kok Universiteit Leiden.
Explorations in Neural Networks Tianhui Cai Period 3.

Explorations in Neural Networks Tianhui Cai Period 3.
Backpropagation An efficient way to compute the gradient Hung-yi Lee.

Backpropagation An efficient way to compute the gradient Hung-yi Lee.
A shallow introduction to Deep Learning

A shallow introduction to Deep Learning
COMPARISON OF IMAGE ANALYSIS FOR THAI HANDWRITTEN CHARACTER RECOGNITION Olarik Surinta, chatklaw Jareanpon Department of Management Information System.

COMPARISON OF IMAGE ANALYSIS FOR THAI HANDWRITTEN CHARACTER RECOGNITION Olarik Surinta, chatklaw Jareanpon Department of Management Information System.
Yang, Luyu.  Postal service for sorting mails by the postal code written on the envelop  Bank system for processing checks by reading the amount of.

Yang, Luyu.  Postal service for sorting mails by the postal code written on the envelop  Bank system for processing checks by reading the amount of.
Building high-level features using large-scale unsupervised learning Anh Nguyen, Bay-yuan Hsu CS290D – Data Mining (Spring 2014) University of California,

Building high-level features using large-scale unsupervised learning Anh Nguyen, Bay-yuan Hsu CS290D – Data Mining (Spring 2014) University of California,
Handwritten Hindi Numerals Recognition Kritika Singh Akarshan Sarkar Mentor- Prof. Amitabha Mukerjee.

Handwritten Hindi Numerals Recognition Kritika Singh Akarshan Sarkar Mentor- Prof. Amitabha Mukerjee.

Similar presentations

Ads by Google