A practical guide to learning Autoencoders Dr. Debdoot Sheet Assistant Professor, Department of Electrical Engineering Principal Investigator, Kharagpur Learning, Imaging and Visualization Group Indian Institute of Technology Kharagpur www.facweb.iitkgp.ernet.in/~debdoot/

Autoencoders (Practical Guide and Demystifying) / Debdoot Sheet Contents Introductory concepts Simple neuron Neural network formulation Learning with error backpropagation Gradient checking and optimization Unsupervised pre-training Autoencoder Denoising autoencoders Sparsity in autoencoders Stacked autoencoder Supervised refinement Stacked denoising autoencoders (SDAE) – Deep neural network (DNN) 11 May 2016 Autoencoders (Practical Guide and Demystifying) / Debdoot Sheet

Introductory concepts 11 May 2016 Autoencoders (Practical Guide and Demystifying) / Debdoot Sheet

Autoencoders (Practical Guide and Demystifying) / Debdoot Sheet Simple Neuron Model 11 May 2016 Autoencoders (Practical Guide and Demystifying) / Debdoot Sheet

Neural Network Formulation 11 May 2016 Autoencoders (Practical Guide and Demystifying) / Debdoot Sheet

Autoencoders (Practical Guide and Demystifying) / Debdoot Sheet Error in Prediction 11 May 2016 Autoencoders (Practical Guide and Demystifying) / Debdoot Sheet

Error Backpropagation 11 May 2016 Autoencoders (Practical Guide and Demystifying) / Debdoot Sheet

Gradient Descent Learning Cost-function Epochs 11 May 2016 Autoencoders (Practical Guide and Demystifying) / Debdoot Sheet

Understanding Gradient Descent Some random initial value Cost-function Epochs 11 May 2016 Autoencoders (Practical Guide and Demystifying) / Debdoot Sheet

Unsupervised pre-training 11 May 2016 Autoencoders (Practical Guide and Demystifying) / Debdoot Sheet

Autoencoders (Practical Guide and Demystifying) / Debdoot Sheet 11 May 2016 Autoencoders (Practical Guide and Demystifying) / Debdoot Sheet

Denoising Autoencoder 11 May 2016 Autoencoders (Practical Guide and Demystifying) / Debdoot Sheet

Sparsity in Autoencoder 11 May 2016 Autoencoders (Practical Guide and Demystifying) / Debdoot Sheet

Autoencoders (Practical Guide and Demystifying) / Debdoot Sheet Stacked Autoencoders 11 May 2016 Autoencoders (Practical Guide and Demystifying) / Debdoot Sheet

Supervised Refinement 11 May 2016 Autoencoders (Practical Guide and Demystifying) / Debdoot Sheet

Deep Neural Network Denoising Auto Encoder Denoising Auto Encoder Logistic Reg. 11 May 2016 Autoencoders (Practical Guide and Demystifying) / Debdoot Sheet

Autoencoders (Practical Guide and Demystifying) / Debdoot Sheet Weight Refinement 11 May 2016 Autoencoders (Practical Guide and Demystifying) / Debdoot Sheet

Autoencoders (Practical Guide and Demystifying) / Debdoot Sheet Practical exampleS 11 May 2016 Autoencoders (Practical Guide and Demystifying) / Debdoot Sheet

Demystifying Autoencoders Dr. Debdoot Sheet Assistant Professor, Department of Electrical Engineering Principal Investigator, Kharagpur Learning, Imaging and Visualization Group Indian Institute of Technology Kharagpur www.facweb.iitkgp.ernet.in/~debdoot/

Autoencoders (Practical Guide and Demystifying) / Debdoot Sheet Contents Handwritten Digits Recognition Organ Detection in 4D MRI OCT Tissue Characterization Retinal Vessel Segmentation 11 May 2016 Autoencoders (Practical Guide and Demystifying) / Debdoot Sheet

Handwritten Digit Recognition Vincent, Pascal, et al. "Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion." The Journal of Machine Learning Research 11 (2010): 3371-3408. Handwritten Digit Recognition 11 May 2016 Autoencoders (Practical Guide and Demystifying) / Debdoot Sheet

Autoencoders (Practical Guide and Demystifying) / Debdoot Sheet About the Challenge MNIST Handwritten digit recognition challenge Train set 60,000 samples Test set 10,000 samples Patch size 28 x 28 pixels uint8 gray valued Binary images 11 May 2016 Autoencoders (Practical Guide and Demystifying) / Debdoot Sheet

Autoencoders (Practical Guide and Demystifying) / Debdoot Sheet Interpreting Weights 0% Noise added 25% Noise added 11 May 2016 Autoencoders (Practical Guide and Demystifying) / Debdoot Sheet

Dynamics of Weights with Noise Neuron A Neuron B 0% 10% 20% 50% 11 May 2016 Autoencoders (Practical Guide and Demystifying) / Debdoot Sheet

Appearance Model Variations 11 May 2016 Autoencoders (Practical Guide and Demystifying) / Debdoot Sheet

Autoencoders (Practical Guide and Demystifying) / Debdoot Sheet Performance Metrics 11 May 2016 Autoencoders (Practical Guide and Demystifying) / Debdoot Sheet

Organ Detection in 4D MRI Shin, Hoo-Chang, et al. "Stacked autoencoders for unsupervised feature learning and multiple organ detection in a pilot study using 4D patient data.“ IEEE Trans. Pattern Analysis and Machine Intelligence, 35.8 (2013): 1930-1943. Organ Detection in 4D MRI 11 May 2016 Autoencoders (Practical Guide and Demystifying) / Debdoot Sheet

Autoencoders (Practical Guide and Demystifying) / Debdoot Sheet About the Challenge Dataset A: 256 x 256 images, 7-12 contiguous coronal slices, 40 temporal points, Patients # 46 Dataset B: 256 x 247 images, 14 contiguous coronal slices, 40 temporal points, Patients # 3 Dataset C: 209 x 256 images, 12 contiguous coronal slices, 40 temporal points, Patients # 29 11 May 2016 Autoencoders (Practical Guide and Demystifying) / Debdoot Sheet

Autoencoders (Practical Guide and Demystifying) / Debdoot Sheet The Autoencoder Model 11 May 2016 Autoencoders (Practical Guide and Demystifying) / Debdoot Sheet

Learnt Representations Input image Weights in first layer Class predictions 11 May 2016 Autoencoders (Practical Guide and Demystifying) / Debdoot Sheet

OCT Tissue Characterization Sheet, Debdoot, et al. "Deep learning of tissue specific speckle representations in optical coherence tomography and deeper exploration for in situ histology.“ IEEE 12th International Symposium on Biomedical Imaging (ISBI), 2015. OCT Tissue Characterization 11 May 2016 Autoencoders (Practical Guide and Demystifying) / Debdoot Sheet

Autoencoders (Practical Guide and Demystifying) / Debdoot Sheet About the Challenge Data Collection School of Medical Science and Technology, Indian Institute of Technology Kharagpur 1300 nm (HPBW 100 nm) Swept Source OCT System OCS 1300 SS, ThorLabs, NJ, USA 8 bit bitmap images Histology for ground truth HE stained Samples Mus musculus (small mice) 16 healthy skin 2 wounds on skin DNN architecture Patch size – 36 × 36 px DAE1 – 400 nodes DAE2 – 100 nodes Target – Logistic Reg. 5 outputs Sparsity – 20% Mini-batch training In situ Histology Performance Epithelium – 96% Papillary dermis – 93% Dermis – 99% Adipose tissue – 98% 11 May 2016 Autoencoders (Practical Guide and Demystifying) / Debdoot Sheet

Unfurling the Deep Network 11 May 2016 Autoencoders (Practical Guide and Demystifying) / Debdoot Sheet

Learning of Representations Sparsity of representations learned by DAE2 Representation of speckle appearance models learned by DAE1 11 May 2016 Autoencoders (Practical Guide and Demystifying) / Debdoot Sheet

Retinal Vessel Segmentation Maji, Debapriya; Santara, Anirban, et al. "Deep Neural Network and Random Forest Hybrid Architecture for Learning to Detect Retinal Vessels in Fundus Images.“ 34th Annual Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), 2015. Retinal Vessel Segmentation 11 May 2016 Autoencoders (Practical Guide and Demystifying) / Debdoot Sheet

Autoencoders (Practical Guide and Demystifying) / Debdoot Sheet About the Challenge Images Color Fundus 563 x 582 pixels Train set 20 images Test set 11 May 2016 Autoencoders (Practical Guide and Demystifying) / Debdoot Sheet

Autoencoders (Practical Guide and Demystifying) / Debdoot Sheet About the Hybrid Model 11 May 2016 Autoencoders (Practical Guide and Demystifying) / Debdoot Sheet

Train Patterns and Weights DNN (SAE) weights Layer 1 Layer 2 11 May 2016 Autoencoders (Practical Guide and Demystifying) / Debdoot Sheet

Autoencoders (Practical Guide and Demystifying) / Debdoot Sheet Performances 11 May 2016 Autoencoders (Practical Guide and Demystifying) / Debdoot Sheet

Autoencoders (Practical Guide and Demystifying) / Debdoot Sheet Take Home Messages LeCunn, Yan, et al. “Deep Learning.” Nature, 521 (2015): 436-444. Bengio, Yoshua, Aaron Courville, and Pierre Vincent. "Representation learning: A review and new perspectives." IEEE Trans. Pattern Analysis and Machine Intelligence, 35.8 (2013): 1798-1828. Vincent, Pascal, et al. "Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion." The Journal of Machine Learning Research, 11 (2010): 3371-3408. 11 May 2016 Autoencoders (Practical Guide and Demystifying) / Debdoot Sheet