Deep Learning What’s the buzz all about? Dr. Debdoot Sheet Assistant Professor, Department of Electrical Engineering Principal Investigator, Kharagpur Learning, Imaging and Visualization Group Indian Institute of Technology Kharagpur

11 May 20162Deep Learning - What's the buzz all about? / Debdoot Sheet

Learning? A computer program is said to learn from experience E with respect to some class of tasks T and performance measure P, if its performance at tasks in T, as measured by P, improves with experience E -Tom Mitchell 11 May 2016Deep Learning - What's the buzz all about? / Debdoot Sheet3

Demystifying Learning 11 May 2016Deep Learning - What's the buzz all about? / Debdoot Sheet4 Man 1 Man 2 Man 3Man 4 Great Wall logo Great Wall tower Kim Jung Wang Debdoot Experience (E) Performance (P) Debdoot, Kim, Jung and Wang are standing near the Great Wall logo and the Great Wall tower is behind them.

How was it Learning? 11 May 2016Deep Learning - What's the buzz all about? / Debdoot Sheet5 Man 1 Man 2 Man 3Man 4 Great Wall logo Great Wall tower Kim Jung Wang Debdoot Salient Segments Objectify Detect humans Recognize inanimate Describe Scene Debdoot, Kim, Jung and Wang are standing near the Great Wall logo and the Great Wall tower is behind them. Recognize humans

Challenges 11 May 2016Deep Learning - What's the buzz all about? / Debdoot Sheet6 Salient Segments Objectify Detect humans Recognize inanimate Describe Scene Recognize humans Salient Segments Objectify Detect humans Recognize inanimate Describe Scene Recognize humans Salient Segments Objectify Detect humans Recognize inanimate Describe Scene Recognize humans Salient Segments Objectify Detect humans Recognize inanimate Describe Scene Recognize humans

Challenges 11 May 2016Deep Learning - What's the buzz all about? / Debdoot Sheet7 Salient Segments Objectify Recognize inanimate Describe Scene Recognize humans LBP Wavelets HoG Body part recognition Human appearance Chroma clustering Posture realign Silhouette matching Recognize human Detect humans

Dilemma only in Machine Vision? Speech Recognition and Signal Processing – Dahl, et al,(2012). Context dependent pre-trained deep neural networks for large vocabulary speech recognition. IEEE Trans. Audio, Speech, and Language Processing, 20(1), 33–42. Handwriting Recognition – Bengio, et al (2006). Greedy layer-wise training of deep networks. In NIPS’2006. Natural Language Processing – Hinton, (1986). Learning distributed representations of concepts. In Proc. 8th Conf. Cog. Sc. Society, pp. 1–12. Hierarchical and Transfer Learning – Goodfellow, et al. (2011). Spike-and slab sparse coding for unsupervised feature discovery. In NIPS’2011 Workshop on Challenges in Learning Hierarchical Models. Medical Imaging – Karri and Sheet, et al (2014). Deep learnt random forests for segmentation of retinal layers in optical coherence tomography images. In ISBI’ May 2016Deep Learning - What's the buzz all about? / Debdoot Sheet8

How to tackle this dilemma? 11 May 2016Deep Learning - What's the buzz all about? / Debdoot Sheet9 Great Wall behind Great Wall logo beside Debdoot, Kim, Jung, Wang

Multilayer Perceptron (MLP) 11 May 2016Deep Learning - What's the buzz all about? / Debdoot Sheet10

MLP Learning, troubles thereof 11 May 2016Deep Learning - What's the buzz all about? / Debdoot Sheet11 P T1T1 T2T2

MLP Learning troubles, why so? 11 May 2016Deep Learning - What's the buzz all about? / Debdoot Sheet12 P T1T1 T2T2 LBP Wavelets HoG Body part recognition Human appearance Chroma clustering Posture realign Silhouette matching Recognize human ?

Deep Learning, origin and growth Around 1950 – NN age – Neural Nets (McCulloch and Pitts, 1943) – Unsupervised Learn. (Hebb, 1949) – Supervised Learn. (Rosenblatt, 1958) – Associative Memory (Palm, 1980; Hopfield, 1982) 1960 – Discovery of visual sensory cells that respond to Edges (Hubel and Wiesel, 1962) – Feed Forward Multi Layer Perceptron (FF-MLP) (Ivakhnenko, 1968) 1980 – Neocognition – Convolution + WeightReplication + Subsampling (Fukushima, 1980) – Max Pooling – Back-propagation (Werbos, 1981; LeCunn, 1985, 1988) 11 May 2016Deep Learning - What's the buzz all about? / Debdoot Sheet13

Deep Learning, origin and growth – Search for simple, low-complexity, problem-solvers – Recurrent Neural Network (RNN) (Hochreiter and Schmidhuber, 1996) – Local learning Feed forward NN (Dayan and Hinton, 1996) – Advanced gradient descent (Schaback and Werner, 1992) – Sequential Network Construction (Honavar and Uhr, 1988) – Unsupervised Pre-training (Ritter and Kohonen, 1989) – Auto-Encoder (Hinton et al., 1989) – Back Propagating Convolutional Neural Networks (LeCun et al., 1989, 1990a, 1998) 11 May 2016Deep Learning - What's the buzz all about? / Debdoot Sheet14

Deep Learning, origin and growth 2000 – Era of Deep Learning – NIPS 2003 Feature Selection Challenge (Neal and Zhang, 2006) – MNIST digit recognition (LeCun et al., 1989) – Deep Belief Network (DBN) / Restricted Boltzmann Machines (Hinton et al., 2006) – Auto Encoders (Bengio, 2009) 2006 – GPU based CNN (Chellapilla et al., 2006) 2009 – GPU DBN (Raina et al., 2009) 2011 – Max-Pooling CNN on the GPU (Ciresan et al., 2011) 2012 – Image Net (Krizhevsky et al., 2012) 11 May 2016Deep Learning - What's the buzz all about? / Debdoot Sheet15

Families of Deep Learning Fully connected networks – Autoencoders Autoencoders Stacked Autoencoder Sparse Autoencoder Denoising Autoencoder Convolutional Autoencoder – Belief Networks Restricted Boltzmann Machines Deep Belief Networks Convolutional Networks – Conv-Nets – U-Nets – Res-Nets Recurrent Neural Networks – Long short-term memory (LSTM) 11 May 2016Deep Learning - What's the buzz all about? / Debdoot Sheet16

DEEP LEARNING TODAY Is it reality? 11 May 2016Deep Learning - What's the buzz all about? / Debdoot Sheet17

Science fiction or reality Synthetic facial expression Machine that can dream 11 May 2016Deep Learning - What's the buzz all about? / Debdoot Sheet18 Susskind, Joshua M., et al. "Generating facial expressions with deep belief nets." Affective Computing, Emotion Modelling, Synthesis and Recognition (2008):

Deep Learning, where else? 11 May 2016Deep Learning - What's the buzz all about? / Debdoot Sheet19

Deep Learning, Google’s stand 11 May 2016Deep Learning - What's the buzz all about? / Debdoot Sheet20 DNNresearch

Deep Learning, Microsoft’s view 11 May 2016Deep Learning - What's the buzz all about? / Debdoot Sheet21

COMPUTATIONAL MEDICAL IMAGING My experiences with Deep Learning (or not) for 11 May 2016Deep Learning - What's the buzz all about? / Debdoot Sheet22

(Not so Deep) Transfer Learning 11 May 2016Deep Learning - What's the buzz all about? / Debdoot Sheet23 D. Sheet, et al, Iterative self- organizing atherosclerotic tissue labeling in intravascular ultrasound images and comparison with virtual histology, IEEE TBME, 59(11), 2012 D. Sheet, et al, Hunting for necrosis in the shadows of intravascular ultrasound, CMIG, 38(2), 2014 D. Sheet, et al, Joint learning of ultrasonic backscattering statistical physics and signal confidence primal for characterizing atherosclerotic plaques using intravascular ultrasound, Med. Image Anal,18(1), 2014 Training set of IVUS signals Ground truth Random forest learning Test IVUS signal Labeled tissue Multi-scale Speckle stats.

Local Minima Juggling, experience 11 May 2016Deep Learning - What's the buzz all about? / Debdoot Sheet24 P T1T1 T2T2 LBP Wavelets HoG Body part recognition Human appearance Chroma clustering Posture realign Silhouette matching Recognize human

Transfer vs. Full Deep Architecture 11 May 2016Deep Learning - What's the buzz all about? / Debdoot Sheet25 Multi-scale modeling of OCT speckles Training image set Ground truth Random forest learning Multi-scale modeling of OCT speckles Test image Labeled tissue Stacked Auto- Encoders, Logistic Regression Random Forest Training image set Ground truth Autoencoder kernels

Heuristics in State of Art Deep Learning - What's the buzz all about? / Debdoot Sheet2611 May 2016

The Solution Deep Learning - What's the buzz all about? / Debdoot Sheet27 Denoising Auto Encoder Logistic Reg. 11 May 2016

Using a Deep Network Deep Learning - What's the buzz all about? / Debdoot Sheet2811 May 2016

Learning of Representations Deep Learning - What's the buzz all about? / Debdoot Sheet29 Representation of speckle appearance models learned by DAE1 Sparsity of representations learned by DAE2 11 May 2016

Results in Wounds Deep Learning - What's the buzz all about? / Debdoot Sheet30 (a) OCT image of wound(b) Ground truth (c) In situ histology Epithelium, Papillary dermis, Dermis, Adipose 11 May 2016

Endnote (almost there) “It’s like in quantum physics at the beginning of the 20th century” Trishul Chilimbi (MSR, DNN, Adam) “The experimentalists and practitioners were ahead of the theoreticians. They couldn’t explain the results. We appear to be at a similar stage with DNNs. We’re realizing the power and the capabilities, but we still don’t understand the fundamentals of exactly how they work.” 11 May 2016Deep Learning - What's the buzz all about? / Debdoot Sheet31

Take home message Hardware Resources – Custom workstations GTX TITAN X / Tesla K40 GTX 1080 (Yet to be on shelf) – Deep Learning Box NVIDIA DGX-1 Toolboxes – Theano (Python/SciPy) – Torch – Pylearn2 – Matlab DeepLearningToolbox (GitHub) – Matlab 2016 NN Toolbox Autoencoders Convolutional Neural Network More information – – – Schmidhuber (2014). Deep Learning in Neural Networks: An Overview (arXiv: ) – Deng and Yu (2013). Deep Learning: Methods and Applications. Conferences – Int. Conf. Learning Representations (ICLR) – Neural Inf. Process. Sys. (NIPS) 11 May 2016Deep Learning - What's the buzz all about? / Debdoot Sheet32