Combining CNN with RNN for scene labeling (segmentation) Tao Zeng

Scene labeling Problem Image classification Classifying each image into K class Image segmentation: Classifying each pixel in the image Krizhevsky, Alex, Ilya Sutskever, and Geoffrey E. Hinton. "Imagenet classification with deep convolutional neural networks." Advances in neural information processing systems. 2012. Labeling every pixel in the image with object class it belongs to. Chen, Liang-Chieh, et al. "Semantic image segmentation with deep convolutional nets and fully connected crfs." arXiv preprint arXiv:1412.7062 (2014).

Scene labeling Problem Spatiotemporal segmentation: Challenge: Solving segmentation and recognition simultaneously X t Labeling every pixel in the image with object class it belongs to. Seguin, Guillaume, et al. "Instance-level video segmentation from object tracks." (2016).

Patch wise training & patch wise prediction Slow due to redundant computation Li, Hongsheng, Rui Zhao, and Xiaogang Wang. "Highly efficient forward and backward propagation of convolutional neural networks for pixelwise classification." arXiv preprint arXiv:1412.4526 (2014).

Fully convolutional Networks Long, Jonathan, Evan Shelhamer, and Trevor Darrell. "Fully convolutional networks for semantic segmentation." Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2015.

Motivation Motivation CNN does not have an explicit mechanism to modulate feature with context (CRF) Need to model the relationship between labels

Contextual information is important

Idea of recurrent CNN Providing feedback from the output into the input allows the network to model label dependencies, and correct its own previous predictions Ensuring the object coherence in scene labeling

Recurrent CNN Model 1 Model 2 Liang, Ming, and Xiaolin Hu. "Recurrent convolutional neural network for object recognition." Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2015. Pinheiro, Pedro HO, and Ronan Collobert. "Recurrent Convolutional Neural Networks for Scene Labeling." ICML. 2014.

=? Weights sharing Recurrent RCL Residual Network Weight Layer X +

determining the label of a pixel in an image The model is able to perform local feature extraction and context integration simultaneously in each parameterized layer, therefore particularly fits this application because both local and global information are critical for determining the label of a pixel in an image

Spatiotemporal sequence prediction

Problem Goal: Predicting the future rainfall intensity in a local region over a relatively short period of time M x N 2D space , p measurements Predict the most likely length-K sequence in the future given the previous J observations

1D LSTM to 2D conv LSTM Donahue, Jeffrey, et al. "Long-term recurrent convolutional networks for visual recognition and description." Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2015.

Combining Fully Convolutional and Recurrent Neural networks for 3D Biomedical Image Segmentation

Problem 3D biomedical images are often anisotropic: high solution in x-y axis but low in Z axis Previous Methods: 2D convolutional for each slice and then followed by concatenating them into 3D 3D convolution 2D-3D hybrid Approach Combine FCN (u-Net) and LSTM (BDC-LSTM) to Exploit intra-slice and inter-slice contexts High resolution Low resolution

U-Net: Biomedical image segmentation

kU-Net

CLSTM

Nz slices - -> kU-Net --> 64 x Nx x Ny feature map f2dZ f2dZ --> BDC-LSTM ----> f3dZ -->softmax -->prob Decoupling kU-Net and BDC-LSTM training due to GPU memory and context consideration

Thank you! Questions ?