1. Combining CNN with RNN for scene labeling (segmentation)
Tao Zeng

2. Scene labeling Problem
Image classification
Classifying each image into K class
Image segmentation:
Classifying each pixel in the image
Krizhevsky, Alex, Ilya Sutskever, and Geoffrey E. Hinton. "Imagenet classification with deep convolutional neural networks." Advances in neural information processing systems
Labeling every pixel in the image with object class it belongs to.
Chen, Liang-Chieh, et al. "Semantic image segmentation with deep convolutional nets and fully connected crfs." arXiv preprint arXiv: (2014).

3. Scene labeling Problem
Spatiotemporal segmentation:
Challenge: Solving segmentation and recognition simultaneously
X t
Labeling every pixel in the image with object class it belongs to.
Seguin, Guillaume, et al. "Instance-level video segmentation from object tracks." (2016).

4. Patch wise training & patch wise prediction
Slow due to redundant computation
Li, Hongsheng, Rui Zhao, and Xiaogang Wang. "Highly efficient forward and backward propagation of convolutional neural networks for pixelwise classification." arXiv preprint arXiv: (2014).

5. Fully convolutional Networks
Long, Jonathan, Evan Shelhamer, and Trevor Darrell. "Fully convolutional networks for semantic segmentation." Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition

6. Motivation Motivation
CNN does not have an explicit mechanism to modulate feature with context (CRF)
Need to model the relationship between labels

7. Contextual information is important

8. Idea of recurrent CNN
Providing feedback from the output into the input allows the network to model label dependencies, and correct its own previous predictions
Ensuring the object coherence in scene labeling

9. Recurrent CNN Model 1 Model 2
Liang, Ming, and Xiaolin Hu. "Recurrent convolutional neural network for object recognition." Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition
Pinheiro, Pedro HO, and Ronan Collobert. "Recurrent Convolutional Neural Networks for Scene Labeling." ICML

11. =? Weights sharing Recurrent RCL Residual Network Weight Layer X +

12. determining the label of a pixel in an image
The model is able to
perform local feature extraction and context integration simultaneously in each parameterized layer, therefore particularly fits this application because both local and global information are critical for
determining the label of a pixel in an image

15. Spatiotemporal sequence prediction

16. Problem Goal: Predicting the future rainfall intensity in a
local region over a relatively short period of time
M x N 2D space , p measurements
Predict the most likely length-K sequence in the future given the previous J observations

17. 1D LSTM to 2D conv LSTM
Donahue, Jeffrey, et al. "Long-term recurrent convolutional networks for visual recognition and description." Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition

21. Combining Fully Convolutional and Recurrent Neural networks for 3D Biomedical Image Segmentation

22. Problem
3D biomedical images are often anisotropic: high solution in x-y axis but low in Z axis
Previous Methods:
2D convolutional for each slice and then followed by concatenating them into 3D
3D convolution
2D-3D hybrid
Approach
Combine FCN (u-Net) and LSTM (BDC-LSTM) to Exploit intra-slice and inter-slice contexts
High resolution
Low resolution

23. U-Net: Biomedical image segmentation

24. kU-Net

25. CLSTM

27. Nz slices - -> kU-Net --> 64 x Nx x Ny feature map f2dZ
f2dZ --> BDC-LSTM ----> f3dZ -->softmax -->prob
Decoupling kU-Net and BDC-LSTM training due to GPU memory and context consideration

29. Thank you!
Questions ?