1. Convolutional Neural Fabrics by Shreyas Saxena, Jakob Verbeek
CS559 – Deep learning paper presentation
Presenter: Burak Eserol

2. Outline About Authors and Paper Introduction Related Work
Convolutional Neural Fabrics
Experimental Evaluation Results
Conclusion
References

3. About Authors and Paper
Shreyas Saxena
Phd student at INRIA – France
Interested in deep networks
Jakob Verbeek
Researcher in Computer Vision and Machine Learning at INRIA – France
Interested in machine learning
Advances in Neural Information Processing System (NIPS)
Dec 2016, Barcelona, Spain

4. Introduction Problem Definition Success of CNN
Krizhevsky – Hand crafted features, to end-to-end trainable systems
Problems with CNN
Lack of efficient systematic ways to explore the large architecture space
Number of layers, number of channels per layer, filter size per layer, stride per layer, number of pooling vs. convolutional layers, type of pooling operator per layer, size of the pooling regions, ordering of pooling and convolutional layers, channel connectivity pattern between layers, type of activation
Number of resulting architecture is huge

5. Introduction Problem Solution Convolutional Neural Fabric
All possible network architectures are embedded in a “fabric”
3 Dimensional
Response maps of various resolutions
Only local connections across neighboring layers, scales, channels
2 parameters – number of layers and channels

6. Introduction Learning Parameter Sharing Contributions of the Work
Back-Propagation – Learning is done efficiently
Parameter Sharing
Contributions of the Work
CNN model architecture selection problem
Massively shared weights between architectures
Can generate output at multiple resolutions
Image classification, semantic segmentation, multi-scale object detection within a single network structure.

7. Related Work
Several chain-structured CNN architectures – are they best?
[1] Knowledge Transfer – “Transferring the knowledge from a previous network to each new deeper or wider network” – Accelerate training
Lack of more effective methods to find good architectures than trying one-by-one
Architectures induce networks with multiple parallel paths from input to output.[2][3]
All such networks are embedded in fabric
Multi-Dimensional networks
[4] Cross-stitch networks that can produce two different outputs
Based on AlexNet[5] and two copies of the same architecture
Does not address the network architecture selection problem
[1] Chen et al., Accelerating learning via knowledge transfer, ICLR 2016
[2] Farabet et al., Learning hierarchical features for scene labeling, PAMI 2013
[3] Honari et al., Learning coarse to fine feature aggregation, CVPR 2016
[4] Misra et al. Cross-stich networks for multi-task learning. In CVPR, 2016.

8. Convolutional Neural Fabrics
Each node represents one response map
Layer Axis: Same output, Same filter (depth axis of CNN)
Scale Axis: Different resolutions are organized, stride is increased (go down), 0 padding applied (go up)
Channels Axis: Different response maps with same scale and layer
Number of Scales calculated as when input
E.g. 32x32 input, S = 6, to have 1x1

9. Convolutional Neural Fabrics
How various architectural choices can be “implemented” in fabrics?
Resampling Operators
Stride-two convolutions can be used on fine-to-coarse edges
Zero padding can be used on coarse-to-fine edges
Max-pooling
Filter Sizes
Repetition within the channels enable to obtain any size of filters

10. Convolutional Neural Fabrics
Ordering Convolution
Chain structured networks = paths in fabric
Weights on edges outside path are zero

11. Convolutional Neural Fabrics
Channel Connectivity Pattern
Most networks use dense connectivity across channels between successive layers
Fabric, sparsely connected along channel axis , suffices to emulate densely connected convolutional layers.
Copying channels
Convolving channels
Aggregating channels

12. Convolutional Neural Fabrics
Analysis of Number of Parameters and Activations
Number of Response Maps through-out the fabric
Number of Parameters when channels are sparsely or densely connected
Number of Activations – that determines the memory usage of back-propagation during learning
Example: 32x32 input -> they used L = 16 layers and C = 256 channels => 2M Parameters and 6M activations.
Example: 256x256 input -> L = 16 layers and C = 64 channels => 0.7M Parameters and 89M activations

13. Experimental Evaluation Results
Datasets and Their Results
Part Labels Dataset
~3000 face images from LFW
dataset with pixel level annotation
into classes hair, skin,
and background
95.6% super-pixel accuracy
using both sparse and dense
fabrics.

14. Experimental Evaluation Results
From left to Right
Input image
Ground truth labels
Superpixel-level prediction
Pixel-level prediction

15. Experimental Evaluation Results
MNIST
28x28 images of handwritten digits
50k training, 10k validation, 10k test
Pixel values are normalized to [0,1]
Error rates of 0.48% and 0.33% with sparse and dense fabrics

16. Experimental Evaluation Results
CIFAR10
32x32 images and 10 classes
45k training, 5k validation, 10k test
Error rate of 7.43%
Training of Fabric
SGD
Momentum of 0.9
After each node, Batch Normalization
Did not apply dropout
Use validation set to determine optimal number of training epochs

17. Conclusion Convolutional Neural Fabrics
Network architecture selection problem
Specifying, training, and testing of individual networks to find best ones
Subsume large class of networks
Image classification, and semantic segmentation
Fabrics are competitive with the best-crafted CNN architectures
Better optimization schemes Adam, dropout, dropconnect…

18. References
[1] T. Chen, I. Goodfellow, and J. Shlens. Net2net: Accelerating learning via knowledge transfer. In ICLR, 2016
[2] C.Farabet, C.Couprie, L.Najman,andY.LeCun. Learning hierarchical features for scene labeling. PAMI, 35(8):1915–1929, 2013.
[3] S. Honari, J. Yosinski, P. Vincent, and C. Pal. Recombinator networks: Learning coarse-to-fine feature aggregation. In CVPR, 2016.
[4] I. Misra, A. Shrivastava, A. Gupta, and M. Hebert. Cross-stich networks for multi-task learning. In CVPR, 2016.
[5] A. Krizhevsky, I. Sutskever, and G. Hinton. Imagenet classification with deep convolutional neural networks. In NIPS, 2012

19. Thank You