1. Inception and Residual Architecture in Deep Convolutional Networks
Wenchi Ma
Computer Vision Group
EECS,KU

2. Inception: From NIN to Googlenet
micro network
Enhance the abstraction ability of the local model
A general nonlinear function approximator
Better results in image recognition and detection

3. Residual learning: for deeper neural networks
With the network depth gets increasing, accuracy gets saturated and then degrades!
Such degradation is not caused by overfitting, and adding more layers to a suitable deep model leads to higher training error
Residual learning: for deeper neural networks
Residual learning: a building block

4. Inception: Balance model size and computation cost
Deeper:
(a)integrate low/mid/high-level features and classifiers
(b)the “levels” of features can be enriched by the number of stacked layers(depth)
Wider:
More powerful ability of local abstraction
Contradiction: Increasing model size and computational cost tend to translate to immediate quality gains for most tasks wile computational efficiency decreases and high parameter count suffers

5. Inception: Balance model size and computation cost
Main source of computation load : high dimensional convolution
Higher dimensional representations are easier to process locally within a network.
Increasing the activations per tile in a convolutional network allows for more disentangled features. The resulting networks will train faster
General principles:
Avoid representational bottlenecks, especially early in the network
Maintain higher dimensional representations
Balance the width and depth of the network
The representation size should gently decrease from the inputs and outputs

6. Inception: Balance model size and computation cost
fully-connected
convolution
Same receptive filed
Mini-network replacing the convolutions
Mini-network replacing the convolutions
Less parameters
Same inputs and outputs
3*3
3*1
1*3

7. Inception: Efficient Grid Size Reduction
Expensive computation
bottleneck

8. Inception-v3(factorization idea)

9. Inception-v3 Inception-v1 Inception-v2
All evaluations are done on the non-blacklisted examples on the ILSVRC-2012 validation set(227)
Train the networks with stochastic gradient utilizing the TensorFlow distribution machine learning system using 50 replicas each on a Nvidia Kepler GPU
BN:The fully connected layer is also batch-normalized

10. Inception-V4

11. Residual connections and Inception ResNet model

12. Inception-ResNet-v1 and Inception-ResNet-v2 networks

13. Inception-ResNet-v1 and Inception-ResNet-v2 networks
Top-5 error
Top-1 error
Data: ILSVRC-2012 validation set
Train the networks with stochastic gradient utilizing the TensorFlow distribution machine learning system using 20 replicas running each on a Nvidia Kepler GPU

14. Combination of Inception and Residual
Large scale, Deep Convolutional Network:
Ensure stable training(Residual)
Decrease the scale of the net as a whole(Less parameters)
More efficient computation
Improve the accuracy

15. Deeper and Wider but not Bigger
Thank you!