Inception and Residual Architecture in Deep Convolutional Networks Wenchi Ma Computer Vision Group EECS,KU

Inception: From NIN to Googlenet micro network Enhance the abstraction ability of the local model A general nonlinear function approximator Better results in image recognition and detection

Residual learning: for deeper neural networks With the network depth gets increasing, accuracy gets saturated and then degrades! Such degradation is not caused by overfitting, and adding more layers to a suitable deep model leads to higher training error Residual learning: for deeper neural networks Residual learning: a building block

Inception: Balance model size and computation cost Deeper: (a)integrate low/mid/high-level features and classifiers (b)the “levels” of features can be enriched by the number of stacked layers(depth) Wider: More powerful ability of local abstraction Contradiction: Increasing model size and computational cost tend to translate to immediate quality gains for most tasks wile computational efficiency decreases and high parameter count suffers

Inception: Balance model size and computation cost Main source of computation load : high dimensional convolution Higher dimensional representations are easier to process locally within a network. Increasing the activations per tile in a convolutional network allows for more disentangled features. The resulting networks will train faster General principles: Avoid representational bottlenecks, especially early in the network Maintain higher dimensional representations Balance the width and depth of the network The representation size should gently decrease from the inputs and outputs

Inception: Balance model size and computation cost fully-connected convolution Same receptive filed Mini-network replacing the 5 5 convolutions Mini-network replacing the 3 3 convolutions Less parameters Same inputs and outputs 3*3 3*1 1*3

Inception: Efficient Grid Size Reduction Expensive computation bottleneck

Inception-v3(factorization idea)

Inception-v3 Inception-v1 Inception-v2 All evaluations are done on the 48238 non-blacklisted examples on the ILSVRC-2012 validation set(227) Train the networks with stochastic gradient utilizing the TensorFlow distribution machine learning system using 50 replicas each on a Nvidia Kepler GPU BN:The fully connected layer is also batch-normalized

Inception-V4

Residual connections and Inception ResNet model

Inception-ResNet-v1 and Inception-ResNet-v2 networks

Inception-ResNet-v1 and Inception-ResNet-v2 networks Top-5 error Top-1 error Data: ILSVRC-2012 validation set Train the networks with stochastic gradient utilizing the TensorFlow distribution machine learning system using 20 replicas running each on a Nvidia Kepler GPU

Combination of Inception and Residual Large scale, Deep Convolutional Network: Ensure stable training(Residual) Decrease the scale of the net as a whole(Less parameters) More efficient computation Improve the accuracy

Deeper and Wider but not Bigger Thank you!