1. Lecture 3a Analysis of training of NN

2. Agenda Analysis of deep networks Variance analysis
Non-linear units
Weight initialization
Local Response Normalization (LRN)
Batch Normalization

3. Understanding the difficulty of training convolutional networks
The key idea : debug training through monitoring of
mean and variance of activation variables 𝑦 𝑙 (output of non-linear unit)
mean and variance of gradients 𝜕 𝐸 𝜕 𝑦 𝑙−1 , and 𝜕 𝐸 𝜕 𝑊 𝑙 .
Reminder: variance of x: 𝑉𝑎𝑟 𝑥 =𝐸𝑥𝑝 (𝑥− 𝑥 ) 2 ,
We will compute scalar mean and variance for each layer, and then average over images in the test set.
X. Glorot ,Y. Bengio, Understanding the difficulty of training deep feedforward neural networks

4. Understanding the difficulty of training convolutional networks
Activation graph for MLP: 4 x [fully connected layers + sigmoid]
Mean and standard deviation of the activation (output of the sigmoid) during learning, for 4 hidden layers .
The top hidden layer quickly saturates at 0 (slowing down all learning), but then slowly desaturates ~ epoch 100.
Sigmoid is non-symmetric -> difficult to train

5. Understanding the non-linear function behavior
Let’s try MLP with symmetric non-linear functions: tanh & soft-sign
Tanh: 𝒆 𝒙 − 𝒆 −𝒙 𝒆 𝒙 + 𝒆 −𝒙 |
Soft-sign : 𝒙 𝟏+|𝒙|
98 % (markers only) and
standard deviation (solid lines with markers)

6. MLP: Debugging the forward path
How we can use variance analysis to debug NN training?
Let’ start with classical Multi Layer Perceptron (MLP)
Forward propagation for FC layer:
𝑦 𝑖 =𝑓( 𝑖=1 𝑛 𝑤 𝑖𝑗 ∗ 𝑥 𝑗 )
where
𝑥=[𝑥 𝑗 ] – layer inputs,
𝑛 – number of inputs
W – layer matrix
𝑦=[ 𝑦 𝑖 ] – layer outputs (hidden nodes)
Assume that 𝑓 is symmetric non-linear activation function. For ini
tial analysis we will ignore non-linear unit𝑓 and it’s derivative ( 𝑓 is not saturate and f’ ~ 1).

7. MLP: Debugging the forward path
Assume that:
all 𝑥 𝑗 are independent and have the variance 𝑉𝑎𝑟 𝑋 ,
all 𝑤 𝑖𝑗 are independent and have the variance 𝑉𝑎𝑟 𝑊 .
Then
𝑉𝑎𝑟 𝑦 = 𝑛 𝑖𝑛 ∗ 𝑉𝑎𝑟 𝑤 ∗ 𝑉𝑎𝑟(𝑥)
We want to keep the output 𝑦 at the same dynamic range as input 𝑥:
𝑛 𝑖𝑛 ∗ 𝑉𝑎𝑟 𝑤 =1  𝑉𝑎𝑟 𝑊 = 1 𝑛 𝑖𝑛
Xavier rule for weight initialization with uniform rand( ): 𝑊=𝑢𝑛𝑖_𝑟𝑎𝑛𝑑 − 3 𝑛 𝑖𝑛 ; 𝑛 𝑖𝑛
X. Glorot ,Y. Bengio, Understanding the difficulty of training deep feedforward neural networks

8. MLP: Debugging the backward propagation
Backward propagation of gradients: 𝜕 𝐸 𝜕𝑥 = 𝜕 𝐸 𝜕𝑦 ∗ 𝑤 𝑇 ; 𝜕 𝐸 𝜕𝑤 = 𝜕 𝐸 𝜕𝑦 ∗𝑥 Then 𝑉𝑎𝑟 𝜕𝐸 𝜕𝑥 = 𝑛 𝑜𝑢𝑡 ∗𝑉𝑎𝑟 𝑤 ∗ 𝑉𝑎𝑟( 𝜕𝐸 𝜕𝑦 ) 𝑉𝑎𝑟 𝜕𝐸 𝜕𝑤 = 𝑉𝑎𝑟 𝑥 ∗𝑉𝑎𝑟( 𝜕𝐸 𝜕𝑦 ) We want to keep gradients 𝜕 𝐸 𝜕𝑥 from vanishing and from exploding: 𝑛 𝑜𝑢𝑡 ∗𝑉𝑎𝑟 𝑤 =1  𝑉𝑎𝑟 𝑊 = 1 𝑛 𝑜𝑢𝑡 . Combining with formula from forward path: 𝑉𝑎𝑟 𝑊 = 2 𝑛 𝑖𝑛 +𝑛 𝑜𝑢𝑡 ; Xavier rule 2 for weight initialization with uniform rand( ): 𝑊=𝑢𝑛𝑖_𝑟𝑎𝑛𝑑 − 6 𝑛 𝑖𝑛 + 𝑛 𝑜𝑢𝑡 ; 6 𝑛 𝑖𝑛 + 𝑛 𝑜𝑢𝑡

9. Extension of gradient analysis for convolutional networks
Convolutional layer: Forward propagation: 𝑌 𝑖 =𝑓( 𝑗=1 𝑀 𝑊 𝑖𝑗 ∗ 𝑋 𝑗 ) where: 𝑌 𝑖 - output feature map (H’ x W’) 𝑊 𝑖𝑗 - convolutional filter (K x K) 𝑋 𝑗 - input feature map (H x W) 𝑊 𝑖𝑗 ∗ 𝑋 𝑖 - convolution of input feature map X with filter W M – number of input features (each feature map H x W ) Backward propagation: 𝜕𝐸 𝜕 𝑋 𝑗 = 𝑖=1 𝑁 𝜕𝐸 𝜕 𝑌 𝑖 ∗ 𝑊 𝑖𝑗 , 𝜕𝐸 𝜕 𝑊 𝑖𝑗 = 𝜕𝐸 𝜕 𝑌 𝑖 * 𝑋 𝑗 Here : * is convolution.

10. Extension of gradient analysis for convolutional networks
Convolutional layer: Forward propagation: 𝑉𝑎𝑟 𝑌 = 𝒏 𝒊𝒏 ∗ 𝑉𝑎𝑟 𝑊 ∗𝑉𝑎𝑟(𝑋) 𝒏 𝒊𝒏 = # input feature maps * k2 Backward propagation: 𝑉𝑎𝑟 𝜕𝐸 𝜕𝑥 = 𝑛 𝑜𝑢𝑡 ∗ 𝑉𝑎𝑟 𝑊 ∗𝑉𝑎𝑟( 𝜕𝐸 𝜕𝑦 ) 𝒏 𝒐𝒖𝒕 = # output feature maps * k2 For weight gradients: 𝑉𝑎𝑟 𝜕𝐸 𝜕𝑤 ~ (𝑯∗𝑾)∗ 𝑉𝑎𝑟 𝑋 ∗𝑉𝑎𝑟( 𝜕𝐸 𝜕𝑦 ) We can compensate (𝑯∗𝑾) with layer learning rate

11. Local Contrast Normalization
Local Contrast Normalization - can be performed on the state of every layer, including the input
Subtractive Local Contrast Normalization:
Subtracts from every value in a feature a Gaussian-weighted average of its neighbors (high-pass filter)
Divisive Local Contrast Normalization
Divides every value in a layer by the standard deviation of its neighbors over space and over all feature maps
Subtractive + Divisive LCN

12. Local Response Normalization Layer
LRN layer “damps” responses that are too large by normalization in a local neighborhood inside the feature map:
𝑦 𝑙 𝑥,𝑦 = 𝒚 𝒍−𝟏 𝑥,𝑦 𝟏+ 𝛼 𝑵 𝟐 ∗ 𝑥 ′ =𝑥−𝑁/2 𝑥+𝑁/2 𝑦 ′ =𝑦−𝑁/2 𝑦+𝑁/2 𝑦 𝑙−1 ( 𝑥 ′ , 𝑦 ′ ) 2
where :
y l−1 𝑥,𝑦  is the activity map prior to normalization,
N is the size of the region to use for normalization.
- 1 is used to prevent numerical issues for small numbers.

13. Local Response Normalization Layer
Soft Max
Inner Product
LRN layer
Pooling
ReLUP
Convolutional layer
LRN layer
Pooling
ReLUP
Convolutional layer
Data Layer

14. Batch Normalization Layer
Layer which normalizes the output of conv. layer before non-linear layer:
Whitening: normalize each element of feature map over mini-batch. All locations of the same feature map are normalized in the same way.
Adaptive scale γ and shift β (per map) – learned parameters
S. Ioffe, C. Szegedy Batch Normalization: Accelerating Deep Network Training , 2015

15. Batch Normalization Layer
Soft Max
Inner Product
Pooling
ReLUP
Batch Normalization layer
Convolutional layer
Pooling
ReLUP
Batch Normalization layer
Convolutional layer
Data Layer

16. Batch Normalization: training
Back-propagation for BN layer:
Implemented in caffe:

17. Batch Normalization: inference
During inference we don’t have batch to normalize, so we use instead fixed mean and variance over all train set: 𝑥 = 𝑥−𝐸[𝑥] 𝑉𝑎𝑟 𝑥 + 𝜖 For testing during training we can use estimation of E[x] and Var[x]:

18. Batch Normalization: performance
Networks with batch normalization train much faster :
Much high learning rate with fast exponential decay
No need in LRN
Baseline: caffe cifar_full
VGG-16:caffe VGG_ILSVRC_16

19. Batch Normalization: Imagenet performance
Models:
Goog;le Inception:(ILSVRC 2014) with the learning rate of
BN-Baseline: Inception + Batch Normalization before each ReLU
BN-x5: Inception + Batch Normalization w/o dropout and LRN. The initial learning rate was increased by 5x to
BN-x30: Like BN-x5, but with the initial learning rate (30x of Inception).
BN-x5-Sigmoid: Like BN-x5, but with sigmoid instead of ReLU