Download presentation
Presentation is loading. Please wait.
1
Recurrent Neural Networks (RNNs)
2
Today’s Class Recurrent Neural Network Cell
Recurrent Neural Networks (RNNs) Bi-Directional Recurrent Neural Networks (Bi-RNNs) Multiple-layer / Stacked / Deep Bi-Direction Recurrent Neural Networks LSTMs and GRUs. Applications in Vision: Caption Generation.
3
Geoff Hinton Backpropagation
ACM Turing Award 2019 Geoff Hinton Backpropagation Yann LeCun CNNs Yoshua Bengio GANs
4
Recurrent Neural Network Cell
𝑅𝑁𝑁 ℎ 0 ℎ 1 𝑥 1
5
Recurrent Neural Network Cell
ℎ 1 =tanh( 𝑊 ℎℎ ℎ 0 + 𝑊 ℎ𝑥 𝑥 1 ) 𝑅𝑁𝑁 ℎ 0 ℎ 1 𝑥 1
6
Recurrent Neural Network Cell
𝑦 1 ℎ 1 𝑅𝑁𝑁 ℎ 0 ℎ 1 ℎ 1 =tanh( 𝑊 ℎℎ ℎ 0 + 𝑊 ℎ𝑥 𝑥 1 ) 𝑥 1 𝑦 1 =softmax( 𝑊 ℎ𝑦 ℎ 1 )
7
Recurrent Neural Network Cell
𝑦 1 ℎ 1 𝑅𝑁𝑁 ℎ 0 ℎ 1 𝑥 1
8
Recurrent Neural Network Cell
𝑦 1 =[0.1, 0.05, 0.05, 0.1, 0.7] ℎ 1 =[ −0.3 −0.1 ] 𝑅𝑁𝑁 ℎ 0 =[ ] 𝑥 1 = [ ] a b c d e c
9
Recurrent Neural Network Cell
𝑦 1 ℎ 1 𝑅𝑁𝑁 ℎ 0 ℎ 1 𝑥 1
10
Recurrent Neural Network Cell
ℎ 1 𝑅𝑁𝑁 ℎ 0 ℎ 1 𝑥 1
11
(Unrolled) Recurrent Neural Network
<<space>> 𝑦 1 𝑦 2 𝑦 3 ℎ 1 ℎ 2 ℎ 3 ℎ 0 𝑅𝑁𝑁 ℎ 1 𝑅𝑁𝑁 ℎ 2 𝑅𝑁𝑁 ℎ 3 𝑥 1 𝑥 2 𝑥 3 c a t
12
(Unrolled) Recurrent Neural Network
cat likes eating 𝑦 1 𝑦 2 𝑦 3 ℎ 1 ℎ 2 ℎ 3 ℎ 0 𝑅𝑁𝑁 ℎ 1 𝑅𝑁𝑁 ℎ 2 𝑅𝑁𝑁 ℎ 3 𝑥 1 𝑥 2 𝑥 3 the cat likes
13
(Unrolled) Recurrent Neural Network
positive / negative sentiment rating 𝑦 ℎ 3 ℎ 0 𝑅𝑁𝑁 ℎ 1 𝑅𝑁𝑁 ℎ 2 𝑅𝑁𝑁 ℎ 3 𝑥 1 𝑥 2 𝑥 3 the cat likes
14
Bidirectional Recurrent Neural Network
gato quiere comer 𝑦 1 𝑦 2 𝑦 3 ℎ 1 ℎ 2 ℎ 3 ℎ 0 𝐵𝑅𝑁𝑁 ℎ 1 B𝑅𝑁𝑁 ℎ 2 𝐵𝑅𝑁𝑁 ℎ 3 𝑥 1 𝑥 2 𝑥 3 the cat wants
15
Stacked Recurrent Neural Network
𝑦 1 𝑦 2 𝑦 3 ℎ 1 ℎ 2 ℎ 3 ℎ 0 𝑅𝑁𝑁 ℎ 1 𝑅𝑁𝑁 ℎ 2 𝑅𝑁𝑁 ℎ 3 ℎ 1 ℎ 2 ℎ 3 ℎ 0 𝑅𝑁𝑁 ℎ 1 𝑅𝑁𝑁 ℎ 2 𝑅𝑁𝑁 ℎ 3 𝑥 1 𝑥 2 𝑥 3 c a t
16
Stacked Bidirectional Recurrent Neural Network
𝑦 1 𝑦 2 𝑦 3 ℎ 1 ℎ 2 ℎ 3 ℎ 0 𝑅𝑁𝑁 ℎ 1 𝑅𝑁𝑁 ℎ 2 𝑅𝑁𝑁 ℎ 3 ℎ 1 ℎ 2 ℎ 3 ℎ 0 𝑅𝑁𝑁 ℎ 1 𝑅𝑁𝑁 ℎ 2 𝑅𝑁𝑁 ℎ 3 𝑥 1 𝑥 2 𝑥 3 c a t
17
RNN in Pytorch
18
LSTM Cell (Long Short-Term Memory)
ℎ 0 ℎ 1 𝐿𝑆𝑇𝑀 𝑐 0 𝑐 1 𝑥 1
19
LSTM in Pytorch
20
GRU in Pytorch
21
Questions?
Similar presentations
