Introduction to Recurrent neural networks (RNN), Long short-term memory (LSTM) Wenjie Pei In this coffee talk, I would like to present you some basic knowledge about the RNN and LSTM models. I was going to present a paper about RNN model, but I think it is better to give you a introduction to these models first and hence you can have a overview about these models.

Artificial Neural Networks Feedforward neural networks ANNs without cycle connections between nodes (Feedback) Recurrent neural networks ANNs with cycle connections between nodes

Feedforward Neural Networks Multilayer perceptron (MLP) Universal function approximation theory: Sufficient nonlinear hidden units  approximate any continuous mapping function For each node here, it first calculate the weighted sum over all the inputs and then process it by the activation function. So can we see that this model is very powerful, right? But the Drawback is that it only Take into account Drawback: Output depends only on the current input No temporal information dependencies

Recurrent Neural Networks Feedback from hidden unit activation of last time step to current time step Universal approximation theory: An RNN with sufficient hidden units Any measurable sequence-to-sequence mapping or dynamic system From MLP to RNNs is a transition from static process to dynamic process which takes into account the time dimension. Advantage: Memory of previous inputs Incorporate contextual information

Recurrent Neural Networks Bidirectional RNNs

Recurrent Neural Networks Vanishing gradient problem This model cannot have a longer memory 2 reasons: 1. activation function 2. diluted by other input Sensitivity decay exponentially over the time

Long Short-Term Memory (LSTM) Input gate [0, 1]: How much information from input could go into the cell Forget gate [0, 1]: How much information from last time step could enter the cell Output gate [0, 1]: How much information to output In this model, it replaces the hidden layer by the memory blocks, in each block, there could be several memory cells. Here is the model for one cell. We can see that it has input and output and three gates: What is the roles of these three gates? All the values of these gate are from 0 to 1.

Long Short-Term Memory Advantage: long-period time memory Conveyed without decay

Applications Applications to sequence labeling problems: Handwritten character recognition Speech recognition Protein secondary structure prediction … Want to know more about the latest papers: Waiting for my next coffee talk