1. Deep Learning Libraries
Giuseppe Attardi

2. General DL Libraries
Theano
Tensorflow
Pytorch
Caffe
Keras

3. Caffe Deep Learning framework from Berkeley focus on vision
coding in C++ or Python
Network consists in layers:
layer = { name = “data”, …}
layer = { name = “conv”, ...}
data and derivatives flow through net as blobs

4. Theano
Python linear algebra compiler that optimizes symbolic mathematical computations generating C code
integration with NumPy
transparent use of a GPU
efficient symbolic differentiation

5. Example
creates symbolic variable
>>> x = theano.dscalar('x') >>> y = x ** 2 >>> gy = theano.grad(y, x) >>> f = theano.function([x], gy) >>> f(4) array(8.0)
looks like a normal expression

6. 2-layer Neural Network
z1 = X.dot(W1) + b1 # first layer a1 = tanh(z1) # activation z2 = a1.dot(W2) + b2 # second layer y_hat = softmax(z2) # output probabilities # class prediction prediction = T.argmax(y_hat, axis=1) # the loss function to optimize loss = categorical_crossentropy(y_hat, y).mean()

7. Gradient Descent
dW2 = T.grad(loss, W2) db2 = T.grad(loss, b2) dW1 = T.grad(loss, W1) db1 = T.grad(loss, b1) gradient_step = theano.function([X, y], updates = ((W2, W2 – learn_rate * dW2), (W1, W1 – learn_rate* dW1), (b2, b2 – learn_rate* db2), (b1, b1 – learn_rate* db1)))

8. Training loop
# Gradient descent. For each batch... for i in xrange(0, epochs): # updates parameters W2, b2, W1 and b1! gradient_step(train_X, train_y)

9. Theano outlook
Joshua Bengio, of MILA, decided to stop supporting Theano:

10. TensorFlow C++ core code using Eigen
Eigen template metaprogramming linear algebra generates code for CPU or GPU.
For example, if A, B and C are tensors:
A = B + C
avoids both nested loops and producing an intermediate matrix
Build architecture as dataflow graphs:
Nodes implement mathematical operations, or data feed or variables
Nodes are assigned to computational devices and execute asynchronously and in parallel
Edges carry tensor data between nodes

11. Training Gradient based machine learning algorithms
Automatic differentiation
Define the computational architecture, combine that with your objective function, provide data -- TensorFlow handles computing the derivatives

12. Performance Support for threads, queues, and asynchronous computation
TensorFlow allows assigning graph to different devices (CPU or GPU)

13. Keras

14. Keras
Keras is a high-level neural networks API, written in Python and capable of running on top of TensorFlow, CNTK, or Theano.
Guiding principles
User friendlyness, modularity, minimalism, extensibility, and Python-nativeness.
Keras has out-of-the-box implementations of common network structures:
convolutional neural network
Recurrent neural network
Runs seamlessly on CPU and GPU.

15. Outline Install Keras Preprocess input Define model Compile model
Import libraries
Load image data
Preprocess input
Define model
Compile model
Fit model on training data
Evaluate model

16. Install Keras Full instructions Basic instructions:
Install backend: Tensorflow or Theano
> pip install keras

17. 30 sec. example
from keras.models import Sequential from keras.layers import Dense model = Sequential() model.add(Dense(units=64, activation='relu', input_dim=100)) model.add(Dense(units=10, activation='softmax')) model.compile(loss='categorical_crossentropy', optimizer='sgd', metrics=['accuracy']) model.compile(loss=keras.losses.categorical_crossentropy, optimizer=keras.optimizers.SGD(lr=0.01, momentum=0.9, nesterov=True)) model.fit(x_train, y_train, epochs=5, batch_size=32) classes = model.predict(x_test, batch_size=128)

18. Models Model = layers, loss and an optimizer
Add layer to model, compile() and fit()
Model can be saved and chckpoimted for later use

19. Layers Layers are used to define the network Examples: Dense
Convolutional
Pooling
Dropout

20. Loss Function
Measures the error by comparing the network predicted output with the expected output
The loss must be minimized by udatng the weights thgough backpropagation
Common loss functions
Mean squared root
Cross-entropy

21. Optimizers
Optimizers are strategies used to update the weights during backpropagation
Examples:
RMSpro
Adam
AdaGrad

22. Backends Keras may use several backends: Exploits either: TensorFlow
Theano
CNTK
Exploits either:
CPU
GPU

23. Keras Resources Tutorial at: Cheat sheet:
Cheat sheet:

24. DeepNL Deep Learning library for NLP C++ with Eigen
Wrapper for Python, Java, PHP automatically generated with SWIG
Git repository