Minerva: A Scalable and Highly Efficient Training Platform for Deep Learning M. Wang, T. Xiao, J. Li, J. Zhang, C. Hong, & Z. Zhang (2014) Presentation by Cameron Hamilton

Overview Problem: disparity between deep learning tools oriented towards productivity/generality (e.g. MATLAB) and task-specific tools designed for speed and scale (e.g. CUDA-Convnet). Solution: A matrix-based API, known as Minerva, with a MATLAB-like procedural coding style. Program is translated into an internal dataflow graph at runtime, which is generic enough to be implemented on different types of hardware.

Minerva System Overview Every training iteration has two phases Generate dataflow graph from user code Evaluate dataflow graph

Example of User Code

System Overview: Performance via Parallelism Performance of deep learning algorithms dependent on whether operations can be performed in parallel. Minerva utilizes two forms of parallelism: Model parallelism: model replicas used to train the same model Replicas exchange updates via “logically centralized parameter server” (p. 4). Data parallelism: model replicas assigned to different portions of the data sets Always evaluates on GPU if available

Programming Model Minerva API  3 stages for deep learning Define model architecture Model model; Layer layer1 = model.AddLayer(dim); model.AddConnection(layer1,layer2,FULL); Declaring primary matrices (i.e. weights & biases) Matrix W = Matrix(layer2,layer1,RANDOM); Matrix b(layer2,1,RANDOM); Vector<Matrix> inputs = LoadBatches(layer1,…);

Programming Model Specifying training procedure Convolutional neural networks (CNNs) are specified with a different syntax. The architecture is specified with a single line: AddConvConnect(layer1,layer2,…). Minerva then handles the arrangement of these layers (p.4).

Programming Model Expressing Parallelism Model Parallelism SetPartition(layer1,2);SetPartition(layer2,2); Data Parallelism ParameterSet pset; pset.Add(“W”,W); pset.Add(“V”,V); pset.Add(“b”,b); pset.Add(“c”,c); RegisterToParameterServer(pset); …//Learning Procedure Here if(epoch % 3 == 0) PushToParameterServer(pset); if(epoch % 6 == 0) PullFromParameterServer(pset); EvalAll();

Putting it All Together

Putting it All Together

System Design: More on Parallelism Within a neural network, the operations that will occur at each computing vertex (i.e. forward propagation, backward propagation, weight update) are predefined. This allows for network training to be partitioned for theoretically any number of threads. Updates shared between local parameter servers Load-balance by dividing task up amongst partitions Coordination and Overhead by determining ownership of computing vertex based on location of its input and output vertices. Partitions stick to their vertices. Locality by receiving input to vertex in layer n from n-1 and outputting layer n+1

Model Parallelism

Convolutional Networks Partitions handle patches of the input data, the patches are merged, then convolved with a kernel.

More on Data Parallelism Each machine/partition has its own local parameter server that updates and exchanges with its neighbor servers. Coordination done through belief-propagation-like algorithm (p.7) Merge updates with neighbors, then server “gossips to each of them the missing portion”

Experience and Evaluation Minerva Implementation Highlights ImageNet (CNN) 1K classification task (Krzhevsky et al., 2012) 42.7% top-1 error rate 15x faster than MATLAB implementation 4.6x faster with 16-way partition on 16 core machine than no partitions Speech-net 1100 input neurons, 2000 sigmoid neurons x 8 hidden layers, 9000 softmax output layer 1.5-2x faster than MATLAB implementation RNN 10000 input, 1000 hidden, 10000 flat outputs

Experience and Evaluation Scaling Up (Figure 8): CNN using mini-batch size of 128 Minerva(GPU) trained faster than Caffee using 256 and 512 mini-batch sizes

Experience and Evaluation

Experience and Evaluation

Experience and Evaluation

Conclusion Powerful and versatile framework for big data and deep learning Pipeline may be more preferable than partitioned fully connected layers which cause traffic My Comments Lacks restricted Boltzmann machine (RBM) so deep belief network (DBN) is not currently possible API appears to be concise and readable Lacks implementation of algorithm for genetic design of network (e.g. NEAT), however population generation would be ideal for partitioning. Not clear how Minerva handles situations where partitions do not evenly divide # of nodes within a given layer

References Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2012). Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems (pp. 1097-1105). Wang, M., Xiao, T., Li, J., Zhang, J., Hong, C., & Zhang, Z. (2014). Minerva: A scalable and highly efficient training platform for deep learning. All figures appearing within this presentation are borrowed from Wang et al., 2014.