Progress Report of Apache SINGA Wei 28/09/2015

Features in the first version Usability – DL Models Feed-forward models – CNN, MLP, Auto-encoder RNN – Vanilla RNN for language modeling RBM – Tools Zookeeper for job management GNU auto tool for installation Scalability/Efficiency – Training frameworks (CPU) Synchronous – AllReduce, Sandblaster Asynchronous – Downpour – Distributed Hogwild!

Features in the second version Usability Scalability/efficiency

Usability Adding support for popular models – Feed-forward models AlexLet, GoogleLet and VGG for image classification Fast RCNN for Object detection – Recurrent neural networks LSTM and GRU – Energy models Skip in this version.

Usability Helpers for model configuration – current job configuration file is large Each layer takes 5-20 lines Some models have more than 10 layers – Write helpers (builder) for popular models MLP::AddTanhLayer(layer_size), MLP::AddLogisticLayer(layer_size) – conf=MLP::AddTanhLayer(100).AddTanhLayer(200).AddTanhLayer(300); ConvNet::AddConv(), ConvNet::AddPool(), etc. More built-in input and output layers – CVS format files – HDFS as input and output sources

Usability Integration with Mesos and Docker – Mesos Manages cluster resources (e.g., CPU and memory) Support multiply jobs concurrently – Docker Package dependent libs for deployment on Amazon EC2 or other environments. Python Binding – Data scientists prefer python – Create python wrappers for major components E.g., Layer, Updater Performance logging/report – Try to remove dependency on GLOG – Format logging messages

Scalability and Efficiency GPU support – Unified math interface for CPU and GPU Add, Sub, Dot, Mult, etc. Underlying implementation – OpenBLAS or MKL (Intel) or ATLAS for CPU – cuBLAS + cuDNN for GPU – Extend the worker-stub-server framework GPU-CPU memory management – Compilation

Development Flow Create JIRA ticket – Describe the changes to be made or has been done. Git checkout feature/fixbug/improve-xxx – Git commit do not commit too frequently Make commit msg as clear and clean as possible Put JIRA ticket No. and title as the first line, and put one blank line after it – Git checkout master Git pull asf master – Git checkout working branch Git rebase (-i) – Git push origin … Goto you github website, send pull request. – Title is the same as the jira ticket – Add detailed description

Continue with tickets by others Git remote add – Bob created the ticket Git fetch bob : Git checkout Work on that branch – Commit, commit… – Git pull bob or – git fetch bob : -1 – Git rebase -1 – Git push bob

Merge Pull Requests Git fetch pull/ /head:feature- foo Or Git fetch bob feature-foo:feature-foo Bob is the guy who sent the pull request foo is the branch name you want to call it – Git checkout master – Git pull asf master – Git merge feature-foo – Git push asf master