Speakers: Luo Mai, Jingqing Zhang TensorLayer: A Versatile Deep Learning Library for Developers and Scientists Speakers: Luo Mai, Jingqing Zhang

Deep Learning Frameworks Key reasons for success Simplicity & Performance Large production adoption Large community ~ 115000 GitHub stars ~ 22000 GitHub stars ~ 15000 GitHub stars ~ 15000 GitHub stars

Wrapper libraries that bridge the gap Abstraction Mismatch Deep learning high-level elements: neural networks stacked with layers TensorFlow low-level API: dataflow graph, placeholder, session, queue runner, devices … Wrapper libraries that bridge the gap TFLearn

TensorLayer – Deep Learning Toolbox Design Training Deployment Reference networks Abstract layers Diverse learning functions Customizing layers Data augmentation Training control Distributed training Embedded platforms Model management

Comparison with Existing Libraries TensorLayer Keras TF-Learn Abstraction Modest High Flexibility Excellent Limited Portability No Yes Performance Good Deployment TensorLayer is a unique choice for its simplicity, flexibility and performance.

Flexible API to Build Complex Networks def model(x : tf.Tensor, train : bool) -> tl.Layer: with tf.variable_scope(”model”, reuse=TF.AUTO_REUSE): net = tl.InputLayer(x) net = tl.Conv2d(net, 64, (5, 5), (1, 1), padding='SAME') net = tl.BatchNormLayer(net, is_train=train, act=lambda: x:5*x) net = tl.MaxPool2d(net, (3, 3), (2, 2), pool=tf.nn.max_pool) net = tl.LambdaLayer(net, fn=tf.keras.Dropout) net = tl.DenseLayer(net, 10) return net Hide low-level TF APIs Flexible API Customizable Import Keras and TF-Slim TensorLayer has 100+ layers for computer vision, reinforcement learning and natural language processing ….

Zero-cost Abstraction Titan X Pascal GPU 2017 When training common CNNs, TensorLayer is usually 1.2 – 5x faster than Keras implementations

Scaling Training: Default Approach Synchronized Stochastic Gradient Descent (S-SGD) Extra works to include parameter server in TF programs Requirements from TensorLayer users: Minimal changes to user programs Homogeneous cluster High-performance Parameter Servers Parameters Local gradients GPU workers Optimal resource configuration is hard to determine

Scaling Training: TensorLayer + Horovod S-SGD through peer-to-peer communication Aggregate gradients Broadcast parameters GPU Workers Initialize environment Enable synchronization

Scaling Training: What Next? S-SGD is a weak scaling approach Experiment of training ResNet-32 Training collapse with large batch size Increased batch size does not reduce training time

Scaling Training: What Next? “If the training rate of some models is restricted to small batch sizes, then we will need to find other algorithmic and architectural approaches to their acceleration” – Jeff Dean & David Patterson

Scaling Training: What Next? Imperial College researchers are working towards strong scaling Constantly reduce training time Converge faster TensorLayer adopt timely research results to resolve practical AI problems

December 2018 ~ 4500 Stars ~ 1000 Forks ~ 70 Contributors Impact - Github December 2018 ~ 4500 Stars ~ 1000 Forks ~ 70 Contributors

Impact - Media and Award

Computer Vision OpenPose Plus TensorLayer Real-time pose estimation Multi-person keypoint detection Applications Automated grocery stores Security surveillance Smart home Deep Learning CNN VGG, VGG tiny, MobileNet TensorLayer Flexible architecture Distributed training TensorRT compatibility OpenPose Plus https://github.com/tensorlayer/openpose-plus

https://github.com/tensorlayer/srgan Adversarial Learning Super Resolution Generating realistic super-resolution images from their low-resolution counterparts. Applications Image recovery Image compression Deep Learning SR-GAN ResNet VGG TensorLayer Pretrained model Adversarial training   https://github.com/tensorlayer/srgan

Deep Poincare Map (DPM) for MRI Segmentation Medical Imaging Deep Poincare Map (DPM) for MRI Segmentation Cardiac segmentation in MRI Applications Medical imaging analysis Image segmentation Deep Learning CNN TensorLayer High performance data augmentation Visualization The Deep Poincare Map: A Novel Approach for Left Ventricle Segmentation. Y. Mo, F. Liu et al. MICCAI 2018.

Natural Language Processing Semantic Image Synthesis Image manipulation by natural language Applications Image manipulation Deep Learning VGG GAN LSTM TensorLayer More examples waiting for your exploration! Semantic Image Synthesis via Adversarial Learning. H. Dong, S. Yu et al. ICCV 2017.

Tutorials and Examples Actor-Critic DDPG RNN Distributed Training SRGAN A3C Policy Gradient VAE Inception Word2vec GAN-CLS Text generation Chatbot TRPO Data augmentation Autoencoder Anti-Spam DCGAN DAGGER LSTM CNN VAE-GAN Translation Semantic Image Synthesis Deep Q Network VGG Image-to-image translation Brain tumor segmentation

https://github.com/tensorlayer Luo Mai: Jingqing Zhang: luo.mai11@imperial.ac.uk  jz9215@imperial.ac.uk  https://github.com/tensorlayer