Download presentation
Presentation is loading. Please wait.
1
Speakers: Luo Mai, Jingqing Zhang
TensorLayer: A Versatile Deep Learning Library for Developers and Scientists Speakers: Luo Mai, Jingqing Zhang
2
Deep Learning Frameworks
Key reasons for success Simplicity & Performance Large production adoption Large community ~ GitHub stars ~ GitHub stars ~ GitHub stars ~ GitHub stars
3
Wrapper libraries that bridge the gap
Abstraction Mismatch Deep learning high-level elements: neural networks stacked with layers TensorFlow low-level API: dataflow graph, placeholder, session, queue runner, devices … Wrapper libraries that bridge the gap TFLearn
4
TensorLayer – Deep Learning Toolbox
Design Training Deployment Reference networks Abstract layers Diverse learning functions Customizing layers Data augmentation Training control Distributed training Embedded platforms Model management
5
Comparison with Existing Libraries
TensorLayer Keras TF-Learn Abstraction Modest High Flexibility Excellent Limited Portability No Yes Performance Good Deployment TensorLayer is a unique choice for its simplicity, flexibility and performance.
6
Flexible API to Build Complex Networks
def model(x : tf.Tensor, train : bool) -> tl.Layer: with tf.variable_scope(”model”, reuse=TF.AUTO_REUSE): net = tl.InputLayer(x) net = tl.Conv2d(net, 64, (5, 5), (1, 1), padding='SAME') net = tl.BatchNormLayer(net, is_train=train, act=lambda: x:5*x) net = tl.MaxPool2d(net, (3, 3), (2, 2), pool=tf.nn.max_pool) net = tl.LambdaLayer(net, fn=tf.keras.Dropout) net = tl.DenseLayer(net, 10) return net Hide low-level TF APIs Flexible API Customizable Import Keras and TF-Slim TensorLayer has 100+ layers for computer vision, reinforcement learning and natural language processing ….
7
Zero-cost Abstraction
Titan X Pascal GPU 2017 When training common CNNs, TensorLayer is usually 1.2 – 5x faster than Keras implementations
8
Scaling Training: Default Approach
Synchronized Stochastic Gradient Descent (S-SGD) Extra works to include parameter server in TF programs Requirements from TensorLayer users: Minimal changes to user programs Homogeneous cluster High-performance Parameter Servers Parameters Local gradients GPU workers Optimal resource configuration is hard to determine
9
Scaling Training: TensorLayer + Horovod
S-SGD through peer-to-peer communication Aggregate gradients Broadcast parameters GPU Workers Initialize environment Enable synchronization
10
Scaling Training: What Next?
S-SGD is a weak scaling approach Experiment of training ResNet-32 Training collapse with large batch size Increased batch size does not reduce training time
11
Scaling Training: What Next?
“If the training rate of some models is restricted to small batch sizes, then we will need to find other algorithmic and architectural approaches to their acceleration” – Jeff Dean & David Patterson
12
Scaling Training: What Next?
Imperial College researchers are working towards strong scaling Constantly reduce training time Converge faster TensorLayer adopt timely research results to resolve practical AI problems
13
December 2018 ~ 4500 Stars ~ 1000 Forks ~ 70 Contributors
Impact - Github December 2018 ~ 4500 Stars ~ 1000 Forks ~ 70 Contributors
14
Impact - Media and Award
15
Computer Vision OpenPose Plus TensorLayer Real-time pose estimation
Multi-person keypoint detection Applications Automated grocery stores Security surveillance Smart home Deep Learning CNN VGG, VGG tiny, MobileNet TensorLayer Flexible architecture Distributed training TensorRT compatibility OpenPose Plus
16
https://github.com/tensorlayer/srgan
Adversarial Learning Super Resolution Generating realistic super-resolution images from their low-resolution counterparts. Applications Image recovery Image compression Deep Learning SR-GAN ResNet VGG TensorLayer Pretrained model Adversarial training
17
Deep Poincare Map (DPM) for MRI Segmentation
Medical Imaging Deep Poincare Map (DPM) for MRI Segmentation Cardiac segmentation in MRI Applications Medical imaging analysis Image segmentation Deep Learning CNN TensorLayer High performance data augmentation Visualization The Deep Poincare Map: A Novel Approach for Left Ventricle Segmentation. Y. Mo, F. Liu et al. MICCAI 2018.
18
Natural Language Processing
Semantic Image Synthesis Image manipulation by natural language Applications Image manipulation Deep Learning VGG GAN LSTM TensorLayer More examples waiting for your exploration! Semantic Image Synthesis via Adversarial Learning. H. Dong, S. Yu et al. ICCV 2017.
19
Tutorials and Examples
Actor-Critic DDPG RNN Distributed Training SRGAN A3C Policy Gradient VAE Inception Word2vec GAN-CLS Text generation Chatbot TRPO Data augmentation Autoencoder Anti-Spam DCGAN DAGGER LSTM CNN VAE-GAN Translation Semantic Image Synthesis Deep Q Network VGG Image-to-image translation Brain tumor segmentation
20
https://github.com/tensorlayer
Luo Mai: Jingqing Zhang:
Similar presentations
