Presentation is loading. Please wait.

Presentation is loading. Please wait.

Search-Based Approaches to Accelerate Deep Learning

Published byRobyn Norman Modified over 5 years ago

Similar presentations

Presentation on theme: "Search-Based Approaches to Accelerate Deep Learning"— Presentation transcript:

1 Search-Based Approaches to Accelerate Deep Learning
Zhihao Jia 9/9/2019 Stanford University

2 Deep Learning is Everywhere
Convolutional Neural Networks Recurrent Neural Networks Neural Architecture Search Reinforcement Learning

3 Deep Learning Deployment is Challenging
Distributed Heterogenous Hardware Platforms Diverse and Complex DNN Models What operators to execute? How to distribute these operators?

4 Existing Approach: Heuristic Optimizations
Device 1 Device N DNN Architecture Graph Optimizations Parallelization Rule-based Operator Fusion Data/Model Parallelism Miss model- and hardware-specific optimizations Performance is suboptimal

5 Search-Based Optimizations
A search space of possible strategies A cost model and a search algorithm Optimized strategies + = Challenge 1: How to build a search space including optimized strategies? Challenge 2: How to efficiently explore the search space?

6 + = … Overview A search space of possible strategies A cost model and
Device 1 Device N Parallelization Graph Optimizations A search space of possible strategies The SOAP search space Auto-generated graph substitutions A cost model and a search algorithm + Markov Chain Monte Carlo Cost-based backtracking search Optimized strategies Fast parallelization strategies Optimized computation graphs = Outperform data/model parallelism by 3.3x Outperform rule-based operator fusion by 2.9x

7 + = … Overview A search space of possible strategies A cost model and
Device 1 Device N Parallelization Graph Optimizations A search space of possible strategies The SOAP search space Auto-generated graph substitutions A cost model and a search algorithm + Markov Chain Monte Carlo Cost-based backtracking search Optimized strategies Fast parallelization strategies Optimized computation graphs =

8 Beyond Data and Model Parallelism for Deep Neural Networks
ICML’18, SysML’19

9 Current Approaches: Data and Model Parallelism
Data parallelism is the default strategy in existing DNN frameworks Manually-designed strategies [1, 2] Combine data and model parallelism to accelerate specific DNNs Automatic generated strategies ColocRL [3] uses RL to find device placement for model parallelism Exploring dimensions beyond data and model parallelism can further accelerate DNN training (by up to 3.3x) [1] Alex Krizhevsky. One weird trick for parallelizing convolutional neural networks. 2014 [2] Wu et. al. Google’s neural machine translation system: Bridging the gap between human and machine translation. 2016 [3] Mirhoseini et. al. Device placement optimization with reinforcement learning. 2017

10 The SOAP Search Space Samples Operators Attributes Parameters

11 Parallelizing a 1D convolution
The SOAP Search Space Samples: partitioning training samples (Data Parallelism) Operators Attributes Parameters Pixel Parameter Sample GPU1 GPU2 GPU3 GPU4 Parallelizing a 1D convolution

12 The SOAP Search Space Samples: partitioning training samples (Data Parallelism) Operators: partitioning DNN operators (Model Parallelism) Attributes Parameters Pixel Pixel Pixel Parameter Parameter Parameter Sample Convolution#1 Sample Convolution#2 Sample Convolution#3 GPU1 GPU2 GPU3

13 Parallelizing a 1D convolution
The SOAP Search Space Samples: partitioning training samples (Data Parallelism) Operators: partitioning DNN operators (Model Parallelism) Attributes: partitioning attributes in a sample (e.g., different pixels) Parameters Pixel GPU4 Parameter GPU3 GPU2 GPU1 Sample Parallelizing a 1D convolution

14 Parallelizing a 1D convolution
The SOAP Search Space Samples: partitioning training samples (Data Parallelism) Operators: partitioning DNN operators (Model Parallelism) Attributes: partitioning attributes in a sample (e.g., different pixels) Parameters: partitioning parameters in an operator GPU1 Pixel GPU2 GPU3 Parameter GPU4 Sample Parallelizing a 1D convolution

15 Hybrid Parallelism in SOAP
Example parallelization strategies for 1D convolution Different strategies perform the same computation.

16 A possible parallelization strategy in the SOAP search space
Parameter GPU1 Sample GPU2 GPU3 GPU4 Data parallelism A possible parallelization strategy in the SOAP search space

17 A possible parallelization strategy in the SOAP search space
Parameter GPU1 Sample GPU2 GPU3 GPU4 Data parallelism Adding speedup A possible parallelization strategy in the SOAP search space

18 FlexFlow Search Algorithm Cost Model DNN Architecture Device Topology
MatMul Network Concat CPU CPU Conv Conv GPU GPU GPU GPU Execution Optimizer Simulated Performance MCMC Search Alg. Execution Simulator Candidate Strategy Search Algorithm Cost Model Best Found Strategy Distributed Runtime

19 Number of nodes (four K80 GPUs per node)
Evaluation 1.7x faster Training Throughput (samples per second) Number of nodes (four K80 GPUs per node) Speedup Over SOTA DNNs AlexNet ResNet-50 Inception-v3 RNNTC RNNLM GNMT FlexFlow 3.3x 1.1x 1.6x 1.7x 1.9x 2.4x

20 + = … Overview A search space of possible strategies A cost model and
Device 1 Device N Parallelization Graph Optimizations A search space of possible strategies The SOAP search space Auto-generated graph substitutions A cost model and a search algorithm + Markov Chain Monte Carlo Cost-based backtracking search Optimized strategies Fast parallelization strategies Optimized computation graphs =

21 Optimizing DNN Computation with Automated Generation of Graph Substitutions
SysML’19

22 Current Practice: Rule-Based Graph Transformations
Apply graph transformations designed by domain experts E.g., fuse a convolution and a relu into a ``conv + relu’’ Input Input Conv3x3 Conv1x1 Conv3x3 + Relu Conv1x1 + Relu fuse conv and relu Relu Relu Conv3x3 Conv3x3 add add relu relu

23 Limitations of Rule-based Approaches
Robustness Experts’ heuristics do not apply to all DNNs/hardware When I turned on XLA (TensorFlow’s graph optimizer), the training speed is about 20% slower. With XLA, my program is almost 2x slower than without XLA

24 Limitations of Rule-based Approaches
Robustness Experts’ heuristics do not apply to all DNNs/hardware Scalability New operators and graph structures require more rules Performance Miss subtle optimizations for specific DNNs/hardware TensorFlow involves ~4K LOC to optimize a new operator

25 A Missing Graph Optimization
Conv3x3 + Relu Input Relu Conv3x3 + Relu Conv1x1 Input Add Relu Conv3x3 + Relu Input Add Relu Conv3x3 + Relu Input Add Relu Split Conv3x3 + Relu Input Enlarge convs Fuse convs Fuse conv & add Fuse conv & relu The final graph is 1.3x faster on V100 but 10% slower on K80.

26 Can we automatically find these optimizations?
Automatically generated graph substitutions

27 … … … XFlow Cost-Based Search Alg. Graph Subst. Verifier
Verified Substitutions … Graph Subst. Verifier Candidate Substitutions … Input Comp. Graph Optimized Comp. Graph Graph Subst. Generator Operator Specifications

28 End-to-end Inference Performance
Use ~500 automatically generated substitutions End-to-end Inference Performance 2.9x 1.5x 1.4x 1.3x 1.0x Competitive with SOTA Outperform SOTA on unconventional DNNs

29 Open Problems Can we design better search space for parallelization and graph optimizations? Can we find more efficient search algorithms? Can we use search-based optimizations in other domains?

30 + = … Conclusion A search space of possible strategies
Device 1 Device N Parallelization Graph Optimizations A search space of possible strategies The SOAP search space Auto-generated graph substitutions A cost model and a search algorithm + Markov Chain Monte Carlo Cost-based backtracking search Optimized strategies Fast parallelization strategies Optimized computation graphs =

31 Backup Slides

Download ppt "Search-Based Approaches to Accelerate Deep Learning"

Similar presentations

SAGE: Self-Tuning Approximation for Graphics Engines

SAGE: Self-Tuning Approximation for Graphics Engines
Tree-Based Density Clustering using Graphics Processors

Tree-Based Density Clustering using Graphics Processors
Introduction to MCMC and BUGS. Computational problems More parameters -> even more parameter combinations Exact computation and grid approximation become.

Introduction to MCMC and BUGS. Computational problems More parameters -> even more parameter combinations Exact computation and grid approximation become.
Overview of the Course Copyright 2003, Keith D. Cooper, Ken Kennedy & Linda Torczon, all rights reserved. Students enrolled in Comp 412 at Rice University.

Overview of the Course Copyright 2003, Keith D. Cooper, Ken Kennedy & Linda Torczon, all rights reserved. Students enrolled in Comp 412 at Rice University.
Avoiding Segmentation in Multi-digit Numeral String Recognition by Combining Single and Two-digit Classifiers Trained without Negative Examples Dan Ciresan.

Avoiding Segmentation in Multi-digit Numeral String Recognition by Combining Single and Two-digit Classifiers Trained without Negative Examples Dan Ciresan.
4.2.1 Programming Models Technology drivers – Node count, scale of parallelism within the node – Heterogeneity – Complex memory hierarchies – Failure rates.

4.2.1 Programming Models Technology drivers – Node count, scale of parallelism within the node – Heterogeneity – Complex memory hierarchies – Failure rates.
ELeaRNT: Evolutionary Learning of Rich Neural Network Topologies Authors: Slobodan Miletic 3078/2010 Nikola Jovanovic 3077/2010

ELeaRNT: Evolutionary Learning of Rich Neural Network Topologies Authors: Slobodan Miletic 3078/2010 Nikola Jovanovic 3077/2010
CISC 879 - Machine Learning for Solving Systems Problems John Cavazos Dept of Computer & Information Sciences University of Delaware www.cis.udel.edu/~cavazos/cisc879.

CISC Machine Learning for Solving Systems Problems John Cavazos Dept of Computer & Information Sciences University of Delaware
Slides for “Data Mining” by I. H. Witten and E. Frank.

Slides for “Data Mining” by I. H. Witten and E. Frank.
Advanced Computer Architecture & Processing Systems Research Lab Framework for Automatic Design Space Exploration.

Advanced Computer Architecture & Processing Systems Research Lab Framework for Automatic Design Space Exploration.
M. Wang, T. Xiao, J. Li, J. Zhang, C. Hong, & Z. Zhang (2014)

M. Wang, T. Xiao, J. Li, J. Zhang, C. Hong, & Z. Zhang (2014)
Philipp Gysel ECE Department University of California, Davis

Philipp Gysel ECE Department University of California, Davis
Deep Learning and Deep Reinforcement Learning. Topics 1.Deep learning with convolutional neural networks 2.Learning to play Atari video games with Deep.

Deep Learning and Deep Reinforcement Learning. Topics 1.Deep learning with convolutional neural networks 2.Learning to play Atari video games with Deep.
ConvNets for Image Classification

ConvNets for Image Classification
Accelerating K-Means Clustering with Parallel Implementations and GPU Computing Janki Bhimani Miriam Leeser Ningfang Mi

Accelerating K-Means Clustering with Parallel Implementations and GPU Computing Janki Bhimani Miriam Leeser Ningfang Mi
Variational Autoencoders Theory and Extensions

Variational Autoencoders Theory and Extensions
Understanding AlphaGo. Go Overview Originated in ancient China 2,500 years ago Two players game Goal - surround more territory than the opponent 19X19.

Understanding AlphaGo. Go Overview Originated in ancient China 2,500 years ago Two players game Goal - surround more territory than the opponent 19X19.
Igor EPIMAKHOV Abdelkader HAMEURLAIN Franck MORVAN

Igor EPIMAKHOV Abdelkader HAMEURLAIN Franck MORVAN
TensorFlow The Deep Learning Library You Should Be Using.

TensorFlow The Deep Learning Library You Should Be Using.
TensorFlow CS 5665 F16 practicum Karun Joseph, A02240287 Reference:

TensorFlow CS 5665 F16 practicum Karun Joseph, A Reference:

Similar presentations

Ads by Google