Presentation is loading. Please wait.

Presentation is loading. Please wait.

PipeDream: Pipeline Parallelism for DNN Training

Published byHillary Ferguson Modified over 6 years ago

Similar presentations

Presentation on theme: "PipeDream: Pipeline Parallelism for DNN Training"— Presentation transcript:

1 PipeDream: Pipeline Parallelism for DNN Training
Aaron Harlap, Deepak Narayanan, Amar Phanishayee*, Vivek Seshadri*, Greg Ganger, Phil Gibbons

2 Example DNN 1000s

3 DNN - How they Work Good NFL Coach? Yes: 12% No: 88% Yes: 99% No: 1%
Input Output

4 DNN Training - How do they Learn
Forward Pass - Make a Prediction Input Output EVALUATE! Backward Pass - Update Solution Depending on Error

5 Data-Parallel Training
Separate copy of model on each machine Communicate updates to model parameters various synchronization models can be used

6 Problem w/ Data Parallel
Communication overhead can be high Less communication lowers quality of work Communication Computation VGG OverFeat Inception-BN GooLeNet

7 New Approach: Pipeline Parallel
Assign layers to machines communicate inter-layer activations Machine 1 Machine 2 Machine 3

8 Alternate forward / backward work
Each machine does forward work then backwards work, then forward work etc… Forward Compute Work #5 Backward Compute Work #1 Fw. Send #5 Backward Compute Work #2 Fw. Send #4 Backward Compute Work #3 Machine 1 Forward Compute Work #4 Machine 2 Bw. Send #1 Forward Compute Work #3 Machine 3 Bw. Send #2

9 One Quick Result - Pipeline 2.5x better
VGG Network w/ ImageNet Dataset

10 Many More Interesting Details
Staleness vs Speed trade-off How staleness in pipelining is different from data-parallel staleness How to divide works amongst machines GPU Memory Management This and more at our poster!

11 Summary New way to parallelize for DNN training
Communicate layer activations instead of model parameters Improves over data-parallel for networks w/ large solution state

Download ppt "PipeDream: Pipeline Parallelism for DNN Training"

Similar presentations

A Real Time Radiosity Architecture for Video Games

A Real Time Radiosity Architecture for Video Games
also known as the “Perceptron”

also known as the “Perceptron”
Machine Learning: Connectionist McCulloch-Pitts Neuron Perceptrons Multilayer Networks Support Vector Machines Feedback Networks Hopfield Networks.

Machine Learning: Connectionist McCulloch-Pitts Neuron Perceptrons Multilayer Networks Support Vector Machines Feedback Networks Hopfield Networks.
S. Mandayam/ ANN/ECE Dept./Rowan University Artificial Neural Networks 0909.560.01/0909.454.01 Fall 2004 Shreekanth Mandayam ECE Department Rowan University.

S. Mandayam/ ANN/ECE Dept./Rowan University Artificial Neural Networks / Fall 2004 Shreekanth Mandayam ECE Department Rowan University.
Neural Networks. Background - Neural Networks can be : Biological - Biological models Artificial - Artificial models - Desire to produce artificial systems.

Neural Networks. Background - Neural Networks can be : Biological - Biological models Artificial - Artificial models - Desire to produce artificial systems.
Chapter 9 Neural Network.

Chapter 9 Neural Network.
Backpropagation An efficient way to compute the gradient Hung-yi Lee.

Backpropagation An efficient way to compute the gradient Hung-yi Lee.
Hartman1P1004 Leo Hartman Canadian Space Agency A VHDL Implementation of an On-board Autonomy Solution.

Hartman1P1004 Leo Hartman Canadian Space Agency A VHDL Implementation of an On-board Autonomy Solution.
11 1 Backpropagation. 11 2 Multilayer Perceptron R – S 1 – S 2 – S 3 Network.

11 1 Backpropagation Multilayer Perceptron R – S 1 – S 2 – S 3 Network.
A Neural Network Implementation on the GPU By Sean M. O’Connell CSC 7333 Spring 2008.

A Neural Network Implementation on the GPU By Sean M. O’Connell CSC 7333 Spring 2008.
M. Wang, T. Xiao, J. Li, J. Zhang, C. Hong, & Z. Zhang (2014)

M. Wang, T. Xiao, J. Li, J. Zhang, C. Hong, & Z. Zhang (2014)
Solving the straggler problem with bounded staleness Jim Cipar, Qirong Ho, Jin Kyu Kim, Seunghak Lee, Gregory R. Ganger, Garth Gibson, Kimberly Keeton*,

Solving the straggler problem with bounded staleness Jim Cipar, Qirong Ho, Jin Kyu Kim, Seunghak Lee, Gregory R. Ganger, Garth Gibson, Kimberly Keeton*,
ImageNet Classification with Deep Convolutional Neural Networks Presenter: Weicong Chen.

ImageNet Classification with Deep Convolutional Neural Networks Presenter: Weicong Chen.
Previous Lecture Perceptron W  t+1  W  t  t  d(t) - sign (w(t)  x)] x Adaline W  t+1  W  t  t  d(t) - f(w(t)  x)] f’ x Gradient.

Previous Lecture Perceptron W  t+1  W  t  t  d(t) - sign (w(t)  x)] x Adaline W  t+1  W  t  t  d(t) - f(w(t)  x)] f’ x Gradient.
Joe Bradish Parallel Neural Networks. Background  Deep Neural Networks (DNNs) have become one of the leading technologies in artificial intelligence.

Joe Bradish Parallel Neural Networks. Background  Deep Neural Networks (DNNs) have become one of the leading technologies in artificial intelligence.
Lecture 3a Analysis of training of NN

Lecture 3a Analysis of training of NN
Data and Computer Communications Eighth Edition by William Stallings Lecture slides by Lawrie Brown Chapter 10 – Circuit Switching and Packet Switching.

Data and Computer Communications Eighth Edition by William Stallings Lecture slides by Lawrie Brown Chapter 10 – Circuit Switching and Packet Switching.
TensorFlow The Deep Learning Library You Should Be Using.

TensorFlow The Deep Learning Library You Should Be Using.
When deep learning meets object detection: Introduction to two technologies: SSD and YOLO Wenchi Ma.

When deep learning meets object detection: Introduction to two technologies: SSD and YOLO Wenchi Ma.
Stanford University.

Stanford University.

Similar presentations

Ads by Google