1. Approximating the Buffer Allocation Problem Using Epochs
Jan Bækgaard Pedersen (U. of Nevada, Las Vegas)
Alex Brodsky, (U. of Winnipeg, Canada)

2. Parallel Applications
Goal: Overlap communication and computation.
Parallel Applications
Consist of a number of asynchronous interconnected hosts <click>
That run at different speeds, with no access to a global clock
Communicate by send messages to each other over a local area network <click>
To ensure applications are efficient <click>
Applications typically overlap communication and computation
2/25/2019

3. Parallel Applications (cont.)
Today, parallel applications typically
Run on heterogeneous platforms
Are ported to systems with varying resource
Run on hardware platforms that differ from those they were designed on.
Goals:
Ensure that ported applications do not deadlock
Determine application resource requirements
Focus:
Determine an application’s message buffer needs
See points on slide.
Emphasize porting applications, applications that can be run on original hardware to compute comm. Graph (but don’t mention comm. Graph yet)
2/25/2019

4. Message Buffering Unbuffered (synchronous op.) Buffered
MPI_Send
MPI_Recv
Buffered
(asynchronous op.)
MPI_Send
MPI_Recv
If a message send is unbuffered <click>
a sending process performs a send and then
must wait for the receiver to perform a receive <click>
Before the sender can
complete the send and
perform its own computation <click>
With buffered message passing <click>
the process can send the message <click>
store the message in the buffer and compute
the receiver can get the message from buffer at their convenience <click>
thus allowing an overlap of computation and communications
How if the buffer is filled up <click>
The send will block <click>
until the receiver performs a receive <click>
If the application assumes that a buffer is available, this could be a problem.
Buffer filled
(synchronous op.)
MPI_Send
MPI_Recv
2/25/2019

5. The Problem Succeeds as long as 1 process has a free buffer.
MPI_Send
MPI_Send
MPI_Send
MPI_Recv
MPI_Recv
MPI_Recv
Succeeds as long as 1 process has a free buffer.
If all buffers are exhausted deadlock occurs!
Let’s look at the problem in more detail
Consider the following communication pattern where
each process performs a send
then presumably performs some computations
and then performs a receive
The first process has a buffer,
so when the third process performs a send <click>
the message is buffered and does not block the sender
who can then perform the receive and <click>
allow the remaining processes to complete their sends
However if no buffers are available <click>
And the processes perform sends <click>
then none of the sends can proceed <click>
and deadlock ensues <click>
Thus, we need to ensure <click>
that the application has enough buffers
MPI_Send
MPI_Send
MPI_Send
Solution: Ensure application has sufficient # of buffers.
2/25/2019

6. Deadlock results by untimely use of buffer.
Complications
MPI_Send
MPI_Send
MPI_Send
Deadlock results by untimely use of buffer.
Such problems arise even if the communication
is oblivious (independent of application input)
is point-to-point (no broadcast or multicast)
does not use wild-cards (receiver must specify sender)
Things can get even more complicated
By performing a send, <click>
A process can steal a buffer ment for anoth message
Creating deadlock <click>
because other sends cannot
In fact <click>
Follow points
Interestingly <click>
The problem of allocating buffers for an application is
is NP-hard to find an optimal allocation [BPW’05]
is coNP-complete to verify if an allocation is deadlock-free
needs to be approximated
2/25/2019

7. The Model (Communication Graphs)
MPI_Send
MPI_Send
MPI_Send
MPI_Recv
MPI_Recv
MPI_Recv
comp.
arc
comm.
send
vert.
recv
We model this problem using communication graphs
Given an application and its communication pattern
We model the application as a graph <click>
Describe communication graph….
One key point <click>
Since we assume an oblivious communication pattern
all application behaviour of interest (i.e., communication)
Is represented by the communication graph
Key point: An application is represented by a communication graph.
2/25/2019

8. Basic Approach Treat as a dependency graph Adding a buffer not enough
reverse computation arcs
bidirectional communication arcs
A cycle indicates a need for a buffer
Adding a buffer not enough
buffers may be stolen!
Add buffer for every receive
although, this is overkill
ensures delay-freedom
Our basic approach is to induce a dependency graph
by reversing computation arcs and
making communication arcs bidirectionnal
A cycle in the graph indicates the need for a buffer <click>
If there are no buffers <click>
the sends will never complete.
As we have seen <click>
this may not be enough
since buffers may be stolen by
other processesthat perform untimely sends
Thus, out initial approximation <clock>
is to assign a buffer to each receive
Although this guarantees that no process ever blocks
this is clearly overkill in the number of buffers needed.
A better approach is needed….
If no buffers, receive and send depend on each other.
2/25/2019

9. A Better Approximation
If send (2) always occurs before (3)
buffer will not be stolen
deadlock will be prevented
only one buffer is needed
Implementation issues
how to order sends?
how to enforce order?
what kind of order?
how many buffers?
(1)
(2)
Observe that if we can ensure that send occurs before send 3
Then send 2 will grab the buffer and no deadlock will ensue
The key point is that sends 1 and 2 must be ordered <click>
before send 3
How do we do this? We need to address several issues <click>
Follow points….
(3)
2/25/2019

10. Our Approach Partition communication graph into epochs
Epochs are well ordered
Sends within an epoch may not be ordered
Determine per-epoch buffer allocations
Compute maximum buffer allocation over all per-epoch allocations
This will be the buffer allocation for the application
Group consecutive epochs into super-epochs
Use barrier primitive to enforce execution order on the super-epochs
Follow points….
2/25/2019

11. Epochs Are maximal connected subgraphs All epochs are strictly ordered
Represent phases of an application
No communication between epochs
Execute in serial order
All epochs are strictly ordered
Each epoch is executed when
preceding epoch finishes
all buffers are freed
Enforced via barriers
Describe what epoch are s…
<click>
Describe how epochs are ordered
To enforce epoch ordering <click>
we use a barrier synchronization primitive
2/25/2019

12. Number of Buffers Needed
Compute # of buffers for each epoch
Epochs with cycles: > 0 buffers
Use approximation of [BPW’05]
Epochs with no cycles: 0 buffers
No cycles  no deadlock
At end of each epoch
all buffers are freed
available for next epoch
Total # of buffers needed is: the maximum over all epochs
1 buffer
We then compute the number of buffers for each epoch
Cycles need at least one buffer….
As in the first epoch below <click>
Epochs with no cycles need no buffers… <click>
because no deadlock can occur
Note that <click>
at end of each epoch
all buffers are freed
and become available to the next epoch
Consequently <click>
The total # of buffers needed
is simply the maximum over all epochs
0 buffers
Total # of buffers needed is max{1,0} = 1 buffer.
2/25/2019

13. Avoiding Unnecessary Barriers
Barriers are expensive operations
In many cases barriers are not needed
In following example
each send is in its own epoch
a barrier occurs after every send
even though no buffers are needed
Idea: combine epochs into a super-epoch
eliminates unneeded barriers
increases parallelism
does not increase number of buffers
Super-Epochs are treated like epochs
serially ordered
separated by barriers
no explicit synchronization within
use at most max # of buffers over all epochs
However <click>
barriers are expensive operations
are not needed in many cases, explain why…
To avoid unnecessary barriers <click>
we combine epochs into superepochs
Describe superepochs (follow points)
2/25/2019

14. Proposed Implementation
App.
MPI
O/S
App.
MPI
O/S
App.
MPI
O/S
App.
MPI
O/S
Compute barrier and buffer allocations
Generate communication graph from the application
Compute epochs and per-epoch buffer allocations
Compute super-epochs and barrier locations
Upload barrier locations and buffer allocations
Before each send, MPI checks if a barrier is needed
We propose the following implementations
We first compute the barrier and buffer allocations by <click>
generating the communication graph <click>
computing the epochs <click>
and performing per epoch buffer allocations.
Then <click>
we combine epochs into super-epochs <click>
and determine locations of barriers.
We then use a modified MPI library <click>
into which we upload the barrier and buffer allocation information
Before each send … describe what MPI should do.
App.
2/25/2019

15. Technical Contributions
Algorithms for:
partitioning the communication graph into epochs
computing the buffer allocation for each epoch
composing super-epochs from epochs
Descriptions of:
possible mechanism implementation within existing parallel systems
possible mechanism extensions
Analysis:
comparing our approach to existing approaches
complexity of proposed algorithms
Follow points….
2/25/2019