Presentation is loading. Please wait.

Presentation is loading. Please wait.

Asynchronous I/O with MPI Anthony Danalis. Basic Non-Blocking API  MPI_Isend()  MPI_Irecv()  MPI_Wait()  MPI_Waitall()  MPI_Waitany()  MPI_Test()

Published byGalilea Sheward Modified over 10 years ago

Similar presentations

Presentation on theme: "Asynchronous I/O with MPI Anthony Danalis. Basic Non-Blocking API  MPI_Isend()  MPI_Irecv()  MPI_Wait()  MPI_Waitall()  MPI_Waitany()  MPI_Test()"— Presentation transcript:

1 Asynchronous I/O with MPI Anthony Danalis

2 Basic Non-Blocking API  MPI_Isend()  MPI_Irecv()  MPI_Wait()  MPI_Waitall()  MPI_Waitany()  MPI_Test()  MPI_Testall()  MPI_Testany()

3 API: sending MPI_ISEND(buf, count, datatype, dest, tag, comm, request) [ IN buf] initial address of send buffer (choice) [ IN count] number of elements in send buffer (integer) [ IN datatype] datatype of each send buffer element (handle) [ IN dest] rank of destination (integer) [ IN tag] message tag (integer) [ IN comm] communicator (handle) [ OUT request] communication request (handle)

4 API: receiving MPI_IRECV (buf, count, datatype, source, tag, comm, request) [ OUT buf] initial address of receive buffer (choice) [ IN count] number of elements in receive buffer (integer) [ IN datatype] datatype of each receive buffer element (handle) [ IN source] rank of source (integer) [ IN tag] message tag (integer) [ IN comm] communicator (handle) [ OUT request] communication request (handle)

5 API: blocking/polling MPI_WAIT(request, status) [ INOUT request] request (handle) [ OUT status] status object (Status) MPI_TEST(request, flag, status) [ INOUT request] communication request (handle) [ OUT flag] true if operation completed (logical) [ OUT status] status object (Status)

6 Why N-B I/O? Example 1 CALL MPI_COMM_RANK(comm, rank, ierr) IF(rank.EQ. 0) THEN CALL MPI_ISEND(a(1), 10, MPI_REAL, 1, tag, comm, request, ierr) CALL MPI_WAIT(request, status, ierr) ELSE CALL MPI_IRECV(a(1), 15, MPI_REAL, 0, tag, comm, request, ierr) CALL MPI_WAIT(request, status, ierr) END IF

7 Why N-B I/O? Example 2 CALL MPI_COMM_RANK(comm, rank, ierr) IF(rank.EQ. 0) THEN CALL MPI_ISEND(a(1), 10, MPI_REAL, 1, tag, comm, request, ierr) **** do some computation to mask latency **** CALL MPI_WAIT(request, status, ierr) ELSE CALL MPI_IRECV(a(1), 15, MPI_REAL, 0, tag, comm, request, ierr) **** do some computation to mask latency **** CALL MPI_WAIT(request, status, ierr) END IF

8 Why N-B I/O? Deadlocks(?) IF(rank.EQ. 0) THEN CALL MPI_SEND() CALL MPI_RECV() ELSE CALL MPI_SEND() CALL MPI_RECV() END IF Blocking IF(rank.EQ. 0) THEN CALL MPI_ISEND() CALL MPI_IRECV() ELSE CALL MPI_ISEND() CALL MPI_IRECV() END IF CALL MPI_WAITALL() Non-Blocking

9 Why N-B I/O? Efficiency (painless) for(i=1; i<NP; i++){ sender = (NP+myRank-i)%NP; receiver = (myRank+i)%NP; MPI_Irecv(sender,...) MPI_Isend(receiver,...) } MPI_Waitall() 1 2 0 1 1 2 1 1 2 2

10 Why N-B I/O? Overlapping  Communication / Computation  Communication / Communication Asynchronous I/O Overlap

11 Non-Blocking VS Asynchronous  Non-Blocking => function call returns  Asynchronous => CPU does other things IF(rank.EQ. 0) THEN CALL MPI_ISEND(a(1), 10, MPI_REAL, 1, tag, comm, request, ierr) **** do some computation to mask latency **** CALL MPI_WAIT(request, status, ierr) ELSE CALL MPI_IRECV(a(1), 15, MPI_REAL, 0, tag, comm, request, ierr) **** do some computation to mask latency **** CALL MPI_WAIT(request, status, ierr) END IF

12 Multidimensional FFT P1 P2 P1 I/O 1-D FFT Dimension X Dimension Y

13 FFT code: MPI_ALLTOALL() DO IZ = 1, (NZ/NPROC) DO IY = 1, NYP CALL REAL_TO_COMPLEX( REAL_IN=AP(:,IY,IZ), COMPLEX_OUT=A3 ) DO IX = 1, (NX/2) IPROC = (IX-1)/((NX/2)/NPROC) IXEFF = IX - IPROC*((NX/2)/NPROC) CAT2(IXEFF,IY,IZ,IPROC+1) = A3(IX) ENDDO NT = NYP*(NZ/NPROC)*((NX/2)/NPROC) CALL MPI_BARRIER( MPI_COMM_WORLD, ERR ) CALL MPI_ALLTOALL( CAT2(1,1,1,1), NT, MPI_DOUBLE_COMPLEX, & CAT(1,1,1,1), NT, MPI_DOUBLE_COMPLEX, MPI_COMM_WORLD, ERR )

14 FFT code: MPI_SEND() + MPI_RECV() DO IZ = 1, (NZ/NPROC) DO IY = 1, NYP CALL REAL_TO_COMPLEX( REAL_IN=AP(:,IY,IZ), COMPLEX_OUT=A3 ) DO IPROC = 1, NPROC-1 REAL_RECEIVER = MOD(IPROC+MYNUM, NPROC) CHUNCK = ((NX/2)/NPROC)*REAL_RECEIVER+1 CALL MPI_SEND(A3(CHUNCK), ((NX/2)/NPROC), MPI_DOUBLE_COMPLEX, & REAL_RECEIVER, 0, MPI_COMM_WORLD, ERR) REAL_SUBMITER = MOD(NPROC+MYNUM-IPROC, NPROC) CALL MPI_RECV(DAN_A3(1), ((NX/2)/NPROC), MPI_DOUBLE_COMPLEX, & REAL_SUBMITER, 0, MPI_COMM_WORLD, STATUS, ERR ) DO IX = 1, ((NX/2)/NPROC) TARG_IZ = IZ + REAL_SUBMITER*(NZ/NPROC) AUX(TARG_IZ,IY,IX) = DAN_A3(IX); ENDDO DO IX = 1, ((NX/2)/NPROC) TARG_IZ = IZ + MYNUM*(NZ/NPROC) AUX(TARG_IZ,IY,IX) = A3( IX+((NX/2)/NPROC)*MYNUM ); ENDDO

15 DO IZ = 1, (NZ/NPROC) DO IY = 1, NYP CALL REAL_TO_COMPLEX( REAL_IN=AP(:,IY,IZ), COMPLEX_OUT=A3 ) CALL MPI_WAITALL( (2*(NPROC-1)), REQST(1), STATS, ERR) IF( IY > 1 ) THEN DO IPROC = 1, NPROC-1 REAL_SUBMITER = MOD(NPROC+MYNUM-IPROC, NPROC) DO IX = 1, ((NX/2)/NPROC) TARG_IZ = IZ + REAL_SUBMITER*(NZ/NPROC) AUX(TARG_IZ,IY-1,IX) = DAN_A3(IX, IPROC); ENDDO ENDIF DO IPROC = 1, NPROC-1 REAL_SUBMITER = MOD(NPROC+MYNUM-IPROC,NPROC) CALL MPI_IRECV( DAN_A3(1,IPROC), ((NX/2)/NPROC), MPI_DOUBLE_COMPLEX, REAL_SUBMITER, 0, & MPI_COMM_WORLD, REQST((2*IPROC-1)), ERR) REAL_RECEIVER = MOD(IPROC+MYNUM, NPROC) CHUNCK = ((NX/2)/NPROC)*REAL_RECEIVER+1 CALL MPI_ISEND(A3(CHUNCK), ((NX/2)/NPROC), MPI_DOUBLE_COMPLEX, REAL_RECEIVER, 0, & MPI_COMM_WORLD, REQST((2*IPROC)), ERR) ENDDO DO IX = 1, ((NX/2)/NPROC) TARG_IZ = IZ + MYNUM*(NZ/NPROC) AUX(TARG_IZ,IY,IX) = A3( IX+((NX/2)/NPROC)*MYNUM ); ENDDO CALL MPI_WAITALL( (2*(NPROC-1)), REQST(1), STATS, ERR) DO IPROC = 1, NPROC-1 REAL_SUBMITER = MOD(NPROC+MYNUM-IPROC, NPROC) TARG_IZ = IZ + REAL_SUBMITER*(NZ/NPROC) DO IX = 1, ((NX/2)/NPROC) AUX(TARG_IZ,NYP,IX) = DAN_A3(IX, IPROC); ENDDO FFT code: MPI_iSEND() + MPI_iRECV() race?

16 DO IZ = 1, (NZ/NPROC) !{ DO IY = 1, NYP !{ CALL REAL_TO_COMPLEX( REAL_IN=AP(:,IY,IZ), COMPLEX_OUT=A3(1,1+MOD(IY-1,K_LOOPS), 1+MOD((IY-1)/K_LOOPS,2))) IF( (IY > K_LOOPS).AND. (MOD(IY,K_LOOPS) == 0) ) THEN !{ CALL MPI_WAITALL( (2*(NPROC-1)), REQST(1), STATS, ERR) DO IPROC = 1, NPROC-1 !{ REAL_SUBMITER = MOD(NPROC+MYNUM-IPROC, NPROC) TARG_IZ = IZ + REAL_SUBMITER*(NZ/NPROC) DO IIK = 1, K_LOOPS !{ OLDIY = IY-2*K_LOOPS+IIK DO IX = 1, ((NX/2)/NPROC) !{ AUX(TARG_IZ,OLDIY,IX) = DAN_RECV(IX, IIK, IPROC); ENDDO !} ENDIF !} IF( MOD(IY,K_LOOPS) == 0 ) THEN !{ DO IPROC = 1, NPROC-1 !{ REAL_SUBMITER = MOD(NPROC+MYNUM-IPROC,NPROC) CALL MPI_IRECV( DAN_RECV(1,1,IPROC), K_LOOPS*((NX/2)/NPROC), MPI_DOUBLE_COMPLEX, & REAL_SUBMITER, 0, MPI_COMM_WORLD, REQST((2*IPROC-1)), ERR) REAL_RECEIVER = MOD(IPROC+MYNUM, NPROC) CHNK_STR = ((NX/2)/NPROC)*REAL_RECEIVER+1 CHNK_END = CHNK_STR+((NX/2)/NPROC)-1 CALL MPI_ISEND(A3( CHNK_STR:CHNK_END, :, 1+MOD((IY-1)/K_LOOPS,2) ), K_LOOPS*((NX/2)/NPROC), & MPI_DOUBLE_COMPLEX, REAL_RECEIVER, 0, MPI_COMM_WORLD, REQST((2*IPROC)), ERR) ENDDO !} ENDIF !} DO IX = 1, ((NX/2)/NPROC) !{ TARG_IZ = IZ + MYNUM*(NZ/NPROC) AUX(TARG_IZ,IY,IX) = A3( IX+((NX/2)/NPROC)*MYNUM, 1+MOD(IY-1,K_LOOPS), 1+MOD((IY-1)/K_LOOPS,2)); ENDDO !} FFT code: MPI_iSEND() + MPI_iRECV() (K)

17 FFT code: MPI_iSEND() + MPI_iRECV() (K) cont. LEFTOVERS = MOD(NYP,K_LOOPS) IF( LEFTOVERS /= 0 ) THEN !{ DO IPROC = 1, NPROC-1 !{ REAL_SUBMITER = MOD(NPROC+MYNUM-IPROC, NPROC) CALL MPI_IRECV(LEFTOVERS_RECV(1,1,IPROC), LEFTOVERS*(NZ/NPROC), MPI_DOUBLE_COMPLEX, & REAL_SUBMITER, 0, MPI_COMM_WORLD, LASTREQST((2*IPROC-1)), ERR) REAL_RECEIVER = MOD(IPROC+MYNUM, NPROC) CHNK_STR = ((NX/2)/NPROC)*REAL_RECEIVER+1 CHNK_END = CHNK_STR+((NX/2)/NPROC)-1 CALL MPI_ISEND(A3( CHNK_STR:CHNK_END, 1:LEFTOVERS, 1+MOD((NYP-1)/K_LOOPS,2) ), LEFTOVERS*((NX/2)/NPROC),& MPI_DOUBLE_COMPLEX, REAL_RECEIVER, 0, MPI_COMM_WORLD, LASTREQST((2*IPROC)), ERR) ENDDO !} ENDIF !} !!!!! deal with the chuncks where IY=K_LOOPS*(NYP/K_LOOPS)-K_LOOPS+1:K_LOOPS*(NYP/K_LOOPS) CALL MPI_WAITALL( (2*(NPROC-1)), REQST(1), STATS, ERR) DO IPROC = 1, NPROC-1 !{ REAL_SUBMITER = MOD(NPROC+MYNUM-IPROC, NPROC) TARG_IZ = IZ + REAL_SUBMITER*(NZ/NPROC) DO IIK = 1, K_LOOPS !{ OLDIY = NYP-LEFTOVERS-K_LOOPS+IIK DO IX = 1, ((NX/2)/NPROC) !{ AUX(TARG_IZ,OLDIY,IX) = DAN_RECV(IX, IIK, IPROC); ENDDO !}

18 FFT code: MPI_iSEND() + MPI_iRECV() (K) cont. IF( LEFTOVERS /= 0 ) THEN !{ !!!!! deal with the chuncks where IY=NYP-MOD(NYP,K_LOOPS)+1:NYP CALL MPI_WAITALL( (2*(NPROC-1)), LASTREQST(1), STATS, ERR) DO IPROC = 1, NPROC-1 !{ REAL_SUBMITER = MOD(NPROC+MYNUM-IPROC, NPROC) TARG_IZ = IZ + REAL_SUBMITER*(NZ/NPROC) DO IIK = 1, LEFTOVERS !{ OLDIY = NYP-LEFTOVERS+IIK DO IX = 1, ((NX/2)/NPROC) !{ AUX(TARG_IZ,OLDIY,IX) = LEFTOVERS_RECV(IX, IIK, IPROC); ENDDO !} ENDIF !} ENDDO !}

19 Pseudocode: MPI_iSEND() + MPI_iRECV() (K&D) do i = 1, NX do j = 1, NY kernel(indata[j], outdata[j]) if( (j > D*K) && (j%K == 0) ) then MPI_WAITALL(request[j-D]) endif if( j%K == 0 ) then MPI_iSEND(outdata[j-K], SIZE=K) MPI_iRECV(recv_data, request[j]) endif enddo do m = 0, D-1 MPI_WAITALL(request[NY-D+m]) enddo other_computation() enddo

20 Performance gain

21 Comput/Commun Overlapping time progress communication computation

Download ppt "Asynchronous I/O with MPI Anthony Danalis. Basic Non-Blocking API  MPI_Isend()  MPI_Irecv()  MPI_Wait()  MPI_Waitall()  MPI_Waitany()  MPI_Test()"

Similar presentations

ECMWF Slide 1Porting MPI Programs to the IBM Cluster 1600 Peter Towers March 2004.

ECMWF Slide 1Porting MPI Programs to the IBM Cluster 1600 Peter Towers March 2004.
Problems with M things. Review: NULL MPI handles All MPI_*_NULL are handles to invalid MPI objects MPI_REQUEST_NULL is an exception Calling MPI_TEST/MPI_WAIT.

Problems with M things. Review: NULL MPI handles All MPI_*_NULL are handles to invalid MPI objects MPI_REQUEST_NULL is an exception Calling MPI_TEST/MPI_WAIT.
1 Tuning for MPI Protocols l Aggressive Eager l Rendezvous with sender push l Rendezvous with receiver pull l Rendezvous blocking (push or pull)

1 Tuning for MPI Protocols l Aggressive Eager l Rendezvous with sender push l Rendezvous with receiver pull l Rendezvous blocking (push or pull)
1 What is message passing? l Data transfer plus synchronization l Requires cooperation of sender and receiver l Cooperation not always apparent in code.

1 What is message passing? l Data transfer plus synchronization l Requires cooperation of sender and receiver l Cooperation not always apparent in code.
MPI Message Passing Interface

MPI Message Passing Interface
VSCSE Summer School Programming Heterogeneous Parallel Computing Systems Lecture 6: Basic CUDA and MPI.

VSCSE Summer School Programming Heterogeneous Parallel Computing Systems Lecture 6: Basic CUDA and MPI.
MPI version of the Serial Code With One-Dimensional Decomposition Presented by Timothy H. Kaiser, Ph.D. San Diego Supercomputer Center Presented by Timothy.

MPI version of the Serial Code With One-Dimensional Decomposition Presented by Timothy H. Kaiser, Ph.D. San Diego Supercomputer Center Presented by Timothy.
1 Non-Blocking Communications. 2 #include int main(int argc, char **argv) { int my_rank, ncpus; int left_neighbor, right_neighbor; int data_received=-1;

1 Non-Blocking Communications. 2 #include int main(int argc, char **argv) { int my_rank, ncpus; int left_neighbor, right_neighbor; int data_received=-1;
1 Introduction to Collective Operations in MPI l Collective operations are called by all processes in a communicator. MPI_BCAST distributes data from one.

1 Introduction to Collective Operations in MPI l Collective operations are called by all processes in a communicator. MPI_BCAST distributes data from one.
Parallel Processing1 Parallel Processing (CS 667) Lecture 9: Advanced Point to Point Communication Jeremy R. Johnson *Parts of this lecture was derived.

Parallel Processing1 Parallel Processing (CS 667) Lecture 9: Advanced Point to Point Communication Jeremy R. Johnson *Parts of this lecture was derived.
MPI Collective Communications

MPI Collective Communications
Sahalu Junaidu ICS 573: High Performance Computing 8.1 Topic Overview Matrix-Matrix Multiplication Block Matrix Operations A Simple Parallel Matrix-Matrix.

Sahalu Junaidu ICS 573: High Performance Computing 8.1 Topic Overview Matrix-Matrix Multiplication Block Matrix Operations A Simple Parallel Matrix-Matrix.
1 Implementing Master/Slave Algorithms l Many algorithms have one or more master processes that send tasks and receive results from slave processes l Because.

1 Implementing Master/Slave Algorithms l Many algorithms have one or more master processes that send tasks and receive results from slave processes l Because.
1 Buffers l When you send data, where does it go? One possibility is: Process 0Process 1 User data Local buffer the network User data Local buffer.

1 Buffers l When you send data, where does it go? One possibility is: Process 0Process 1 User data Local buffer the network User data Local buffer.
Point-to-Point Communication Self Test with solution.

Point-to-Point Communication Self Test with solution.
SOME BASIC MPI ROUTINES With formal datatypes specified.

SOME BASIC MPI ROUTINES With formal datatypes specified.
Distributed Memory Programming with MPI. What is MPI? Message Passing Interface (MPI) is an industry standard message passing system designed to be both.

Distributed Memory Programming with MPI. What is MPI? Message Passing Interface (MPI) is an industry standard message passing system designed to be both.
1 Parallel Computing—Introduction to Message Passing Interface (MPI)

1 Parallel Computing—Introduction to Message Passing Interface (MPI)
E.Papandrea 06/11/2003 DFCI COMPUTING - HW REQUIREMENTS1 Enzo Papandrea COMPUTING HW REQUIREMENT.

E.Papandrea 06/11/2003 DFCI COMPUTING - HW REQUIREMENTS1 Enzo Papandrea COMPUTING HW REQUIREMENT.
MPI Point-to-Point Communication CS 524 – High-Performance Computing.

MPI Point-to-Point Communication CS 524 – High-Performance Computing.

Similar presentations

Ads by Google