1. Sending large message counts (The MPI_Count issue)

2. Motivation – initial requests HP received several requests to scale applications in a way that required sending messages greater than 2GB. – Did not want to change their Fortran source – compiled with –I8 (compiler flags to promote ALL integers to be 64- bytes) – Also wanted Scalapack that would scale similarly. – requested that the MPI implementation “handle” this.

3. Motivation – HP’s partial response Fortran: – Produced an –I8 compatible mode – Determined at run time if application was compiled with –I8 (and/or – R8) – Correctly interpret arguments as either 32 or 64-bit integers – Dynamically defines MPI data types – Internally MPI library always uses 64-bit integers for all count related arguments. C code: – HP-MPI produced new C interfaces (MPI_SendL) to allow modification of Scalapack and other libraries.

4. A complete solution Do we want a more complete solution: – Works for all languages – Does not rely on compiler flags or non-standard API’s

5. #0 – Fortran only solution Proposal #1: – Do nothing. Let MPI’s choose to support –I8 (See Platform MPI for an example) Fortran applications that need to use long counts much use –I8 – Pros: No work for the Forum – Cons: Fortran only solution Big hammer even for Fortran since it may double memory use

6. #1 – more API’s Proposal #1: – Create new interfaces that accept a 64-bit count. Example: MPI_SendL The count could simply be a long int, but preferably a new type: MPI_Count that can be changed based on what the implementation supports – Pros: No backward compatibility issues Fixes the problem NOW. – Cons: Apps must be re-written to access the new capability Explosion of interfaces

7. #2 – Sneak in MPI_Count Proposal #2: – Change relevant functions to use MPI_Count – C++ function overloading handles it (assuming we care about C++) Never need to recompile unless you use MPI_Count. – C automatic type casting handles it. No recompile is even needed if MPI_Count is defined as an int and application passes int’s. – Fortran must be “told” the size of the MPI_Count arguments. Need agreement between application and library. constants must be changed to declared parameters: old: call MPI_Send(buf, 10, MPI_INTEGER, 1, 2, MPI_COMM_WORLD, ierr) new: integer*8 ten parameter (ten= 10) call MPI_Send(buf, ten, MPI_INTEGER, 1, 2, MPI_COMM_WORLD, ierr)

8. #2 – Sneak in MPI_Count Proposal #2: – Pros: Only impacts users who want 64-bit applications – Cons: Fortran77 application re-write is ugly. Existing C applications “work but are not MPI compliant” There are really 2 Fortran interfaces for every affected call: C:int MPI_Send(void* buf, MPI_Count count, MPI_Datatype datatype, int dest, int tag, MPI_Comm comm) Fortran (if implementation is using 32-bit counts) MPI_SEND(BUF, COUNT, DATATYPE, DEST, TAG, COMM, IERROR) BUF(*) INTEGER COUNT, DATATYPE, DEST, TAG, COMM, IERROR Fortran (if implementation is using 64-bit counts) MPI_SEND(BUF, COUNT, DATATYPE, DEST, TAG, COMM, IERROR) BUF(*) INTEGER DATATYPE, DEST, TAG, COMM, IERROR INTEGER*8 COUNT

9. #3 – Drop Fortran 77 support Proposal #3: – Much of the problem with Proposal #2 is the “ugliness” of the Fortran 77 solution. – If we go with Fortran90/95 then we have two options Use TYPE() to create an MPI_Count type – This enters new territory for MPI standard since all Fortran arguments are currently simple types. OR Just allow implementation to specify INTEGER*4 or INTEGER*8 and let casting handle it. (Requires a Fortran wrapper to each routine written in Fortran?)

10. #3 – Drop Fortran 77 support Proposal #3a: Use TYPE to declare a fortran “MPI_Count” – Pros: A solution exists for C, C++ and Fortran – Cons Like C, existing code now “works but isn’t MPI standard compliant” We’ve never used a TYPE for any other Fortran arguments Proposal #3b: Use automatic type matching in both C and Fortran 90 (no MPI_Count) – MPI_Send now takes a long int in C and an INTEGER*8 in Fortran 90/95 – Pros: Existing code is MPI Compliant! User’s can pass ints or long ints, INTEGER*4 or INTEGER*8 and everything will get converted for them! – Cons: Did I mention we would have to drop fortran 77 support? Or… not support long counts in fortran77…

11. #Final – Only change C, C++ and Fortran 90/95 Although cores are getting more plentiful and relatively less powerful, I think we can expect hybrid programming models to increase the need for large off-node message counts – we should do something. Don’t drop Fortran 77, just make it take a different size of integer than Fortran 90/95. Using MPI_Count for C/C++ is optional in my opinion and may be detrimental because it makes people fearful that they have to change their code.