1. Parallel Data Transfer in the Model Coupling Toolkit Robert L. Jacob J. Walter Larson Mathematics and Computer Science Division Argonne National Laboratory

2. A Typical Coupled Model AGCM Temp, Rain, Radiation, Wind Sea Ice Temp, Albedo OGCM Temp, Currents, Salinity Land Surface Temp, Soil Moisture Coupler

3. Basic Function of a Coupler: Move data from a model that has it to a model that needs it Other, minor functions Interpolate data onto different grids Calculate data needed by one model but not calculated by any other model Control model integration Ensure conservation (many other misc. functions)

4. Component 1 Component 2 Coupler Memory Coupling and Computer Memory: Shared memory Component 1 Component 2 Coupler Memory (lightly distributed)

5. Component 1Component 2 Coupler Coupling and Computer memory: Distributed memory Memory Ex: NCAR Parallel Climate Model

6. Component 1Component 2 Coupler Coupling and Computer memory: Distributed memory Ex: NCAR Climate System Model Memory

7. Component 1Component 2 Coupler Coupling and Computer memory: Distributed memory Memory How do you decide where the data goes? On any decomposition?

8. Idea: For efficiency and scalability, send only the data needed to each node. MCT provides data types and algorithms to find this data transfer pattern for any two decompositions of any numerical grid GlobalSegmentMap : Describes decomposition Router : Describes data transfer between GSMaps Attribute Vector : Holds data to be sent/received

9. GlobalSegMap Example Pe_loc start length 0 1 8 1 9 8 2 17 4 0 1 2 1234 5678 9101112 13141516 17181920 1234 5678 9101112 13141516 17181920 01 2 3 Pe_loc start length 0 1 2 0 5 2 0 9 2 1 3 2 1 7 2 1 11 2 2 13 2 2 17 2 3 15 2 3 19 2 Numbering of gridpoints Processor Decomposition Total number of segments = 10 Total number of segments = 3

10. 1234 5678 9101112 13141516 17181920 01 2 3 4 5 6 1234 5678 9101112 13141516 17181920 Pe_loc start length 0 1 2 0 5 2 0 9 2 1 3 2 1 7 2 1 11 2 2 13 2 2 17 2 3 15 2 3 19 2 GlobalSegMap Pe_loc start length 0 1 8 1 9 8 2 17 4 GlobalSegMap Processor Decomposition AtmosphereCoupler Router Router Example

11. Initializing a Router 1234 5678 9101112 13141516 17181920 01 2 3 3 Segments: (1,2) (5,6) (9,10 ) Each processor is given a copy of other component’s GlobalSegMap. Each processor compares each of its segments against other component’s entire GSMap. Each processor constructs a list of what processors it must send to, and which parts of its GSMap must be sent to each This information is stored in the Router datatype Router resembles a GSMap: it’s a list of remote processors and starts and counts of local index values NOTE: Router is between two different GSMaps of the same grid

12. Tqu Real Data MaskVeg typ Integer Data Attribute Vector

13. Pe_loc start length 0 1 2 0 5 2 0 9 2 Relationship Between GlobalSegMap and AttributeVector Processor 0’s portion of GlobalSegMap 6 One Attribute from processor 0’s AttVect 1 2 3 4 5 6 Local index 1 2 5 6 9 10 GSMap values

14. Rules for using GlobalSegmentMap and MCT Parallel Data Transfer Each grid’s gridpoints are numbered with unique integers Model’s which use the same grid must number points the same way Components can exchange only real and integer data as groups of vectors in an AttributeVector Features Any number of components, any grid, any decomposition, any number of processors-per-component is allowed

15. ATM (M nodes) Call MCT World Define GlobalSegMap Define AttrVect Define Router Call MCT World Define GlobalSegMaps Define AttrVects Define Routers Define Accumulators Read Matrix elements Call MCT World Define GlobalSegMap Define AttrVect Define Router CPL (N nodes)OCN (P nodes) Initialization Read Atmosphere Data Read Ocean Data MCT_Send(AtrVect, Router)MCT_Recv(AAtrVect, ARouter) MCT_Recv(OAtrVect, ORouter) Interpolate Compute Fluxes MCT_Send(AAtrVect, ARouter) Accumulate MCT_Send(OAtrVect, ORouter) MCT_Send(AtrVect, Router) MCT_Recv(AtrVect, Router) Data Transfer in the MCT unit tester

16. Identical decompositions Atmosphere on 4 nodes, coupler on 4 nodes

17. Total transfer time for easy case No. of atmosphere nodes = No. of coupler nodes

18. Two very different decompositions Atmosphere on 16 nodes, coupler on 4 = 20 nodes total

19. Total transfer time for pathological case Atmosphere on 8 nodes, 16 nodes, 32 nodes

20. Future of MCT Data Transfer Support for Parallel Transpose Intra-model load balancing Increase Performance/Scalability Automatic GlobalSegMap generation for standard decompositions

21. ATM (M nodes) Call MCT World Define GlobalSegMap Define AttrVect Define Router Call MCT World Define GlobalSegMaps Define AttrVects Define Routers Define Accumulators Read Matrix elements Call MCT World Define GlobalSegMap Define AttrVect Define Router CPL (N nodes)OCN (P nodes) Initialization Read Atmosphere Data Read Ocean Data MCT_Send(AtrVect, Router)MCT_Recv(AAtrVect, ARouter) MCT_Recv(OAtrVect, ORouter) Interpolate Compute Fluxes MCT_Send(AAtrVect, ARouter) Accumulate MCT_Send(OAtrVect, ORouter) MCT_Recv(AAtrVect, Router) Interpolate Compute Fluxes MCT_Send(AAtrVect, Router) Accumulate MCT_Recv(AtrVect, Router) Future of MCT Data Transfer?