1. OASIS3 and OASIS4 : the PRISM couplers G. Berti, P. Bourcier, A. Caubel, D. Declat, M.-A. Foujols, J. Latour, S. Legutke, J. Polcher, R. Redler, H. Ritzdorf, T. Schoenemeyer, S. Valcke and R. Vogelsang FLUME-PRISM Workshop Met Office, Exeter, March 15-16, 2004

2. FLUME-PRISM workshop – Met Office, Exeter, 15-16, 2004 Slide 2 Outline OASIS historic OASIS community today PRISM first coupler: Oasis3 model adaptation coupled model configuration communication interpolations/transformations PRISM final coupler: Oasis4 model adaptation the Driver communication interpolations/transformations in the prototype still to be done Conclusions

3. FLUME-PRISM workshop – Met Office, Exeter, 15-16, 2004 Slide 3 OASIS: Ocean Atmosphere Sea Ice Soil OASIS historic Developed since 1991 in CERFACS to couple existing GCMs. At the time: Models at relatively low resolution (~10000-20000 pts) Small number of 2D coupling fields (~10) Low coupling frequency (~once/day)  flexibility was very important, efficiency not so much!

4. FLUME-PRISM workshop – Met Office, Exeter, 15-16, 2004 Slide 4 1993: OASIS 1.0 (CRAY pipes & files) 1995: OASIS 2.0 (PVM3): Heterogeneous coupling (Cathode) 1996: OASIS 2.1: Global warning scenario (SIDDACLICH) 1998: OASIS 2.2 (SIPC): Seasonal forecasting (Demeter) 1999: OASIS 2.3 (GMEM - NEC): Intercomparison (SINTEX) 2000: OASIS 2.4 (MPI2): Climate Natural variability (Predicate) 2003: OASIS 3 (MPI2-MPI1 + PSMILe + I/O): PRISM => new Oasis3 (oasis3_prism_2-1) now available ! OASIS historic

5. FLUME-PRISM workshop – Met Office, Exeter, 15-16, 2004 Slide 5 CERFACS (France) ARPEGE3 -ORCA2LIM ARPEGE3 -OPA 8.1ARPEGE3 - OPAICE METEO-FRANCE (France) ARPEGE4 - ORCA2ARPEGE medias -OPAmed ARPEGE3 - OPA8.1ARPEGE2 - OPA TDH IPSL- LODYC, LMD, LSCE (France) LMDz - ORCA2LIM LMDz - ORCA4LMDz - OPA ATL3/ATL1 IFS - OPA 8.1ECHAM4 -ORCA2 MERCATOR (France) PAM(OPA) MPI - M&D (Germany) ECHAM5 -MPI-OMECHAM5 -C-HOPE PUMA -C-HOPEEMAD -E-HOPE ECHAM5 -E-HOPEECHAM4 -E-HOPE ECMWF (UK) IFS Cy23r4 -E-HOPE IFS Cy15r8 -E-HOPE OASIS community today

6. FLUME-PRISM workshop – Met Office, Exeter, 15-16, 2004 Slide 6 CGAM-Reading (UK) HadAM3 - ORCA2 Southampton University (UK) Inter. Atm -OCCAM lite UCL (Belgium) LMDz - CLIO SMHI (Sweden) ECHAM - RCARCA(region.) – RCO(region.) U. of Bergen (Norway) MM5 -ROMS KNMI (Netherlands) ECHAM5 -MPI-OM DMI (Danemark) ECHAM -HIRLAM INGV (Italy) ECHAM5 – MPI-OM IRI (USA) ECHAM4 - MOM3 JAMSTEC (Japan) ECHAM4 -OPA 8.2 BMRC (Australia) BAM - MOM4BAM3 - ACOM2 U. of Tasmania (Australia)Data Atm. -MOM4 CAS,IIT Delhi (India)MM5 - POM OASIS community today

7. FLUME-PRISM workshop – Met Office, Exeter, 15-16, 2004 Slide 7 PRISM first coupler: Oasis3 Mono-process coupler + parallel coupling library (PSMILe) synchronisation of the component models coupling fields exchange I/O actions mono-process interpolation Flexibility, modularity: Arbitrary number of models and coupling fields Different coupling frequencies for each field Different transformations for each field Static coupling: all coupling parameters defined initially.

8. FLUME-PRISM workshop – Met Office, Exeter, 15-16, 2004 Slide 8 Oasis3: model adaptation (1/3) PRISM System Model Interface Library (PSMILe) API : Initialization: call prism_init_comp (comp_id, comp_name, ierr) Retrieval of component local communicator: call prism_get_localcomm (local_comm, ierr) Global grid definition (master process only) call prism_write_grid (grd_name, nx, ny, lon, lat) call prism_write_corner (grd_name, nx, ny, nc, clon, clat) call prism_write_mask (grd_name, nx, ny, mask) call prism_write_areas (grd_name, nx, ny, area)

9. FLUME-PRISM workshop – Met Office, Exeter, 15-16, 2004 Slide 9 Oasis3: model adaptation (2/3) PRISM System Model Interface Library (PSMILe) API : Local partition definition (by each process in the global index space) call prism_def_partition (part_id, ig_paral, ierr) Coupling or I/O field declaration (name, partition id, status, shape, type) call prism_def_var (var_id, var_name, part_id, var_nbrdims, var_inout, var_shape, var_type, ierr) End of definition call prism_enddef (ierr)

10. FLUME-PRISM workshop – Met Office, Exeter, 15-16, 2004 Slide 10 Oasis3: model adaptation (3/3) Coupling or I/O field sending and receiving:  in model time stepping loop  depending on user’s specifications in namcouple: user’s defined source or target (end-point communication) coupling or I/O sending or receiving at appropriate times automatic averaging/accumulation automatic writing of coupling restart file at end of run call prism_put (var_id, time, var_array, ierr) call prism_get (var_id, time, var_array, ierr) Termination: call prism_terminate (ierr) Auxiliary routines: call prism_put_inquire (var_id, time, info) call prism_put_restart (var_id, time, info) PRISM System Model Interface Library (PSMILe) API: :

11. FLUME-PRISM workshop – Met Office, Exeter, 15-16, 2004 Slide 11 Oasis3: coupled model configuration In text file namcouple, read by Oasis3 main process, and distributed to component model PSMILes at beginning of run: total run time component models number of coupling fields for each coupling field: source and target names (end-point communication) (var_name) grid acronym (grid_name) coupling and/or I/O status coupling or I/O period transformations/interpolations

12. FLUME-PRISM workshop – Met Office, Exeter, 15-16, 2004 Slide 12 Oasis3: communication PSMILe based on MPI1 or MPI2 message passing End-point communication; one-to-one A A A B B B Direct communication between models with same grid and partitioning A A A file I/O functionality (automatic switch between coupled and forced mode): GFDL mpp_io library A A A O O O O Oasis3 Parallel communication between parallel models and Oasis3 interpolation process

13. FLUME-PRISM workshop – Met Office, Exeter, 15-16, 2004 Slide 13 Oasis3: interpolations/transformations  separate sequential process neighbourhood search weight calculation interpolation per se during the run A A A O O O O Oasis3  on 2D scalar fields only Interfacing with RPN Fast Scalar INTerpolator package nearest-neighbour, bilinear, bicubic for regular Lat-Lon grids Interfacing with SCRIP1.4 library nearest-neighbour, 1st and 2nd order conservative remapping for all grids bilinear and bicubic interpolation for «logically-rectangular» grids Bilinear and bicubic interpolation for reduced atmospheric grids Other spatial transformations: flux correction, merging, etc. General algebraic operations

14. FLUME-PRISM workshop – Met Office, Exeter, 15-16, 2004 Slide 14 PRISM final coupler: Oasis4 Situation today: Higher resolution, parallel and scalable models Higher coupling frequencies desirable Higher number of models and (3D) coupling fields Massively parallel platforms Need to optimise and parallelise the coupler Oasis4 is composed of: a Driver a Transformer a new PRISM System Model Interface Library

15. FLUME-PRISM workshop – Met Office, Exeter, 15-16, 2004 Slide 15 PRISM final coupler: Oasis4 Some keynotes:  Developers: Sophie Valcke (CERFACS) - Design, XML input Damien Declat (CERFACS) - Driver, transformer, XML input René Redler (NEC CCRLE)- PSMIle, portability to Linux PC cluster Hubert Ritzdorf (NEC CCRLE) - Parallel search, transformer interaction Reiner Vogelsang (SGI)- I/O, portability to SGI Th. Schoenemeyer (NEC HPCE) - Parallelisation, portability to NEC SX Jean Latour (Fujitsu )- Portability to Fujitsu VPP and PC cluster  Public domain; open source license (LGPL)  All external libraries and modules used exist in public domain versions; vendor optimized versions may also exist: MPI1 and/or MPI2 mpp_io; NetCDF/parallel NetCDF libXML SCRIP  Programming language: Fortran 90 and C

16. FLUME-PRISM workshop – Met Office, Exeter, 15-16, 2004 Slide 16 Oasis4: model adaptation (1/5) New PSMILe API : Application initialization (1 call/application) : call prism_init (appl_name, ierr) Component application (1 call/component) : call prism_init_comp (comp_id, comp_name, ierr) Retrieval of component local communicator (1 call/component) : call prism_get_localcomm (comp_id, local_comm, ierr) Inquire SCC or SMIOCs persistent values: call prism_get_persist (…)

17. FLUME-PRISM workshop – Met Office, Exeter, 15-16, 2004 Slide 17 Oasis4: model adaptation (2/5) Definition of grid (3D) call prism_def_grid (grd_id, grd_name, comp_id, grd_shape, type, ierr) call prism_set_corners (grd_id, nbr_crnr, crnr_shape, crnr_array, ierr) Placement of scalar points, vector points, subgrids on the grid: call prism_set_points ( pt_id, pt_name, grd_id, pt_shape, pt_lon, pt_lat, pt_vert,ierr) call prism_set_vector (vct_id, vct_name, array_of_pt_ids,ierr) call prism_set_subgrid (sgrd_id, sgrd_name, grd_id, n_sgrd, sgrd_shape, sgrd_array, ierr)  Function overloading to keep the interface concise and flexible

18. FLUME-PRISM workshop – Met Office, Exeter, 15-16, 2004 Slide 18 Oasis4: model adaptation (3/5) Definition of grid cell dimension call prism_set_scalefactors (grd_id, scale_shape, scale_array, ierr) Definition of grid cell mask call prism_set_mask (msk_id, grd_id, msk_shape, msk_array, ierr) Definition of point local coordinate system angle call prism_set_angle (pt_id, angle_shape, angle_array, ierr)

19. FLUME-PRISM workshop – Met Office, Exeter, 15-16, 2004 Slide 19 prism_def_grid Example prism_set_mask prism_set_points prism-set-corners Oasis4: model adaptation (4/5) prism_set_vector

20. FLUME-PRISM workshop – Met Office, Exeter, 15-16, 2004 Slide 20 Oasis4: model adaptation (5/5) Coupling or I/O field declaration call prism_def_var (var_id, var_name, grd_id, pt_id, msk_id, var_nbrdims, var_shape, var_type, ierr) End of definition call prism_enddef (ierr) Coupling or I/O field sending and receiving:  in model time stepping loop  depending on user’s specifications in SMIOC: user’s defined source or target, component or file (end-point communication) coupling or I/O sending or receiving at appropriate times automatic averaging/accumulation automatic writing of coupling restart file at end of run call prism_put (var_id, date, date_bounds, var_array, info, ierr) call prism_get (var_id, date, date_bounds, var_array, info, ierr)

21. FLUME-PRISM workshop – Met Office, Exeter, 15-16, 2004 Slide 21 Application and component description (XML files):  For each application (code): one Application Description (AD): possible number of processes components included  For each component in the application: one Potential Model Input and Output Description (PMIOD) component general characteristics: name, component simulated, … code general characteristics: lab, documentation, contact, … grid information: domain, resolution(s), grid type, … potential I/O or coupling variables: local name, standard name units, valid min and max numerical type associated grid and points intent –input and/or output persistent variables: name, units, numerical type, possible values, … Oasis4: coupled model configuration (1/2)

22. FLUME-PRISM workshop – Met Office, Exeter, 15-16, 2004 Slide 22 Coupled model configuration (XML files): The user produces  a Specific Coupling Configuration (SCC): experiment and run start date and end date start mode (MPI1, MPI2) applications, components for each application host(s), number of processes per host, ranks for each component  For each component, a Specific Model Input and Output Configuration (SMIOC) PMIOD component and code general characteristics grid information: chosen resolution, … I/O or coupling variables: local and standard name, units, valid min max, numerical type, grid activated intent –input and/or output source and/or target (component and/or file) coupling or I/O dates transformations/interpolations/combinations persistent variables : name, units, numerical type, chosen value, … Oasis4: coupled model configuration (2/2)

23. Driver T OCE ATM LAND fileV4 Definition Phase OCE PMIOD V1: out, metadata V1 V2: in, metadata V2 ATM PMIOD V1: in, metadata V1 V2: out, metadata V2 V3: out, metadata V3 LAND PMIOD V3: in, metadata V3 V4: in, metadata V4 ATM-LAND AD OCE AD OCE SMIOC V1 : to ATM, T1 to file V1 V2 : from ATM, T2 user ATM SMIOC V1 : from OCE, T1 V2: to OCE, T2 V3 : to LAND user LAND SMIOC V3 : from ATM V4 : from fileV4 user Composition Phase user SCC ATM:... OCE:... LAND:... user Deployment Phase V1 V2 V3 fileV1 V4

24. FLUME-PRISM workshop – Met Office, Exeter, 15-16, 2004 Slide 24 Oasis4: the Driver Manages a static coupling: all applications are started directly in the run script (MPI1 mode) or launched by the Driver at the beginning of the run (MPI2 mode) for the whole run. Participates to communicator set-up Reads the SCC and SMIOcs information and distributes it to the component PSMILes Driver

25. FLUME-PRISM workshop – Met Office, Exeter, 15-16, 2004 Slide 25 More elaborate PSMILe based on MPI1 or MPI2 Parallel communication including repartitioning: based on geographical description of the partitions parallel calculation of communication patterns in source PSMILe OB C C C O1 C C Same grid, different decomposition  direct repartitioning: for each target point, parallel calculation of source matching point in source PSMILe Oasis4: communication (1/3) T OB C C C O2 C C Different grid and decomposition  via Transformer: for each target point, parallel calculation of source interpolation neighbours in source PSMILe

26. FLUME-PRISM workshop – Met Office, Exeter, 15-16, 2004 Slide 26 Oasis4: communication (2/3) x x x x x x x x x x o o o 1/ Envelop exchange 2/ Detailed neighbourhood search 3/ EPIOS and EPIOT definition Parallel calculation of communication patterns in source PSMILe For each pair of source and target processes: x x x x x x x x x o o o x x x x x x x x x o o o EPIOS1 EPIOT1 x x x x x o o o x x x x x x o o o x EPIOS2 EPIOT2

27. FLUME-PRISM workshop – Met Office, Exeter, 15-16, 2004 Slide 27 parallel 3D neighbourhood search in each source process PSMILe new search algorithm: parallel multigrid technique, higher efficiency extraction of useful part of source field only end-point communication one-to-one or one-to-many; combination source data can be consumed by more than one target model or file and at a different frequencies Oasis4: communication (3/3) parallel I/O ( vector, bundles, vector bundles) : GFDL mpp_io, par NetCDF A A A file single parallel A A A file1 file2 file3 distributed A A A file

28. FLUME-PRISM workshop – Met Office, Exeter, 15-16, 2004 Slide 28 PSMILe: time transformations, local transformations global conservation, combinations Transformer: Oasis4: interpolations/transformations T OB C C C O2 C C loops over PSMILe requests EPIOS and EPIOT definition (source PSMILe) For each EPIOT point, neighbours in EPIOS (source PSMILe) =>Transformer calculates interpolation weights Prism_put of EPIOS (source PSMILe) =>Transformer performs interpolation per se Prism_get of EPIOT (target PSMILe) well suited for MPI parallelisation

29. FLUME-PRISM workshop – Met Office, Exeter, 15-16, 2004 Slide 29 Different interpolation schemes: nearest-neighbour, bilinear, bicubic, conservative remapping, … On different types of fields 1D-2D-3D scalar, vector, bundles, bundles of vectors On different types of grids blockstructured (regular and irregular) unstructured static or dynamic Oasis4: interpolations/transformations

30. FLUME-PRISM workshop – Met Office, Exeter, 15-16, 2004 Slide 30 Access of XML information Coupling and I/O of n parallel applications with m components. Coupling exchange with repartitionning, direct or through the (mono-process) Transformer Interpolations: PSMILe (non-exact) parallel neighborhood search 3D nearest-neighbour I/O: single and parallel mode Coupling and I/O exchange from one source to many targets Local transformations (scatter, gather, add or mult scalar, statistics) Basic time transformation (average, accumul, min, max) Oasis4: in the prototype (03/2004)

31. FLUME-PRISM workshop – Met Office, Exeter, 15-16, 2004 Slide 31 Completion of PSMILe calendar routines (internal and external) PSMILe API for model access to SCC and SMIOC information Interpolation: More schemes (conservative, 3D, etc.) Exact parallel neighbour search Transformer parallelisation Field reduction, combination Full support of vector and bundle fields (I/O OK) I/O: distributed mode (parNetCDF) Adaptative grids Unstructured grids Oasis4: still to be done

32. FLUME-PRISM workshop – Met Office, Exeter, 15-16, 2004 Slide 32 Conclusions Calendar: Oasis3_prism_2-1 available Oasis4 prototype available within days Oasis4 final PRISM version due 12/2004 Follow-on: Oasis4 developments will go on after 12/2004 with PRISM sustained support FP6 funding (march 2005)? Static vs dynamic couplers: In the near and mid future, Oasis4 and Palm (Cerfacs dynamical coupler) will remain separate software Fusion may be considered when climate modelling and data assimilation communities will join

33. FLUME-PRISM workshop – Met Office, Exeter, 15-16, 2004 Slide 33 The end