1. Met Office, Exeter, March 15-16, 2004
FLUME-PRISM Workshop
Met Office, Exeter, March 15-16, 2004

2. FLUME-PRISM workshop – Met Office, Exeter, 15-16, 2004
Outline
OASIS historic
OASIS community today
PRISM first coupler: Oasis3
model adaptation
coupled model configuration
communication
interpolations/transformations
PRISM final coupler: Oasis4
the Driver
in the prototype
still to be done
Conclusions
FLUME-PRISM workshop – Met Office, Exeter, 15-16, 2004

3. FLUME-PRISM workshop – Met Office, Exeter, 15-16, 2004
OASIS historic
OASIS: Ocean Atmosphere Sea Ice Soil
Developed since 1991 in CERFACS to couple existing GCMs. At the time:
Models at relatively low resolution (~ pts)
Small number of 2D coupling fields (~10)
Low coupling frequency (~once/day)
flexibility was very important, efficiency not so much!
FLUME-PRISM workshop – Met Office, Exeter, 15-16, 2004

4. FLUME-PRISM workshop – Met Office, Exeter, 15-16, 2004
OASIS historic
1993: OASIS 1.0 (CRAY pipes & files)
1995: OASIS 2.0 (PVM3): Heterogeneous coupling (Cathode)
1996: OASIS 2.1: Global warning scenario (SIDDACLICH)
1998: OASIS 2.2 (SIPC): Seasonal forecasting (Demeter)
1999: OASIS 2.3 (GMEM - NEC): Intercomparison (SINTEX)
2000: OASIS 2.4 (MPI2): Climate Natural variability (Predicate)
2003: OASIS 3 (MPI2-MPI1 + PSMILe + I/O): PRISM
=> new Oasis3 (oasis3_prism_2-1) now available !
FLUME-PRISM workshop – Met Office, Exeter, 15-16, 2004

5. FLUME-PRISM workshop – Met Office, Exeter, 15-16, 2004
OASIS community today
CERFACS (France) ARPEGE ORCA2LIM ARPEGE OPA 8.1 ARPEGE OPAICE
METEO-FRANCE (France) ARPEGE ORCA2 ARPEGE medias -OPAmed ARPEGE OPA8.1 ARPEGE OPA TDH
IPSL- LODYC, LMD, LSCE (France) LMDz ORCA2LIM LMDz ORCA4 LMDz OPA ATL3/ATL1 IFS OPA 8.1 ECHAM ORCA2
MERCATOR (France) PAM(OPA)
MPI - M&D (Germany) ECHAM MPI-OM ECHAM C-HOPE PUMA C-HOPE EMAD E-HOPE ECHAM E-HOPE ECHAM E-HOPE
ECMWF (UK) IFS Cy23r4 - E-HOPE IFS Cy15r8 - E-HOPE
Rajouetr les couplages
FLUME-PRISM workshop – Met Office, Exeter, 15-16, 2004

6. FLUME-PRISM workshop – Met Office, Exeter, 15-16, 2004
OASIS community today
CGAM-Reading (UK) HadAM ORCA2
Southampton University (UK) Inter. Atm - OCCAM lite
UCL (Belgium) LMDz CLIO
SMHI (Sweden) ECHAM - RCA RCA(region.) – RCO(region.)
U. of Bergen (Norway) MM ROMS
KNMI (Netherlands) ECHAM MPI-OM
DMI (Danemark) ECHAM HIRLAM
INGV (Italy) ECHAM – MPI-OM
IRI (USA) ECHAM MOM3
JAMSTEC (Japan) ECHAM OPA 8.2
BMRC (Australia) BAM - MOM4 BAM ACOM2
U. of Tasmania (Australia) Data Atm MOM4
CAS,IIT Delhi (India) MM POM
Rajouetr les couplages
FLUME-PRISM workshop – Met Office, Exeter, 15-16, 2004

7. PRISM first coupler: Oasis3
Mono-process coupler + parallel coupling library (PSMILe)
synchronisation of the component models
coupling fields exchange
I/O actions
mono-process interpolation
Flexibility, modularity:
Arbitrary number of models and coupling fields
Different coupling frequencies for each field
Different transformations for each field
Static coupling: all coupling parameters defined initially.
FLUME-PRISM workshop – Met Office, Exeter, 15-16, 2004

8. Oasis3: model adaptation (1/3)
PRISM System Model Interface Library (PSMILe) API :
Initialization:
call prism_init_comp (comp_id, comp_name, ierr)
Retrieval of component local communicator:
call prism_get_localcomm (local_comm, ierr)
Global grid definition (master process only)
call prism_write_grid (grd_name, nx, ny, lon, lat)
call prism_write_corner (grd_name, nx, ny, nc, clon, clat)
call prism_write_mask (grd_name, nx, ny, mask)
call prism_write_areas (grd_name, nx, ny, area)
FLUME-PRISM workshop – Met Office, Exeter, 15-16, 2004

9. Oasis3: model adaptation (2/3)
PRISM System Model Interface Library (PSMILe) API :
Local partition definition (by each process in the global index space)
call prism_def_partition (part_id, ig_paral, ierr)
Coupling or I/O field declaration (name, partition id, status, shape, type)
call prism_def_var
(var_id, var_name, part_id, var_nbrdims, var_inout,
var_shape, var_type, ierr)
End of definition
call prism_enddef (ierr)
FLUME-PRISM workshop – Met Office, Exeter, 15-16, 2004

10. Oasis3: model adaptation (3/3)
PRISM System Model Interface Library (PSMILe) API: :
Coupling or I/O field sending and receiving:
in model time stepping loop
depending on user’s specifications in namcouple:
user’s defined source or target (end-point communication)
coupling or I/O sending or receiving at appropriate times
automatic averaging/accumulation
automatic writing of coupling restart file at end of run
call prism_put (var_id, time, var_array, ierr)
call prism_get (var_id, time, var_array, ierr)
Termination:
call prism_terminate (ierr)
Auxiliary routines:
call prism_put_inquire (var_id, time, info)
call prism_put_restart (var_id, time, info)
FLUME-PRISM workshop – Met Office, Exeter, 15-16, 2004

11. Oasis3: coupled model configuration
In text file namcouple, read by Oasis3 main process, and distributed to component model PSMILes at beginning of run:
total run time
component models
number of coupling fields
for each coupling field:
source and target names (end-point communication) (var_name)
grid acronym (grid_name)
coupling and/or I/O status
coupling or I/O period
transformations/interpolations
FLUME-PRISM workshop – Met Office, Exeter, 15-16, 2004

12. FLUME-PRISM workshop – Met Office, Exeter, 15-16, 2004
Oasis3: communication
PSMILe based on MPI1 or MPI2 message passing
End-point communication; one-to-one A O
Oasis3
Parallel communication between parallel models and Oasis3 interpolation process A B
Direct communication between models with same grid and partitioning A
file
I/O functionality (automatic switch between coupled and forced mode): GFDL mpp_io library
FLUME-PRISM workshop – Met Office, Exeter, 15-16, 2004

13. Oasis3: interpolations/transformations
separate sequential process
neighbourhood search
weight calculation
interpolation per se during the run A O
Oasis3
on 2D scalar fields only
Interfacing with RPN Fast Scalar INTerpolator package
nearest-neighbour, bilinear, bicubic for regular Lat-Lon grids
Interfacing with SCRIP1.4 library
nearest-neighbour, 1st and 2nd order conservative remapping for all grids
bilinear and bicubic interpolation for «logically-rectangular» grids
Bilinear and bicubic interpolation for reduced atmospheric grids
Other spatial transformations: flux correction, merging, etc.
General algebraic operations
FLUME-PRISM workshop – Met Office, Exeter, 15-16, 2004

14. PRISM final coupler: Oasis4
Situation today:
Higher resolution, parallel and scalable models
Higher coupling frequencies desirable
Higher number of models and (3D) coupling fields
Massively parallel platforms
Need to optimise and parallelise the coupler
Oasis4 is composed of:
a Driver
a Transformer
a new PRISM System Model Interface Library
FLUME-PRISM workshop – Met Office, Exeter, 15-16, 2004

15. PRISM final coupler: Oasis4
Some keynotes:
Developers:
Sophie Valcke (CERFACS) - Design, XML input
Damien Declat (CERFACS) - Driver, transformer, XML input
René Redler (NEC CCRLE) - PSMIle, portability to Linux PC cluster
Hubert Ritzdorf (NEC CCRLE) - Parallel search, transformer interaction
Reiner Vogelsang (SGI) - I/O, portability to SGI
Th. Schoenemeyer (NEC HPCE) - Parallelisation, portability to NEC SX
Jean Latour (Fujitsu ) - Portability to Fujitsu VPP and PC cluster
Public domain; open source license (LGPL)
All external libraries and modules used exist in public domain versions; vendor optimized versions may also exist:
MPI1 and/or MPI2
mpp_io; NetCDF/parallel NetCDF
libXML
SCRIP
Programming language: Fortran 90 and C
FLUME-PRISM workshop – Met Office, Exeter, 15-16, 2004

16. Oasis4: model adaptation (1/5)
New PSMILe API:
Application initialization (1 call/application) :
call prism_init
(appl_name, ierr)
Component application (1 call/component) :
call prism_init_comp
(comp_id, comp_name, ierr)
Retrieval of component local communicator (1 call/component):
call prism_get_localcomm
(comp_id, local_comm, ierr)
Inquire SCC or SMIOCs persistent values:
call prism_get_persist (…)
FLUME-PRISM workshop – Met Office, Exeter, 15-16, 2004

17. Oasis4: model adaptation (2/5)
Definition of grid (3D)
call prism_def_grid
(grd_id, grd_name, comp_id, grd_shape, type, ierr)
call prism_set_corners
(grd_id, nbr_crnr, crnr_shape, crnr_array, ierr)
Placement of scalar points, vector points, subgrids on the grid:
call prism_set_points
(pt_id, pt_name, grd_id, pt_shape, pt_lon, pt_lat, pt_vert ,ierr)
call prism_set_vector
(vct_id, vct_name, array_of_pt_ids,ierr)
call prism_set_subgrid
(sgrd_id, sgrd_name, grd_id, n_sgrd, sgrd_shape, sgrd_array, ierr)
Function overloading to keep the interface concise and flexible
FLUME-PRISM workshop – Met Office, Exeter, 15-16, 2004

18. Oasis4: model adaptation (3/5)
Definition of grid cell dimension
call prism_set_scalefactors
(grd_id, scale_shape, scale_array, ierr)
Definition of grid cell mask
call prism_set_mask
(msk_id, grd_id, msk_shape, msk_array, ierr)
Definition of point local coordinate system angle
call prism_set_angle
(pt_id, angle_shape, angle_array, ierr)
FLUME-PRISM workshop – Met Office, Exeter, 15-16, 2004

19. Example Oasis4: model adaptation (4/5) prism_def_grid
prism-set-corners
prism_set_points
prism_set_vector
prism_set_points
prism_set_points
prism_set_mask
FLUME-PRISM workshop – Met Office, Exeter, 15-16, 2004

20. Oasis4: model adaptation (5/5)
Coupling or I/O field declaration
call prism_def_var
(var_id, var_name, grd_id, pt_id, msk_id, var_nbrdims,
var_shape, var_type, ierr)
End of definition
call prism_enddef (ierr)
Coupling or I/O field sending and receiving:
in model time stepping loop
depending on user’s specifications in SMIOC:
user’s defined source or target, component or file (end-point communication)
coupling or I/O sending or receiving at appropriate times
automatic averaging/accumulation
automatic writing of coupling restart file at end of run
call prism_put (var_id, date, date_bounds, var_array, info, ierr)
call prism_get (var_id, date, date_bounds, var_array, info, ierr)
FLUME-PRISM workshop – Met Office, Exeter, 15-16, 2004

21. Oasis4: coupled model configuration (1/2)
Application and component description (XML files):
For each application (code):
one Application Description (AD):
possible number of processes
components included
For each component in the application:
one Potential Model Input and Output Description (PMIOD)
component general characteristics: name, component simulated, …
code general characteristics: lab, documentation, contact, …
grid information: domain, resolution(s), grid type, …
potential I/O or coupling variables:
local name, standard name
units, valid min and max
numerical type
associated grid and points
intent –input and/or output
persistent variables: name, units, numerical type, possible values, …
FLUME-PRISM workshop – Met Office, Exeter, 15-16, 2004

22. Oasis4: coupled model configuration (2/2)
Coupled model configuration (XML files): The user produces
a Specific Coupling Configuration (SCC):
experiment and run start date and end date
start mode (MPI1, MPI2)
applications, components for each application
host(s), number of processes per host, ranks for each component
For each component,
a Specific Model Input and Output Configuration (SMIOC)
PMIOD component and code general characteristics
grid information: chosen resolution, …
I/O or coupling variables:
local and standard name, units, valid min max, numerical type, grid
activated intent –input and/or output
source and/or target (component and/or file)
coupling or I/O dates
transformations/interpolations/combinations
persistent variables: name, units, numerical type, chosen value, …
FLUME-PRISM workshop – Met Office, Exeter, 15-16, 2004

23. T Definition Phase Composition Phase Deployment Phase Driver user
OCE
ATM
LAND
fileV4
Definition Phase
OCE PMIOD
V1: out, metadata V1
V2: in, metadata V2
ATM PMIOD
V1: in, metadata V1
V2: out, metadata V2
V3: out, metadata V3
LAND PMIOD
V3: in, metadata V3
V4: in, metadata V4
ATM-LAND AD
OCE AD
OCE SMIOC
V1 : to ATM, T1
to file V1
V2 : from ATM, T2
user
ATM SMIOC
V1 : from OCE, T1
V2: to OCE, T2
V3 : to LAND
LAND SMIOC
V3 : from ATM
V4 : from fileV4
Composition Phase
SCC
ATM:...
OCE:...
LAND:...
Deployment Phase V1 V2 V3
fileV1 V4

24. FLUME-PRISM workshop – Met Office, Exeter, 15-16, 2004
Oasis4: the Driver
Driver
Manages a static coupling:
all applications are started directly in the run script (MPI1 mode) or launched by the Driver at the beginning of the run (MPI2 mode) for the whole run.
Participates to communicator set-up
Reads the SCC and SMIOcs information and distributes it to the component PSMILes
FLUME-PRISM workshop – Met Office, Exeter, 15-16, 2004

25. Oasis4: communication (1/3)
More elaborate PSMILe based on MPI1 or MPI2
Parallel communication including repartitioning:
based on geographical description of the partitions
parallel calculation of communication patterns in source PSMILe T OB C O2
Different grid and decomposition
via Transformer:
for each target point, parallel calculation of source interpolation neighbours in source PSMILe OB C O1
Same grid, different decomposition
direct repartitioning:
for each target point, parallel calculation of source matching point in source PSMILe
FLUME-PRISM workshop – Met Office, Exeter, 15-16, 2004

26. Oasis4: communication (2/3)
x x x x x x x x x x x x x x x x x x x x
o o o
Parallel calculation of communication patterns in source PSMILe
For each pair of source and target processes:
1/ Envelop exchange
2/ Detailed neighbourhood search
3/ EPIOS and EPIOT definition
x x x x x x x x x
o o o
EPIOS1
EPIOT1
x x x x x
o o o
x x
EPIOS2
EPIOT2
FLUME-PRISM workshop – Met Office, Exeter, 15-16, 2004

27. Oasis4: communication (3/3)
parallel 3D neighbourhood search in each source process PSMILe
new search algorithm: parallel multigrid technique, higher efficiency
extraction of useful part of source field only
end-point communication
one-to-one or one-to-many; combination
source data can be consumed by more than one target model or file and at a different frequencies
parallel I/O (vector, bundles, vector bundles) : GFDL mpp_io, parNetCDF A
file
single
parallel
file1
file2
file3
distributed
FLUME-PRISM workshop – Met Office, Exeter, 15-16, 2004

28. Oasis4: interpolations/transformations
PSMILe:
time transformations, local transformations
global conservation, combinations
Transformer: T OB C O2
loops over PSMILe requests
EPIOS and EPIOT definition (source PSMILe)
For each EPIOT point, neighbours in EPIOS (source PSMILe)
=>Transformer calculates interpolation weights
Prism_put of EPIOS (source PSMILe)
=>Transformer performs interpolation per se
Prism_get of EPIOT (target PSMILe)
well suited for MPI parallelisation
FLUME-PRISM workshop – Met Office, Exeter, 15-16, 2004

29. Oasis4: interpolations/transformations
Different interpolation schemes:
nearest-neighbour, bilinear, bicubic, conservative remapping, …
On different types of fields
1D-2D-3D scalar, vector, bundles, bundles of vectors
On different types of grids
blockstructured (regular and irregular)
unstructured
static or dynamic
FLUME-PRISM workshop – Met Office, Exeter, 15-16, 2004

30. FLUME-PRISM workshop – Met Office, Exeter, 15-16, 2004
Oasis4: in the prototype (03/2004)
Access of XML information
Coupling and I/O of n parallel applications with m components.
Coupling exchange with repartitionning, direct or through the (mono-process) Transformer
Interpolations:
PSMILe (non-exact) parallel neighborhood search
3D nearest-neighbour
I/O:
single and parallel mode
Coupling and I/O exchange from one source to many targets
Local transformations (scatter, gather, add or mult scalar, statistics)
Basic time transformation (average, accumul, min, max)
FLUME-PRISM workshop – Met Office, Exeter, 15-16, 2004

31. FLUME-PRISM workshop – Met Office, Exeter, 15-16, 2004
Oasis4: still to be done
Completion of PSMILe calendar routines (internal and external)
PSMILe API for model access to SCC and SMIOC information
Interpolation:
More schemes (conservative, 3D, etc.)
Exact parallel neighbour search
Transformer parallelisation
Field reduction, combination
Full support of vector and bundle fields (I/O OK)
I/O: distributed mode (parNetCDF)
Adaptative grids
Unstructured grids
FLUME-PRISM workshop – Met Office, Exeter, 15-16, 2004

32. FLUME-PRISM workshop – Met Office, Exeter, 15-16, 2004
Conclusions
Calendar:
Oasis3_prism_2-1 available
Oasis4 prototype available within days
Oasis4 final PRISM version due 12/2004
Follow-on:
Oasis4 developments will go on after 12/2004 with PRISM sustained support
FP6 funding (march 2005)?
Static vs dynamic couplers:
In the near and mid future, Oasis4 and Palm (Cerfacs dynamical coupler) will remain separate software
Fusion may be considered when climate modelling and data assimilation communities will join
FLUME-PRISM workshop – Met Office, Exeter, 15-16, 2004

33. FLUME-PRISM workshop – Met Office, Exeter, 15-16, 2004
The end
FLUME-PRISM workshop – Met Office, Exeter, 15-16, 2004