WRF Model: Software Architecture and Code Structure John Michalakes, NCAR NCAR: W. Skamarock, J. Dudhia, D. Gill, A. Bourgeois, W. Wang, C. Deluca, R.

WRF Model: Software Architecture and Code Structure John Michalakes, NCAR NCAR: W. Skamarock, J. Dudhia, D. Gill, A. Bourgeois, W. Wang, C. Deluca, R. Loft NOAA/NCEP: Tom Black, Jim Purser, S. Gopal NOAA/FSL: T. Henderson, J. Middlecoff, L.Hart U. Oklahoma: M. Xue AFWA: J. Wegiel, D. McCormick C. Coats (MCNC), J. Schmidt (NRL), V. Balaji (GFDL) S. Chen (UC Davis), J. Edwards (IBM) Acknowledgement : Significant funding for WRF software development from DoD HPCMO CHSSI program (CWO6)

Outline Software architecture and implementation –Goals and design aspects –Parallelism –Code Structure and Model Layer Interface –Registry –I/O Extending WRF: Example

WRF Software Goals Community Model Good performance Portable across a range of architectures Flexible, maintainable, understandable Facilitate code reuse Multiple dynamics/ physics options Run-time configurable Nested Aspects of Design Single-source code Fortran90 modules, dynamic memory, structures, recursion Hierarchical software architecture Multi-level parallelism CASE: Registry Package-neutral APIs –I/O, data formats –Communication IKJ Storage Order

OMP Software Architecture Config Inquiry I/O API Config Module WRF Tile-callable Subroutines Solve Mediation Layer Model Layer Driver Layer DM comm Threads External Packages Package Independent Package Dependent Data formats, Parallel I/O Message Passing Driver Registry

Model domains are decomposed for parallelism on two-levels Patch: section of model domain allocated to a distributed memory node Tile: section of a patch allocated to a shared-memory processor within a node; this is also the scope of a model layer subroutine. Distributed memory parallelism is over patches; shared memory parallelism is over tiles within patches Single version of code for efficient execution on: –Distributed-memory –Shared-memory –Clusters of SMPs –Vector and microprocessors WRF Multi-Layer Domain Decomposition Logical domain 1 Patch, divided into multiple tiles Inter-processor communication

Parallelism and you Some code you add to WRF won’t entail any extra effort on your part other than conforming to the WRF model layer interface specification Exceptions –Horizontal interpolations involving neighboring cells –Upper radiative boundary conditions –Dynamics and diffusion –Fully 3D packages (that is, not “column physics”) –Tell tale sign: if you have i+integer or i+integer array indices in right-hand side expressions in your code

Example (dyn_eh/module_diffusion.F ) SUBROUTINE horizontal_diffusion_s (tendency, rr, var,...... DO j = jts,jte DO k = kts,ktf DO i = its,ite mrdx=msft(i,j)*rdx mrdy=msft(i,j)*rdy tendency(i,k,j)=tendency(i,k,j)- & (mrdx*0.5*((rr(i+1,k,j)+rr(i,k,j))*H1(i+1,k,j)- & (rr(i-1,k,j)+rr(i,k,j))*H1(i,k,j))+ & mrdy*0.5*((rr(i,k,j+1)+rr(i,k,j))*H2(i,k,j+1)- & (rr(i,k,j-1)+rr(i,k,j))*H2(i,k,j ))- & msft(i,j)*(H1avg(i,k+1,j)-H1avg(i,k,j)+ & H2avg(i,k+1,j)-H2avg(i,k,j) & )/dzetaw(k) & ) ENDDO... ! ! 0 pt 4 pt 8 pt 24 pt ! ! * * * * * ! * * * * * * * * * ! + * + * * + * * * + * * ! * * * * * * * * * ! * * * * * !msft x !rr x !

Halo updates patch ims ime jms jme Distributed Memory Communications patch ims ime memory on one processormemory on neighboring processor

Halo updates Distributed Memory Communications memory on one processormemory on neighboring processor * + * * * * Halo update of second time level of rr may be defined in Registry as: halo HALO_DIFF dyn_em 4:rr_2 Halo update of second time level of rr may be defined in Registry as: halo HALO_DIFF dyn_em 4:rr_2 Halo update is invoked in the code (in the solve routine) as: #include “HALO_DIFF.incl” prior to the call to the diffusion routine Halo update is invoked in the code (in the solve routine) as: #include “HALO_DIFF.incl” prior to the call to the diffusion routine

Halo updates Periodic boundary updates Distributed Memory Communications Period update of variable t may be defined in Registry as: period PBDY dyn_em 3:t Period update of variable t may be defined in Registry as: period PBDY dyn_em 3:t Period update is invoked in the code as: #include “PBDY.incl” Period update is invoked in the code as: #include “PBDY.incl”

Halo updates Periodic boundary updates Parallel transposes all z on patch all x on patch all y on patch Distributed Memory Communications Transpose of u defined in Registry as: xpose TRANS_A dyn_em u,ux,uy Note separate definitions of u, ux, and uy: state real u ikj … state real ux ikjx … state real uy ikjy … Transpose of u defined in Registry as: xpose TRANS_A dyn_em u,ux,uy Note separate definitions of u, ux, and uy: state real u ikj … state real ux ikjx … state real uy ikjy … Transpose from Z decomposition to X decomposition is invoked in the code as: #include “TRANS_A_z2x.incl” Transpose from Z decomposition to X decomposition is invoked in the code as: #include “TRANS_A_z2x.incl”

OMP Software Architecture Config Inquiry I/O API Config Module WRF Tile-callable Subroutines Solve Mediation Layer Model Layer Driver Layer DM comm Threads External Packages Package Independent Package Dependent Data formats, Parallel I/O Message Passing Driver Registry

Code Structure OMP Config Inquiry I/O API Config Module WRF Tile-callable Subroutines Solve Mediation Layer Model Layer Driver Layer DM comm Threads External Packages Package Independent Package Dependent Data formats, Parallel I/O Message Passing Driver Registry wrf (main/wrf.F) integrate (frame/module_integrate.F) solve_interface (share/solve_interface.F) solve_em (dyn_em/solve_em.F) advance_uv (dyn_em/module_small_step_em.F) cumulus_driver (physics/cumulus_driver.F)

WRF Model Layer Interface Interface Mediation layer <> Model Layer –All state arrays passed through argument list as simple (not derived) data types –Domain, memory, and run dimensions passed unambiguously in three physical dimensions –Restrictions on model layer subroutines No I/O, communication, no stops or aborts (use wrf_error_fatal in frame/module_wrf_error.F) No common/module storage of decomposed data (exception allowed for set-once/read-only tables) Spatial scope of a Model Layer call is one “tile” Temporal scope of a call is limited by coherency Computation on halos is allowed and considered a model-layer concern Benefits –Ensures that a sub-domain is tile-callable and therefore portable to any decomposition supported by the driver –Provides a common interface that Allows use of other conforming driver layers (this positions the code for new developments such as ESMF) WRF driver layer can (and is) being used to parallelize other models OMP Config Inquiry I/O API Config Module WRF Tile-callable Subroutines Solve DM comm Threads Data formats, Parallel I/O Message Passing Driver SUBROUTINE solve_xxx (... !$OMP DO PARALLEL DO ij = 1, numtiles its = i_start(ij) ; ite = i_end(ij) jts = j_start(ij) ; jte = j_end(ij) CALL model_subroutine( arg1, arg2,... ids, ide, jds, jde, kds, kde, ims, ime, jms, jme, kms, kme, its, ite, jts, jte, kts, kte ) END DO... END SUBROUTINE SUBROUTINE solve_xxx (... !$OMP DO PARALLEL DO ij = 1, numtiles its = i_start(ij) ; ite = i_end(ij) jts = j_start(ij) ; jte = j_end(ij) CALL model_subroutine( arg1, arg2,... ids, ide, jds, jde, kds, kde, ims, ime, jms, jme, kms, kme, its, ite, jts, jte, kts, kte ) END DO... END SUBROUTINE template for model layer subroutine SUBROUTINE model_subroutine ( & arg1, arg2, arg3, …, argn, & ids, ide, jds, jde, kds, kde, & ! Domain dims ims, ime, jms, jme, kms, kme, & ! Memory dims its, ite, jts, jte, kts, kte ) ! Tile dims IMPLICIT NONE ! Define Arguments (S and I1) data REAL, DIMENSION (ims:ime,kms:kme,jms:jme) :: arg1,... REAL, DIMENSION (ims:ime,jms:jme) :: arg7,...... ! Define Local Data (I2) REAL, DIMENSION (its:ite,kts:kte,jts:jte) :: loc1,...... ! Executable code; loops run over tile ! dimensions DO j = jts, jte DO k = kts, kte DO i = MAX(its,ids), MIN(ite,ide) loc(i,k,j) = arg1(i,k,j) + … END DO template for model layer subroutine SUBROUTINE model_subroutine ( & arg1, arg2, arg3, …, argn, & ids, ide, jds, jde, kds, kde, & ! Domain dims ims, ime, jms, jme, kms, kme, & ! Memory dims its, ite, jts, jte, kts, kte ) ! Tile dims IMPLICIT NONE ! Define Arguments (S and I1) data REAL, DIMENSION (ims:ime,kms:kme,jms:jme) :: arg1,... REAL, DIMENSION (ims:ime,jms:jme) :: arg7,...... ! Define Local Data (I2) REAL, DIMENSION (its:ite,kts:kte,jts:jte) :: loc1,...... ! Executable code; loops run over tile ! dimensions DO j = jts, jte DO k = kts, kte DO i = MAX(its,ids), MIN(ite,ide) loc(i,k,j) = arg1(i,k,j) + … END DO

template for model layer subroutine SUBROUTINE model ( & arg1, arg2, arg3, …, argn, & ids, ide, jds, jde, kds, kde, & ! Domain dims ims, ime, jms, jme, kms, kme, & ! Memory dims its, ite, jts, jte, kts, kte ) ! Tile dims IMPLICIT NONE ! Define Arguments (S and I1) data REAL, DIMENSION (ims:ime,kms:kme,jms:jme) :: arg1,... REAL, DIMENSION (ims:ime,jms:jme) :: arg7,...... ! Define Local Data (I2) REAL, DIMENSION (its:ite,kts:kte,jts:jte) :: loc1,...... ! Executable code; loops run over tile ! dimensions DO j = jts, jte DO k = kts, kte DO i = MAX(its,ids), MIN(ite,ide) loc(i,k,j) = arg1(i,k,j) + … END DO template for model layer subroutine SUBROUTINE model ( & arg1, arg2, arg3, …, argn, & ids, ide, jds, jde, kds, kde, & ! Domain dims ims, ime, jms, jme, kms, kme, & ! Memory dims its, ite, jts, jte, kts, kte ) ! Tile dims IMPLICIT NONE ! Define Arguments (S and I1) data REAL, DIMENSION (ims:ime,kms:kme,jms:jme) :: arg1,... REAL, DIMENSION (ims:ime,jms:jme) :: arg7,...... ! Define Local Data (I2) REAL, DIMENSION (its:ite,kts:kte,jts:jte) :: loc1,...... ! Executable code; loops run over tile ! dimensions DO j = jts, jte DO k = kts, kte DO i = MAX(its,ids), MIN(ite,ide) loc(i,k,j) = arg1(i,k,j) + … END DO Domain dimensions Size of logical domain Used for bdy tests, etc.

template for model layer subroutine SUBROUTINE model ( & arg1, arg2, arg3, …, argn, & ids, ide, jds, jde, kds, kde, & ! Domain dims ims, ime, jms, jme, kms, kme, & ! Memory dims its, ite, jts, jte, kts, kte ) ! Tile dims IMPLICIT NONE ! Define Arguments (S and I1) data REAL, DIMENSION (ims:ime,kms:kme,jms:jme) :: arg1,... REAL, DIMENSION (ims:ime,jms:jme) :: arg7,...... ! Define Local Data (I2) REAL, DIMENSION (its:ite,kts:kte,jts:jte) :: loc1,...... ! Executable code; loops run over tile ! dimensions DO j = jts, jte DO k = kts, kte DO i = MAX(its,ids), MIN(ite,ide) loc(i,k,j) = arg1(i,k,j) + … END DO template for model layer subroutine SUBROUTINE model ( & arg1, arg2, arg3, …, argn, & ids, ide, jds, jde, kds, kde, & ! Domain dims ims, ime, jms, jme, kms, kme, & ! Memory dims its, ite, jts, jte, kts, kte ) ! Tile dims IMPLICIT NONE ! Define Arguments (S and I1) data REAL, DIMENSION (ims:ime,kms:kme,jms:jme) :: arg1,... REAL, DIMENSION (ims:ime,jms:jme) :: arg7,...... ! Define Local Data (I2) REAL, DIMENSION (its:ite,kts:kte,jts:jte) :: loc1,...... ! Executable code; loops run over tile ! dimensions DO j = jts, jte DO k = kts, kte DO i = MAX(its,ids), MIN(ite,ide) loc(i,k,j) = arg1(i,k,j) + … END DO Domain dimensions Size of logical domain Used for bdy tests, etc. Memory dimensions Used to dimension dummy arguments Do not use for local arrays

template for model layer subroutine SUBROUTINE model ( & arg1, arg2, arg3, …, argn, & ids, ide, jds, jde, kds, kde, & ! Domain dims ims, ime, jms, jme, kms, kme, & ! Memory dims its, ite, jts, jte, kts, kte ) ! Tile dims IMPLICIT NONE ! Define Arguments (S and I1) data REAL, DIMENSION (ims:ime,kms:kme,jms:jme) :: arg1,... REAL, DIMENSION (ims:ime,jms:jme) :: arg7,...... ! Define Local Data (I2) REAL, DIMENSION (its:ite,kts:kte,jts:jte) :: loc1,...... ! Executable code; loops run over tile ! dimensions DO j = jts, jte DO k = kts, kte DO i = MAX(its,ids), MIN(ite,ide) loc(i,k,j) = arg1(i,k,j) + … END DO template for model layer subroutine SUBROUTINE model ( & arg1, arg2, arg3, …, argn, & ids, ide, jds, jde, kds, kde, & ! Domain dims ims, ime, jms, jme, kms, kme, & ! Memory dims its, ite, jts, jte, kts, kte ) ! Tile dims IMPLICIT NONE ! Define Arguments (S and I1) data REAL, DIMENSION (ims:ime,kms:kme,jms:jme) :: arg1,... REAL, DIMENSION (ims:ime,jms:jme) :: arg7,...... ! Define Local Data (I2) REAL, DIMENSION (its:ite,kts:kte,jts:jte) :: loc1,...... ! Executable code; loops run over tile ! dimensions DO j = jts, jte DO k = kts, kte DO i = MAX(its,ids), MIN(ite,ide) loc(i,k,j) = arg1(i,k,j) + … END DO Domain dimensions Size of logical domain Used for bdy tests, etc. Memory dimensions Used to dimension dummy arguments Do not use for local arrays Tile dimensions Local loop ranges Local array dimensions

Registry OMP Config Inquiry I/O API Config Module WRF Tile-callable Subroutines Solve Mediation Layer Model Layer Driver Layer DM comm Threads External Packages Package Independent Package Dependent Data formats, Parallel I/O Message Passing Driver Registry OMP Config Inquiry I/O API Config Module WRF Tile-callable Subroutines Solve Mediation Layer Model Layer Driver Layer DM comm Threads External Packages Package Independent Package Dependent Data formats, Parallel I/O Message Passing Driver Registry

WRF Registry CASE-like “a ctive data-dictionary” for managing WRF data structures –Database describing attributes of model state, intermediate, and configuration data Dimensionality, number of time levels, staggering Association with physics I/O classification (history, initial, restart, boundary) Communication points and patterns Configuration lists (e.g. namelists) –Program for auto-generating sections of WRF from database: 570 Registry entries  30-thousand lines of automatically generated WRF code Allocation statements for state data, I1 data Argument lists for driver layer/mediation layer interfaces Interprocessor communications: Halo and periodic boundary updates, transposes Code for defining and managing run-time configuration information Code for forcing, feedback and interpolation of nest data Automates time consuming, repetitive, error-prone programming Insulates programmers and code from package dependencies Allow rapid development Documents the data # Registry file # # table entries are of the form # state real ru ikj dyn-rk 2 X irh state real u ikj dyn-rk 2 X irh state real ru_m ikj dyn-rk 1 X i1 real ru_tend ikj dyn-rk 1 X... ## communications halo HALO_RK_A 24:u_2,v_2,w_2,t_2,tp_2,rw_2,rom_2;4:pp,pip halo HALO_RK_B 4:rtp_2 halo HALO_RK_C 4:ru_2,rv_2,du,dv...

Registry Mechanics %compile wrf Registry/Registry registry program: tools/registry inc/*.incl WRF source */*.F Fortran90 wrf.exe

Registry Data Base Currently implemented as a text file: Registry/Registry Types of entry: –State – Describes state variables and arrays in the domain structure –Dimspec – Describes dimensions that are used to define arrays in the model –I1 – Describes local variables and arrays in solve –Typedef – Describes derived types that are subtypes of the domain structure –Rconfig – Describes a configuration (e.g. namelist) variable or array –Package – Describes attributes of a package (e.g. physics) –Halo – Describes halo update interprocessor communications –Period – Describes communications for periodic boundary updates –Xpose – Describes communications for parallel matrix transposes

State entry Elements –Entry: The keyword “state” –Type: The type of the state variable or array (real, double, integer, logical, character, or derived) –Sym: The symbolic name of the variable or array –Dims: A string denoting the dimensionality of the array or a hyphen (-) –Use: A string denoting association with a solver or 4D scalar array, or a hyphen –NumTLev: An integer indicating the number of time levels (for arrays) or hypen (for variables) –Stagger: String indicating staggered dimensions of variable (X, Y, Z, or hyphen) –IO: String indicating whether and how the variable is subject to I/O –DName: Metadata name for the variable –Units: Metadata units of the variable –Descrip: Metadata description of the variable Example # Type Sym Dims Use Tlev Stag IO Dname Descrip # definition of a 3D, two-time level, staggered state array state real ru ikj dyn_eh 2 X irh "RHO_U" "X WIND COMPONENT“ # definition of fields in 4D scalar array “moist” state real qv ikjft moist 2 - irh "QVAPOR" "Water vapor mix ratio“ state real qc ikjft moist 2 - irh "QCLOUD" "Cloud water mixing ratio"

Dimspec entry Elements –Entry: The keyword “dimspec” –DimName: The name of the dimension (single character) –Order: The order of the dimension in the WRF framework (1, 2, 3, or ‘-‘) –HowDefined: specification of how the range of the dimension is defined –CoordAxis: which axis the dimension corresponds to, if any (X, Y, Z, or C) –DatName: metadata name of dimension Example # dimspec i 1 standard_domain x west_east dimspec j 3 standard_domain y south_north dimspec k 2 standard_domain z bottom_top dimspec l 2 namelist=num_soil_layers z soil_layers

Rconfig entry Elements –Entry: the keyword “rconfig” –Type: the type of the namelist variable (integer, real, logical – no strings yet) –Sym: the name of the namelist variable or array –How set: indicates how the variable is set: e.g. namelist or derived, and if namelist, which block of the namelist it is set in –Nentries: specifies the dimensionality of the namelist variable or array. If 1 (one) it is a variable and applies to all domains; otherwise specify max_domains (which is an integer parameter defined in module_driver_constants.F). –Default: the default value of the variable to be used if none is specified in the namelist; hyphen (-) for no default Example # Type Sym How set Nentries Default rconfig integer dyn_opt namelist,namelist_01 1 1

Package Entry Elements –Entry: the keyword “package”, –Package name: the name of the package: e.g. “kesslerscheme” –Associated rconfig choice: the name of a rconfig variable and the value of that variable that choses this package –Package state vars: unused at present; specify hyphen (-) –Associated 4D scalars: the names of 4D scalar arrays and the fields within those arrays this package uses Example # specification of microphysics options package passiveqv mp_physics==0 - moist:qv package kesslerscheme mp_physics==1 - moist:qv,qc,qr package linscheme mp_physics==2 - moist:qv,qc,qr,qi,qs,qg package ncepcloud3 mp_physics==3 - moist:qv,qc,qr package ncepcloud5 mp_physics==4 - moist:qv,qc,qr,qi,qs # namelist entry that controls microphysics option rconfig integer mp_physics namelist,namelist_04 max_domains 0

Comm entries: halo and period Elements –Entry: keywords “halo” or “period” –Commname: name of comm operation –Description: defines the halo or period operation For halo: npts:f1,f2,...[;npts:f1,f2,...]* For period: width:f1,f2,...[;width:f1,f2,...]* Example # first exchange in eh solver halo HALO_EH_A dyn_em 24:u_2,v_2,ru_1,ru_2,rv_1,rv_2,w_2,t_2;4:pp,pip # a periodic boundary update period PERIOD_EH_A dyn_em 2:u_1,u_2,ru_1,ru_2,v_1,v_2,rv_1,rv_2,rw_1,rw_2

OMP I/O Architecture Config Inquiry I/O API Config Module WRF Tile-callable Subroutines Solve Mediation Layer Model Layer Driver Layer DM comm Threads External Packages Package Independent Package Dependent Data formats, Parallel I/O Message Passing Driver Registry

I/O Architecture Requirements of I/O Infrastructure –Efficiency: key concern for operations –Flexibility: key concern in research –Both types of user-institution already heavily invested in I/O infrastructure Operations: GRIB, BUFR Research: NetCDF, HDF –“Portable I/O” – adaptable to range of uses, installations without affecting WRF and other programs that use the I/O infrastructure

I/O Architecture WRF I/O API –Package-independent interface to NetCDF, Fast-binary, HDF (planned) –Random access of fields by timestamp/name –Full transposition to arbitrary memory order –Built-in support for read/write of parallel file systems (planned) –Data-set centric, not file-centric (planned); Grid Computing Additional WRF model functionality –Collection/distribution of decomposed data to serial datasets –Fast, asynchronous, “quilt-server” I/O from NCEP Eta model

WRF I/O API Key functions –Initialization and shutdown –Opening and closing datasets –Reading and writing data –Reading and writing metadata

WRF I/O API 7.4.1 Initialization and shutdown SUBROUTINE ext_pkg_ioinit( Status ) INTEGER, INTENT(INOUT) :: Status Synopsis: Initialize the WRF I/O system. SUBROUTINE ext_pkg_ioexit( Status ) INTEGER, INTENT(INOUT) :: Status Synopsis: Shut down the WRF I/O system.

WRF I/O API 7.4.2 Opening and closing datasets SUBROUTINE ext_pkg_open_for_read ( DatasetName, Comm, IOComm_io, & SysDepInfo, DataHandle, Status ) CHARACTER *(*), INTENT(IN) :: DatasetName INTEGER, INTENT(IN) :: Comm, IOComm CHARACTER *(*), INTENT(IN) :: SysDepInfo INTEGER, INTENT(OUT) :: DataHandle INTEGER, INTENT(OUT) :: Status Synopsis: Opens a WRF dataset for reading using the I/O package pkg. Arguments: DatasetName:The name of the dataset to be opened. Comm: Not used. IOComm:Not used. SysDepInfo:A string-valued variable that provides a means to pass additional control information to the I/O interface. The format of the string is a comma-separated list of variable=value pairs. Currently the only control variable now supported is DATASET, which may have the allowable values RESTART, HISTORY, BOUNDARY, or INITIAL (capitals required). For example, the WRF model will pass the string 'DATASET=INITIAL' to open an initial dataset. DataHandle:Returned to the calling program to serve as a handle to the Open dataset for subsequent I/O API operations.

WRF I/O API SUBROUTINE ext_pkg_open_for_write_begin( DatasetName, Comm, IOComm, & SysDepInfo, DataHandle, Status ) CHARACTER *(*), INTENT(IN) :: DatasetName INTEGER, INTENT(IN) :: Comm, IOComm CHARACTER *(*), INTENT(IN) :: SysDepInfo INTEGER, INTENT(OUT) :: DataHandle INTEGER, INTENT(OUT) :: Status SUBROUTINE ext_pkg_open_for_write_commit( DataHandle, Status ) INTEGER, INTENT(IN ) :: DatasetHandle INTEGER, INTENT(OUT) :: Status Synopsis: Begin data definition phase for the WRF dataset DatasetName using the I/O package Pkg. Arguments: DatasetName:The name of the dataset to be opened. Comm: Not used. IOComm:Not used. SysDepInfo:Same as open for read. DataHandle:integer handle. Two-Stage Open For Write There is significant performance advantage for netCDF and presumably other self- describing data format packages in going through a data-definition phase prior to writing data through the package. Thus, the open-for-write is a two-stage operation. A WRF dataset is first opened for data-definition, at which point the model calls the interface (predominantly the ext_pkg_write_field routine) with a complete serious of writes such as would occur during the output of an actual frame of data; however, no data is actually written. This is strictly for the purpose of “training” the interface. Interface implementations that don’t need training can ignore this by simply opening the dataset on the open-begin call and setting an internal flag that disables writing in the write_field routines. ext_pkg_open_for_write_begin must be paired with an ext_pkg_open_for_write_commit.

WRF I/O API SUBROUTINE ext_pkg_write_field ( DataHandle, DateStr, VarName, & Field, FieldType, Comm, IOComm & DomainDesc, MemoryOrder, Stagger, & DimNames, & DomainStart, DomainEnd, & MemoryStart, MemoryEnd, & PatchStart, PatchEnd, & Status ) INTEGER,INTENT(IN) :: DataHandle CHARACTER*(*),INTENT(IN) :: DateStr CHARACTER*(*),INTENT(IN) :: VarName type,INTENT(IN) :: Field(*) INTEGER,INTENT(IN) :: FieldType INTEGER,INTENT(INOUT) :: Comm INTEGER,INTENT(INOUT) :: IOComm INTEGER,INTENT(IN) :: DomainDesc CHARACTER*(*),INTENT(IN) :: MemoryOrder CHARACTER*(*),INTENT(IN) :: Stagger CHARACTER*(*), DIMENSION (*),INTENT(IN) :: DimNames INTEGER,DIMENSION(*),INTENT(IN) :: DomainStart, DomainEnd INTEGER,DIMENSION(*),INTENT(IN) :: MemoryStart, MemoryEnd INTEGER,DIMENSION(*),INTENT(IN) :: PatchStart, PatchEnd INTEGER,INTENT(OUT) :: Status

WRF I/O API SUBROUTINE ext_pkg_write_field ( DataHandle, DateStr, VarName, & Field, FieldType, Comm, IOComm & DomainDesc, MemoryOrder, Stagger, & DimNames, & DomainStart, DomainEnd, & MemoryStart, MemoryEnd, & PatchStart, PatchEnd, & Status ) INTEGER,INTENT(IN) :: DataHandle CHARACTER*(*),INTENT(IN) :: DateStr CHARACTER*(*),INTENT(IN) :: VarName type,INTENT(IN) :: Field(*) INTEGER,INTENT(IN) :: FieldType INTEGER,INTENT(INOUT) :: Comm INTEGER,INTENT(INOUT) :: IOComm INTEGER,INTENT(IN) :: DomainDesc CHARACTER*(*),INTENT(IN) :: MemoryOrder CHARACTER*(*),INTENT(IN) :: Stagger CHARACTER*(*), DIMENSION (*),INTENT(IN) :: DimNames INTEGER,DIMENSION(*),INTENT(IN) :: DomainStart, DomainEnd INTEGER,DIMENSION(*),INTENT(IN) :: MemoryStart, MemoryEnd INTEGER,DIMENSION(*),INTENT(IN) :: PatchStart, PatchEnd INTEGER,INTENT(OUT) :: Status Data names and time stamps The variable Name provides the name of the variable being read/written; the DateStr character string is the index into the series of values of the variable/array in the dataset. In WRF, this is a 23 character date string of the form “0000-01-01_00:00:00.0000” and is treated as a temporal index. An external implementation of the WRF I/O API is required to randomly access a dataset based on the name and timestamp pair specified in the argument list. From the point of view of a package implementing the I/O API, the timestamp string has no meaning or relationship with any other timestamp other than lexicographic: two strings are either the same or they are different; there is no meaning associated with the timestamp string. The interface does not do time comparisons but it is aware of control breaks (change in the sequence of timestamps in the interface). The interface may also store a record of the sequence in which fields were written to a dataset and provide this information in a way that allows sequential traversal through the dataset but datasets are fundamentally random access by variable name and timestamp string. The interface may assume and expect that all the fields in a frame of data have the same date string and that on writing the calls to the interface to write them will have been called in consecutively.

WRF I/O API SUBROUTINE ext_pkg_write_field ( DataHandle, DateStr, VarName, & Field, FieldType, Comm, IOComm & DomainDesc, MemoryOrder, Stagger, & DimNames, & DomainStart, DomainEnd, & MemoryStart, MemoryEnd, & PatchStart, PatchEnd, & Status ) INTEGER,INTENT(IN) :: DataHandle CHARACTER*(*),INTENT(IN) :: DateStr CHARACTER*(*),INTENT(IN) :: VarName type,INTENT(IN) :: Field(*) INTEGER,INTENT(IN) :: FieldType INTEGER,INTENT(INOUT) :: Comm INTEGER,INTENT(INOUT) :: IOComm INTEGER,INTENT(IN) :: DomainDesc CHARACTER*(*),INTENT(IN) :: MemoryOrder CHARACTER*(*),INTENT(IN) :: Stagger CHARACTER*(*), DIMENSION (*),INTENT(IN) :: DimNames INTEGER,DIMENSION(*),INTENT(IN) :: DomainStart, DomainEnd INTEGER,DIMENSION(*),INTENT(IN) :: MemoryStart, MemoryEnd INTEGER,DIMENSION(*),INTENT(IN) :: PatchStart, PatchEnd INTEGER,INTENT(OUT) :: Status Full 2-D/3D Arrays 1-D (Column) Arrays Non- decomposed Boundary 3-DBoundary 2-D ‘XYZ’‘Z’‘C’‘XSZ’ (west)‘XS’ (west) ‘XZY’ ‘XEZ’ (east)‘XE’ (east) ‘ZXY’0-D (Scalars) ‘YSZ’ (south)‘YS’ (south) ‘XY’‘0’ (zero) ‘YEZ’ (north)‘YE’ (north) ‘YX’

WRF I/O API SUBROUTINE ext_pkg_write_field ( DataHandle, DateStr, VarName, & Field, FieldType, Comm, IOComm & DomainDesc, MemoryOrder, Stagger, & DimNames, & DomainStart, DomainEnd, & MemoryStart, MemoryEnd, & PatchStart, PatchEnd, & Status ) INTEGER,INTENT(IN) :: DataHandle CHARACTER*(*),INTENT(IN) :: DateStr CHARACTER*(*),INTENT(IN) :: VarName type,INTENT(IN) :: Field(*) INTEGER,INTENT(IN) :: FieldType INTEGER,INTENT(INOUT) :: Comm INTEGER,INTENT(INOUT) :: IOComm INTEGER,INTENT(IN) :: DomainDesc CHARACTER*(*),INTENT(IN) :: MemoryOrder CHARACTER*(*),INTENT(IN) :: Stagger CHARACTER*(*), DIMENSION (*),INTENT(IN) :: DimNames INTEGER,DIMENSION(*),INTENT(IN) :: DomainStart, DomainEnd INTEGER,DIMENSION(*),INTENT(IN) :: MemoryStart, MemoryEnd INTEGER,DIMENSION(*),INTENT(IN) :: PatchStart, PatchEnd INTEGER,INTENT(OUT) :: Status Staggering Staggered means with respect to mass-point coordinates. For example, in WRF the Eulerian height solver uses an Arakawa C-grid so that the U-velocity is staggered in X, the V-velelocity is staggered in Y, and the vertical velocity are staggered in Z. The stagger string is specified to the read_field and write_field using a string consisting of the characters X, Y, and/or Z that specifies which dimensions the field is staggered in (or the empty string if the variable is not staggered in any dimension).

WRF I/O API SUBROUTINE ext_pkg_write_field ( DataHandle, DateStr, VarName, & Field, FieldType, Comm, IOComm & DomainDesc, MemoryOrder, Stagger, & DimNames, & DomainStart, DomainEnd, & MemoryStart, MemoryEnd, & PatchStart, PatchEnd, & Status ) INTEGER,INTENT(IN) :: DataHandle CHARACTER*(*),INTENT(IN) :: DateStr CHARACTER*(*),INTENT(IN) :: VarName type,INTENT(IN) :: Field(*) INTEGER,INTENT(IN) :: FieldType INTEGER,INTENT(INOUT) :: Comm INTEGER,INTENT(INOUT) :: IOComm INTEGER,INTENT(IN) :: DomainDesc CHARACTER*(*),INTENT(IN) :: MemoryOrder CHARACTER*(*),INTENT(IN) :: Stagger CHARACTER*(*), DIMENSION (*),INTENT(IN) :: DimNames INTEGER,DIMENSION(*),INTENT(IN) :: DomainStart, DomainEnd INTEGER,DIMENSION(*),INTENT(IN) :: MemoryStart, MemoryEnd INTEGER,DIMENSION(*),INTENT(IN) :: PatchStart, PatchEnd INTEGER,INTENT(OUT) :: Status Dimension Information Each of the six arguments show are integer arrays whose length is the number of dimensions of the data item being written (specified by the MemoryOrder string). DomainStart and Domain are the starts and ends of the undecomposed array in each dimension. MemoryStart and MemoryEnd are the starts and ends of each local array dimension as it is declared in the program. PatchStart and PatchEnd are the starts and ends of the local subdomain on this process.

WRF I/O API call ext_ncd_ioinit(Status) call ext_ncd_open_for_write_begin ( "WRF_DATA", 0, 0, "", dh1, Status ) start_memory(1) = ims ; end_memory(1) = ime start_memory(2) = kms ; end_memory(2) = kme start_memory(3) = jms ; end_memory(3) = jme start_domain(1) = 1 ; end_domain(1) = ide-1 start_domain(2) = 1 ; end_domain(2) = kde-1 start_domain(3) = 1 ; end_domain(3) = jde-1 call ext_ncd_write_field(dh1,"2000-01-24_12:00:00","T",t,WRF_REAL,0,0,0, & "XZY"," ", & dimnames, & start_domain,end_domain, & !domain start_memory,end_memory, & !memory start_domain,end_domain, & !patch Status) start_domain(1) = 1 ; end_domain(1) = ide start_domain(2) = 1 ; end_domain(2) = kde-1 start_domain(3) = 1 ; end_domain(3) = jde-1 call ext_ncd_write_field(dh1,"2000-01-24_12:00:00","U",u,WRF_REAL,0,0,0, & "XZY","X", & dimnames_x, & start_domain,end_domain, & !domain start_memory,end_memory, & !memory start_domain,end_domain, & !patch Status) start_domain(1) = 1 ; end_domain(1) = ide-1 start_domain(2) = 1 ; end_domain(2) = kde-1 start_domain(3) = 1 ; end_domain(3) = jde call ext_ncd_write_field(dh1,"2000-01-24_12:00:00","V",v,WRF_REAL,0,0,0, & "XZY","Y", & dimnames_y, & start_domain,end_domain, & !domain start_memory,end_memory, & !memory start_domain,end_domain, & !patch Status) call ext_ncd_open_for_write_commit ( dh1, Status ) ! file now open for writing

WRF I/O API ! file now open for writing start_domain(1) = 1 ; end_domain(1) = ide-1 start_domain(2) = 1 ; end_domain(2) = kde-1 start_domain(3) = 1 ; end_domain(3) = jde-1 call ext_ncd_write_field(dh1,"2000-01-24_12:00:00","T",t,WRF_REAL,0,0,0, & "XZY"," ", & dimnames, & start_domain,end_domain, & !domain start_memory,end_memory, & !memory start_domain,end_domain, & !domain Status) start_domain(1) = 1 ; end_domain(1) = ide start_domain(2) = 1 ; end_domain(2) = kde-1 start_domain(3) = 1 ; end_domain(3) = jde-1 call ext_ncd_write_field(dh1,"2000-01-24_12:00:00","U",u,WRF_REAL,0,0,0, & "XZY","X", & dimnames_x, & start_domain,end_domain, & !domain start_memory,end_memory, & !memory start_domain,end_domain, & !domain Status) start_domain(1) = 1 ; end_domain(1) = ide-1 start_domain(2) = 1 ; end_domain(2) = kde-1 start_domain(3) = 1 ; end_domain(3) = jde call ext_ncd_write_field(dh1,"2000-01-24_12:00:00","V",v,WRF_REAL,0,0,0, & "XZY","Y", & dimnames_y, & start_domain,end_domain, & !domain start_memory,end_memory, & !memory start_domain,end_domain, & !domain Status) call ext_ncd_ioclose(dh1, Status ) call ext_ncd_ioexit( Status )...

WRF I/O API netcdf WRF_DATA { dimensions: Time = UNLIMITED ; // (1 currently) DateStrLen = 19 ; west_east = 199 ; south_north = 299 ; bottom_top = 34 ; west_east_stag = 200 ; south_north_stag = 300 ; variables: char Times(Time, DateStrLen) ; float T(Time, bottom_top, south_north, west_east) ; T:FieldType = 104 ; T:MemoryOrder = "XYZ" ; float U(Time, bottom_top, south_north, west_east_stag) ; U:FieldType = 104 ; U:MemoryOrder = "XYZ" ; float V(Time, bottom_top, south_north_stag, west_east) ; V:FieldType = 104 ; V:MemoryOrder = "XYZ" ; data: Times = "2000-01-24_12:00:00" ;... Output from ncdump

Additional Notes on WRF I/O Calls for reading and writing time-dependent and time-independent meta data associated with datasets and individual variables A sample reader program is distributed with WRF in external/io_netcdf/diffwrf.F90 The WRF framework provides higher level I/O routines (see Sec. 7.2): Higher level calls to read/write fields from an entire domain Separate Initial, History, Restart and Boundary I/O streams; 4 auxiliary Initial and History streams Separately run-time assignable I/O formats to I/O streams Parallel/Quilted I/O Integrated and controllable through the Registry; little to actually code

Example: Adding a new core Conceptual –WRF framework can slot in new dynamics as run-time selectable option Changes to: –Mediation layer, model layer –Registry Reuse: –Top-level driver layer –I/O infrastructure –Parallel infrastructure Registry

Example: Adding a new core Steps –Develop new or convert existing code: Mediation layer routine: solve Model layer subroutines called by solver –Add to WRF Add code to source tree Incorporate into build mechanism Registry entries: data, solver options, comms Some additional splicing –Single processor testing –Analyze data-dependencies, define and implement communication for parallelism –Multi-processor testing OMP Config Inquiry I/O API Config Module WRF Tile-callable Subroutines Solve DM comm Threads Data formats, Parallel I/O MessagePassing Driver OMP Config Inquiry I/O API Config Module WRF Tile-callable Subroutines Solve DM comm Threads Data formats, Parallel I/O MessagePassing Driver Registry

PSEUDO CODE FOR NEW WRF SOLVER: Time loop outside of solver (part of WRF driver: integrate) DO 1 <- number of iterations subroutine solve_exp ( x_1, x_2 ) for each i,j X 1,i,j+1 X 2,ij <= X 1,i-1,j X 1,i+1,j X 1,i,j-1 for each i,j X 1,ij <= X 2,ij end subroutine solve_exp End time loop SOLVE_EXP

!WRF:MEDIATION_LAYER:SOLVER SUBROUTINE solve_exp ( grid, & ! #include "exp_dummy_args.inc" ! ) CALL set_tiles ( grid,... ) #ifdef DM_PARALLEL # include "HALO_EXP_A.inc" #endif !$OMP PARALLEL DO & !$OMP PRIVATE ( ij ) DO ij = 1, grid%num_tiles CALL comp_1_into_2 ( x_1, x_2, & ids, ide, jds, jde, kds, kde, & ims, ime, jms, jme, kms, kme, & grid%i_start(ij), grid%i_end(ij), & grid%j_start(ij), grid%j_end(ij), & k_start, k_end ) END DO !$OMP PARALLEL DO & !$OMP PRIVATE ( ij ) DO ij = 1, grid%num_tiles CALL copy_2_into_1 ( x_2, x_1, & ids, ide, jds, jde, kds, kde, & ims, ime, jms, jme, kms, kme, & grid%i_start(ij), grid%i_end(ij), & grid%j_start(ij), grid%j_end(ij), & k_start, k_end ) END DO SOLVE_EXP

Example: Adding a new core Steps –Develop new or convert existing code: Mediation layer routine: solve Model layer subroutines called by solver –Add to WRF Add code to source tree Incorporate into build mechanism Registry entries: data, solver options, comms Some additional splicing –Single processor testing –Analyze data-dependencies, define and implement communication for parallelism –Multi-processor testing OMP Config Inquiry I/O API Config Module WRF Tile-callable Subroutines Solve DM comm Threads Data formats, Parallel I/O MessagePassing Driver OMP Config Inquiry I/O API Config Module WRF Tile-callable Subroutines Solve DM comm Threads Data formats, Parallel I/O MessagePassing Driver Registry

!WRF:MODEL_LAYER:DYNAMICS ! MODULE module_exp USE module_state_description CONTAINS !------------------------------------------------------------------- SUBROUTINE comp_1_into_2 ( x1, x2, & ids, ide, jds, jde, kds, kde, & ims, ime, jms, jme, kms, kme, & its, ite, jts, jte, kts, kte ) DO j = jts, jte IF ( j > jds.AND. j < jde-1 ) THEN DO k = kts, kte DO i = its, ite IF ( i > ids.AND. i < ide-1 ) THEN x2(i,k,j) = 0.25*(x1(i+1,k,j)+x1(i-1,k,j)+ & x1(i,k,j+1)+x1(i,k,j-1)) ENDIF ENDDO ENDIF ENDDO END SUBROUTINE comp_1_into_2 SUBROUTINE copy_2_into_1 ( x2, x1, & ids, ide, jds, jde, kds, kde, & ims, ime, jms, jme, kms, kme, & its, ite, jts, jte, kts, kte )... module_exp

Example: Adding a new core Steps –Develop new or convert existing code: Mediation layer routine: solve Model layer subroutines called by solver –Add to WRF Add code to source tree Incorporate into build mechanism Registry entries: data, solver options, comms Some additional splicing –Single processor testing –Analyze data-dependencies, define and implement communication for parallelism –Multi-processor testing

Example: Adding a new core Steps –Develop new or convert existing code: Mediation layer routine: solve Model layer subroutines called by solver –Add to WRF Add code to source tree Incorporate into build mechanism Registry entries: data, solver options, comms Some additional splicing –Single processor testing –Analyze data-dependencies, define and implement communication for parallelism –Multi-processor testing Create dyn_exp/Makefile Edit top-level WRFV1/Makefile Additions to clean script

Example: Adding a new core Steps –Develop new or convert existing code: Mediation layer routine: solve Model layer subroutines called by solver –Add to WRF Add code to source tree Incorporate into build mechanism Registry entries: data, solver options, comms Some additional splicing –Single processor testing –Analyze data-dependencies, define and implement communication for parallelism –Multi-processor testing Additions to Registry/Registry file: # define the state variable for new core state real x ikj dyn_exp 2 - ih "TOYVAR" # value of namelist variable dyn_opt for this core package dyn_exp dyn_opt==4 # four-point halo-exchange on first time level of x halo HALO_EXP_A dyn_exp 4:x_1

Example: Adding a new core Steps –Develop new or convert existing code: Mediation layer routine: solve Model layer subroutines called by solver –Add to WRF Add code to source tree Incorporate into build mechanism Registry entries: data, solver options, comms Some additional splicing –Single processor testing –Analyze data-dependencies, define and implement communication for parallelism –Multi-processor testing four processor run

Example: Adding a new core Steps –Develop new or convert existing code: Mediation layer routine: solve Model layer subroutines called by solver –Add to WRF Add code to source tree Incorporate into build mechanism Registry entries: data, solver options, comms Some additional splicing –Single processor testing –Analyze data-dependencies, define and implement communication for parallelism –Multi-processor testing !WRF:MEDIATION_LAYER:SOLVER SUBROUTINE solve_exp ( grid, & ! #include "exp_dummy_args.inc" ! ) CALL set_tiles ( grid,... ) #ifdef DM_PARALLEL # include "HALO_EXP_A.inc" #endif !$OMP PARALLEL DO & !$OMP PRIVATE ( ij ) DO ij = 1, grid%num_tiles CALL comp_1_into_2 ( x_1, x_2, & ids, ide, jds, jde, kds, kde, & ims, ime, jms, jme, kms, kme, & grid%i_start(ij), grid%i_end(ij), & grid%j_start(ij), grid%j_end(ij), & k_start, k_end ) END DO !$OMP PARALLEL DO & !$OMP PRIVATE ( ij ) DO ij = 1, grid%num_tiles CALL copy_2_into_1 ( x_2, x_1, & ids, ide, jds, jde, kds, kde, & ims, ime, jms, jme, kms, kme, & grid%i_start(ij), grid%i_end(ij), & grid%j_start(ij), grid%j_end(ij), & k_start, k_end ) END DO

Additional Information wrfhelp@ucar.edu www.wrf-model.org WRF Design and Implementation Document

WRF Model: Software Architecture and Code Structure John Michalakes, NCAR NCAR: W. Skamarock, J. Dudhia, D. Gill, A. Bourgeois, W. Wang, C. Deluca, R.

Similar presentations

Presentation on theme: "WRF Model: Software Architecture and Code Structure John Michalakes, NCAR NCAR: W. Skamarock, J. Dudhia, D. Gill, A. Bourgeois, W. Wang, C. Deluca, R."— Presentation transcript:

Similar presentations

About project

Feedback

Log in

Auth with social network:

WRF Model: Software Architecture and Code Structure John Michalakes, NCAR NCAR: W. Skamarock, J. Dudhia, D. Gill, A. Bourgeois, W. Wang, C. Deluca, R.

Similar presentations

Presentation on theme: "WRF Model: Software Architecture and Code Structure John Michalakes, NCAR NCAR: W. Skamarock, J. Dudhia, D. Gill, A. Bourgeois, W. Wang, C. Deluca, R."— Presentation transcript:

Similar presentations

About project

Feedback