1. Development of a high resolution operational model at JMA Kengo Matsubayashi Numerical Prediction Division, Japan Meteorological Agency with Kohei Kawano, Tabito Hara, Kohei Aranami, Hiroshi Kusabiraki, Haruka Kurahashi and Junichi Ishida 1 August 19, 2014 The World Weather Open Science Conference 2014 Montreal, Canada

2. GlobalMesoLocal Objectives Short- and Medium-range forecast Disaster reduction Short-range forecast Aviation forecast Disaster prevention NWP model Global Spectral Model(GSM) Meso-Scale Model(MSM) Local Forecast Model (LFM) Horizontal resolution TL959 (0.1875 deg) 5 km (817x611) 2 km (1581x1301) Vertical levels / Top 100 0.01 hPa 50 21.8 km 60 20.2 km Forecast Hours (Initial time) 84 hours (00, 06, 18 UTC) 264 hours (12 UTC) 39 hours (every 3 hours) 9 hours (every hour) Initial Condition Global Analysis (4D-Var) Meso-scale Analysis (4D-Var) Local Analysis (3D-Var) Forecast Domain NWP systems at JMA (deterministic) Meso Local Global 2 Current operational NWP systems at JMA A high resolution convection-permitting regional NWP system (Local NWP system) has been operated since August 2012. It’s purpose is providing information on aviation weather and disaster prevention.

3. 3 Basic design of the Local NWP system The Local NWP system provides 9-hour period forecasts every hour. In the system design, frequent updates of forecasts (24 times a day!!) assimilating the latest observation are highly emphasized. The Local NWP system consists of two subsystems – NWP model: The Local Forecast Model (LFM) has a 2-km horizontal grid spacing and 60 vertical layers. – high resolution to permit explicit convection – Data assimilation system: The Local Analysis (LA) employs an analysis system based on the three dimensional variational data assimilation (3D-Var) at a 5-km resolution. Forecast:1581x1301 (2km grids) Analysis:633x522 (5km grids) High mountains and valleys are more realistically resolved.

4. 4 1581x1301 with 2km grids. operational domain Domain of the Local NWP system 40N 30N 40N 30N A region of the same size (for reference) Montreal 2600km 3160km

5. 55 Advantage of LFM The LFM has considerable potential to represent the shape of precipitation and its peak values more accurately. 1-hour accumulated precipitation amounts until 1700UTC on July 11 2012 mm/h max 60mm/h 500km 600km

6. We are operating LFM, but… Current dynamical core (JMA-NHM) is too old – Using matured but old technics (developed mainly in 1980s) Insufficient accuracy such as no mass conservation, leap-frog scheme Forecasts are quite sensitive to numerical diffusion – We do not know what value is the best for coefficients of numerical diffusion because we have no established way to find the best parameter (just based on empirical) – Not necessarily compatible with the current computer trend – Almost no room left for further developing Fairly convoluted codes Codes for physics cannot be separated 6

7. DEVELOPMENT OF A NEW DYNAMICAL CORE 7

8. Development of a new dynamical core Motivation A new nonhydrostatic dynamical core named “ASUCA” is under development aiming at 1.Higher accuracy, stability e.g. exact conservation of mass 2.Exclusion of artificial parameters (e.g. numerical diffusion coefficient) as much as possible 3.Higher efficiency on massive parallel scalar multi-core architecture 4.Keeping the codes well structured possible to facilitate the developments and long term maintenance 8

9. 9 The specification comparison of the dynamical core between ASUCA and JMA-NHM ASUCAJMA-NHM Governing equationsFlux form Fully compressible equations Quasi flux form Fully compressible equations Prognostic variablesρu, ρv, ρw, ρθ, ρρu, ρv, ρw, θ, p Spatial discretizationFinite volume methodFinite difference Method Time integrationRunge-Kutta 3 rd (long and short) Leapflog with time filter (long) Forward backward (short) Treatment of soundConservative Split explicitSplit explicit AdvectionCombining 3 rd and 1 st order upwind with flux limiter by Koren(1993) 4 th (hor.) and 2 nd (ver.) order with advection correction Numerical diffusionNone4 th order linear and nonlinear diffusion Treatment of rain-dropTime-splitBox-Lagrangian CoordinateGeneralized coordinate or Conformal mapping + Hybrid-Z Conformal mapping (hor.) Hybrid – Z (ver.) GridArakawa-C (hor.) Lorentz (ver.) Arakawa-C (hor.) Lorentz (ver.) C : Mass conservationA : Higher accuracyS : Higher stability C C C A M : Monotonicity S AMC A C

10. Mass conservation Lack of mass conservation might lead to generate errors in the predicted pressure fields. 10 The total mass is obviously smaller than the reference, and the pressure field is different from the reference. JMA-NHM MSL pressure forecast Difference from a reference pressure field ASUCA Sum of MSL pressure difference from given reference over the domain is zero. The total mass tendency over the domain should be equal to that given by LBC if mass conservation is exactly satisfied. If mass does not conserve, accurate pressure field and resulting synoptic scale phenomenon cannot be reproduced.

11. 4 th order 4 th order + adv. correction + 4 th linear diffusion 4 th order + adv. correction + 4 th linear diffusion + nonlinear diffusion Advection scheme 11 JMA-NHM – Based on 4 th order scheme – Advection correction – 4 th linear diffusion – Nonlinear diffusion adopted in JMA-NHM Cycle advection test in a constant velocity field. Green line is the result of after two cycles.

12. 12 Advection scheme Combining – 3 rd order upwind in smooth regions Higher accuracy, non-monotonicity （ Godunov’s theorem ） – 1 st order upwind in sharp gradient regions Monotonicity, lower accuracy with using flux limiter function proposed by Koren(1993) Koren(1993) Monotonicity is preserved, no overshoot and no undershoot appeared. Cycle advection test in a constant velocity field. Green line is the result of after two cycles.

13. Warm bubble test 13 potential temperature 10km Initial bubble size: 4000m PT perturbation: +2K

14. Warm bubble test In a courser resolution, the position and the shape of the bubble are not well predicted for the poor accuracy of the adv. scheme. And they are quite sensitive to the strength of the numerical diffusion. 14 dx = 125m JMA-NHM w/o diffusion ASUCA JMA-NHM dz = 125m in all cases. dx = 1000m JMA-NHM w/ diffusion ASUCA

15. Current status of ASUCA Main part of the developments of dynamical core has been completed. Physical processes equivalent to or more enhanced than those of current LFM have been implemented. Basic performance is similar to or better than that of JMA-NHM. ASUCA will be operational as LFM later in 2014 along with a 3DVAR system based on ASUCA. 15 ASUCA JMA-NHMObs.

16. Thank you for your attention. To archive higher efficiency – kij-ordering – reduce the number of MPI communication Implementation of physical processes – Physics library Pre-operational run – Large acoustic wave caused by the imbalance of 3DVar Convection in high resolution model – Convective initiation is sensitive to the high wave number modes. – The high wave number modes have a sensitivity to the diffusivity in advection schemes. – Even in 2km horizontal resolution, the initiation of convection is not necessarily resolved. – We employ parameterization for convective initiation. 16 Other topics

17. Governing equations of ASUCA 17 The RHS terms include not only physical processes, Coriolis terms and curvature terms and precipitation. J is a Jacobian of the coordinate transformation u, v, w : velocity components in the physical space ˆu, ˆv, ˆ w : those in the computational space ρ : total mass density q  : ratio of the density of a water substance  to the total mass density ( x 1, x 2, x 3 ) : physical space. (ˆx 1, ˆx 2, ˆx 3 ) : computational space. Density is used as the prognostic variable for mass conservation

18. Finite Volume Method The computational domain is divided into cells. Fluxes at the cell boundaries are evaluated. Continuous equation with differential form (left) and integral form (right) The surface integral of the equation in the integral form is approximated by sum of the fluxes at the boundaries. Conservative Suitable for unstructured meshes ・

19. Mass conservation Variation of the total mass in the computational domain (1) should be sum of inflow and outflow of the mass at boundaries Sum of inflow and outflow of the mass at boundaries = flux of mass at lateral boundaries (2) - loss of mass by precipitation (3) + water vapor flux at the surface (4) 19 ・・・ total mass diff, total flux = -227802864231.908875 -227802864231.934052 total error = 0.251770019531250000E-001 (total lateral flux) = -224912457213.078125 (total prc flux) = -16499942061.0891171 (total surface flux) = 13609535042.2331963 Forecast period (1 ) is computed directly from the density

20. 20 Software design of ASUCA To achieve higher efficiency on massive parallel scalar multi- core architecture – kij - ordering Three-dimensional arrays in space are stored sequentially in the order of z (k), x (i) and y (j). – Aiming at low memory usage to improve cache efficiency » It has an affinity to “column based physical processes” – Advantageous to parallelize at outermost loop. – reduce the number of MPI communication Subroutine for data stock before MPI comm. & subroutine for MPI comm. of stocked data are separately prepared. real(8) :: u(nz, nx, ny) ASUCAJMA-NHM 72600138652 Asuumption of 1hour forecast of LFM(dx=dy=2km) JMA-NHM:dt=8, ASUCA:dt=16 Number of calling MPI comm.

21. Implementation of physical processes Physical processes are expected to provide tendencies. Column based physical processes include only the vertical one- dimensional loop. The horizontal loops are parallelized using OpenMP. real:: pt(nz,nx,ny) real:: tend_pt_pbl(nz) !$OMP PARALLEL DO SCHEDULE(DYNAMIC) & !$OMP& PRIVATE(pt_1d) do j = 1, ny do i = 1, nx pt_1d(1:nz) = pt(1:nx,i,j) call pp_rad_pbl_surface_run(i,j,tend_pt_1d,pt_1d…) end do !$OMP END PARALLEL DO subroutine pp_rad_pbl_surface_run(i,j,tend_pt,pt_1d,…) real:: tend_pt_pbl_1d(nz), pt_1d(nz) call pbl_run(tend_pt_pbl_1d,pt_1d, …) do k = 1, nz tend_pt(k,i,j) = tend_pt(k,i,j) + tend_pt_pbl_1d(k) end do kij - ordering Better cache hit ratio with vertical one dimensional calculation kij - ordering Longer loop length for outermost Better load balance among PEs

22. Schematic structure of the “Physics Library” The “Physics Library” is designed to be plugged in easily to any models The physical processes implemented in the “Physics Library” are vertically one-dimensionalized. ASUCA passes inputs (Vars(nz)) to library, then receives tendencies (tendency(nz)) from library. 22 SURF ・・・・・・・・ ・・・・ SURF_INI GRID PARM SURF_RUN RAD RAD_INI GRID PARM RAD_RUN PBL PBL_INI GRID PARM PBL_RUN CONSTMONITOR Vars(nz) COMM Model B Tendency(nz) Model C Model A