The 2010 Workshop on the Solution of Partial Differential Equations on theSphere August 24-27, 2010 Alteration of vertical grid in NICAM towards the super-high resolution simulations Hirofumi TOMITA RIGC / JAMSTEC NICAM : DX=3.5km MTSAT-1R

Outline Introduction of NICAM Introduction of NICAM  Summary of current version of DC Vertical problem Vertical problem  Horizontal high resolution run PGF errorPGF error Land surface model couplingLand surface model coupling Future research plans using the next generation supercomputer in JAPAN Future research plans using the next generation supercomputer in JAPAN  10PFLOPS machine in Kobe Named “K-computer”Named “K-computer” Summary Summary

Summary of NICAM dynamical core Summary of NICAM dynamical core

NICAM dynamical core (1) Horizontal grid Horizontal grid  Icosahedral grid –modification by spring dynamics –Reallocation of grid to gravitational center of control volume Tomita et al. 2001,2002, JCP

NICAM dynamical core (2) Vertical grid Vertical grid  Terrain-following coordinate with Lorentz gridwith Lorentz grid Governing equation Governing equation  Non-hydrostatic equation with deep atmospherewith deep atmosphere Continuity eq. Horiz.,mom. eq. Vert.,mom. eq. Energy. eq. Tomita & Satoh 2004, Fluid.Dyn.Res Satoh et al.2008 JCP

NICAM dynamical core (3) Solver Solver  Split explicit method Fast mode ( e.g. acoustic wave, gravity wave )Fast mode ( e.g. acoustic wave, gravity wave ) –Small time step ( forward-backward ) Slow mode ( e.g. advection term )Slow mode ( e.g. advection term ) –Large time step ( by RK2 or RK3 ) In slow mode In slow mode  Horizontal explicit / vertical implicit scheme

NICAM dynamical core (3) Horizontal advection scheme Horizontal advection scheme  Miura ( 2007, MWF ) scheme with Thurburn(1996) flux limiter Simple way of flux estimationSimple way of flux estimation 2 nd order and monotonicity.2 nd order and monotonicity. Consistency With Continuity in tracer advection. Consistency With Continuity in tracer advection.  Niwa et al. (2010) Operator splitting into horizontal and vertical directionsOperator splitting into horizontal and vertical directions Use of intermediate density (Easter(1993))Use of intermediate density (Easter(1993))

Option of other grid configuration in NICAM icosahedral grid different topology Non-uniform resolution (Iga 2010, MWR submitted ) Stretched (Tomita2008)

Vertical problem of dynamical core Vertical problem of dynamical core at the high-resolution runs

Numerical problem Owing to steep mountain & terrain following coordinate Owing to steep mountain & terrain following coordinate  In TFC, one reference state should be set in all the region.  Limited area model No problemNo problem  Global model Difference from reference state is large.Difference from reference state is large. > 50K between tropics and polar region.> 50K between tropics and polar region. Pressure gradient force error!Pressure gradient force error! Remedy : setting the mid-latitude profile as the reference state.Remedy : setting the mid-latitude profile as the reference state. –Tibetan Platau, Rocky mountain, Andes mountain are in the mid-latitude. The simulation result depends on the reference state!

Sometimes, the model blow up at the high resolution simulation! Sometimes, the model blow up at the high resolution simulation!  PGF error acts violently! So-called “hydrostatic inconsistencySo-called “hydrostatic inconsistency  Reconsideration of vertical discretization from the terrain-following coordinate to height basis coordinate. –Vertical adaptive mesh for the PBL scheme. The problem becomes severe when the resolution increases!

Problem about coupling with land model(1) Energy balance equation at the interface Energy balance equation at the interface Usual way: Usual way:  explicit estimation of bulk coef in the surface flux : using the previous skin temperature.  oscillatory of skin temp. Ground flux Long wave Short wave Sensible heat latent heat Typical diurnal variation of skin temperature At dawn & suset, bulk coef. is oscillatory!

Problem about coupling with land model(2) Best Solution Best Solution  Atmosphere vertical turbulence & surface energy balance is implicitly solved at the same time.  However,,,,, Modulization?Modulization?  We want to pursue the separate modules for atmosphere turbulence and land surface model. Another solution Another solution  The bulk coefficient at the surface is implicitly solved! Tomita (2009, JHM)

Research plans toward the next- generation supercomputer

The next-generation supercomputer in Kobe The next-generation supercomputer system (~2012) The next-generation supercomputer system (~2012) System System  One nodes: 128GFLOPS128GFLOPS Memory band width : 64GB/sMemory band width : 64GB/s B/F : 0.5B/F : 0.5  80k nodes/640k core  Peak performacne : ~10PFLOPS  Total memory : ~1PB  Network 3D torus bi-direction 5GB/s X6

What can we expect on it? Next-generation supercomputer strategic program --- Field 3 ： Research of prediction of weather and climate contributing to disaster prevention Next-generation supercomputer strategic program --- Field 3 ： Research of prediction of weather and climate contributing to disaster prevention  Change of tropical cyclone in the future climate NICAM3.5km-model (7km model also)NICAM3.5km-model (7km model also) –Intensity –Frequency –Local information Integration time : 10 years orderIntegration time : 10 years order More reliable statisticsMore reliable statistics  Predictability of tropical weather NICAM3.5km, 1.7kmNICAM3.5km, 1.7km MJO, tropical cyclone, monsoonMJO, tropical cyclone, monsoon –How long can we extend the tropical prediction by GCRM? Courtesy of Prof. H.Tanaka(Tsukuba Univ.) Miura et al. 2007

Grand challenge! Global 400m mesh run Global 400m mesh run  Collaboration project with RIKEN and JAMSTEC Horizontally 400m with 100 vertical levelsHorizontally 400m with 100 vertical levels –Truly, global cloud-resolving! Use of the full computational resourceUse of the full computational resource –1PB memory –All computational nodes Integration time ： 1 week?Integration time ： 1 week?  Purpose ： Improve PBL cloud?Improve PBL cloud? Improve detail structure of cumulus?Improve detail structure of cumulus? Diurnal cycle?Diurnal cycle? Computationally,Computationally, –Well work in the case of full use? –Suggestion to the next-next computer systems plan 野田暁氏提供 野田暁氏提供

How about the sustained performance? Vector machine Vector machine  Problem size: 7km/L40  ES ：３０～４０ %  ES2: １０～１５ % Scalar machine ( NO TUNING! ) Scalar machine ( NO TUNING! )  Cray XT4 (AMD Opteron ) : 2.9% Oct ） collabolation with COLA & ECMWFcollabolation with COLA & ECMWF  T2K tsukuba(AMD Opteron) : 3.3%  IBM :BlueGene/P : 6% Good balance between memory access speed and CPU speedGood balance between memory access speed and CPU speed  Recent PC : Core i7(nehalem core) 6% Problem size : 240km/L54Problem size : 240km/L54 –(owing to memory band widith:6.4GB/sX3=19.2GB/s) –B/Fratio ： 0.45 The Next Generation Machine(BF : 0.5) The Next Generation Machine(BF : 0.5)  Default : 5%  Medium tuning : 10%  Highly tuining : 20%???

Summary NICAM: NICAM:  Global non-hydrostatic atmospheric model  Icosahedral grid Vertical discretization: Vertical discretization:  plan to change from terrain following coordinate to height based coordinate. Future plans: Future plans:  On the next-generation supercomputer system Change of tropical cyclone in the future climateChange of tropical cyclone in the future climate Predictability of tropical weatherPredictability of tropical weather Grand Challenge! ( 400m global mesh with 100 levels. )Grand Challenge! ( 400m global mesh with 100 levels. )