The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology On the efficient tracking of grid point storm issues Ilia Bermous 13 October 2011

2 Grid point storm issues  The right modelling of physical processes and finite difference parameter settings to avoid grid point storm issues  Task of efficient handling/tracking grid point storm issues  Informative diagnostic to be able to track grid point storm issues  Efficient usage of computer resources when numerical analysis becomes unstable and produces rubbish

3 Ineffiency of the UM TRAP_W subroutine  Practical aspects  Implementation aspects

4 Ineffiency of the UM TRAP_W subroutine: practical aspects  UM model uses the following capping condition  trap_option flag (0|1|2): 0 – reset, no diagnostic; 1 – reset + diagnostic 2 – diagnostic only if trap_option=2 => additional meaningless (based on a free choice of C w ) output is produced, so nothing new seen so far if trap_option=0|1 => the model uses artificial resetting (“cooking” in numerical analysis have not seen before), this kind of actions may only be useful for debugging purposes of the implemented algorithm NOTE: in tracking down the grid point storm issue we have not been interested on how condition (1) was fullfield, we are interested on a max value of the vertical wind component w. (1)

5 Ineffiency of the UM TRAP_W subroutine: practical aspects (cont #2) C w =1000 – default setting; C w =2&4 – in Gary’s last meeting report Let’s consider 2 cases for the “Jan 13” original crash job C w =1 (job completes making 900 time steps => 75hour) w_max level proc position run w_max level timestep 0.377E % East 60.1% North 0.553E C w =10 (job completes) w_max level proc position run w_max level timestep 0.479E % East 81.9% North 0.186E CwCw time step at which W started to reset Total points for upward resetting and number of time steps used for resetting Total points for downward resetting and number of times steps used for resetting 11 6,020, ,958, , none

6 Ineffiency of the UM TRAP_W subroutine: implementation aspects Let’s take a piece of a UM code referred by Gary: IF ((k>Cw_test_lev).OR.(ic_xy(i,j)>0)) THEN ic_xy(i,j) = 1 ic = ic + 1 w_adv(i,j,k) = w_test(i,j) k>Cw_test_lev condition should be moved outside and merged with the loop Do k=model_levels-1,1,-1 & k>Cw_test_lev condition => Do k=model_levels-1,max(1, Cw_test_lev+1),-1 Condition ic_xy(i,j)>0 is redundant There is a number of other redundant statements in the subroutine

7 Concepts of a new design  2 new terms have been introduced w_soft_limit – value to trigger a diagnostic to be produced if |w| > w_soft_limit w_hard_limit – value to stop execution if |w| > w_hard_limit condition is achieved on a PE  New development has been included in the sources by using a pre-processor logic with #if defined (BOM_TRAP) statements  Only 4 Fortran files have been modified./atmosphere/dynamics_diagnostics/trap_w.F90./control/top_level/atm_step.F90./control/top_level/readlsta.F90./include/common/cruntimc.h

8 Results New implementation was tested using UM7.5 R12 sources Chris R12 xbawq job (13 Jan 2011 case) has been used Here is some output with w_soft_limit=20 and w_hard_limit=100 setting ( “soft_limit” output string was used for an easy grapping) w_max level proc position N points timestep soft_limit 1.5E % East 78.4% North On each PE a single output line for each level is produced with a k_total number of points for which the soft limit condition was fulfilled Execution in this run was stopped after reaching w_hard_limit at time step 218 and 1407 soft_limit messages were produced from all PEs.../dataw1 > grep "soft_limit " *fort6* | wc -l 1407

9 Some numerical/algorithm aspects R12: Δ t=300sec, 70 levels, 400m 29 where grid point storm starts capping C w =1 for k=70, C w =10 for k=30  Semi-implicit finite-difference schema stability Δ t=600sec => |w| > 100 m/sec at T=3* Δ t as |w| may grow during integration this shows that the chosen Δ t is not small enough to satisfy the stability requirement of the finite-difference operators used in the modelling ran case with Δ t=60sec using “tip” from Stuart’s presentation (it takes ~9.5hours for 75h integration on 192 cores), note that for this value of Δ t a different range of C w values should be used for trap_w condition (1) w_max level proc position run w_max level timestep 0.886E % East 87.4% North 0.218E

10 Conclusions  A new developed version of TRAP_W subroutine produces useful information on tracking relatively large values of w vertical wind component  A moderate size of the output data is produced  Numerical analysis is stopped when “unreal” values of vertical wind component w are produced  One of the possible ways to avoid a grid point storm problem is to use a relatively small time step Δ t according to the stability requirement of the finite-difference operators used in the modelling  Results comparison between C w =10, Δ t=300sec “cooked” case and Δ t =60sec shows some large instability near the domain boundary

11 Conclusions (cont #2)

12 Conclusions (cont #3)