1. Zavisa Janjic 1 Nonhydrostatic Multiscale Model on the B grid (NMMB): Global Runs Zavisa Janjic Tom Black Ratko Vasic Dusan Jovic + MMB …

2. Zavisa Janjic 2 Nonhydrostatic Multiscale Model on the B grid (NMMB) Intended for wide range of spatial and temporal scales (from meso to global, and from weather to climate) Further evolution of WRF Nonhydrostatic Mesoscale Model (NMM) Built on NWP and regional climate experience (Janjic et al., 2001, MWR; Janjic, 2003, MAP; Janjic & Gall, 2012, NCAR) Pressure based vertical coordinate, nondivergent flow remains on coordinate surfaces

3. Zavisa Janjic 3 Nonhydrostatic Dynamics Inviscid adiabatic equations No over-specification of w! Difference between hydrostatic pressures at surface and top Hydrostatic pressure Nonhydrostatic pressure Gas law Hypsometric (not “hydrostatic”) Eq. Hydrostatic continuity Eq. Continued …

4. Zavisa Janjic 4 Third Eq. of motion Vertical velocity definition, nonhydrostatic continuity Eq. Momentum Eq. Thermodynamic Eq.

5. Zavisa Janjic 5 Conservation of important properties of the continuous system aka “mimetic” approach in Comp. Math. (Arakawa 1966, 1972, …; Jacobson 2001; Janjic 1977, …; Sadourny, 1968, … ; Tripoli, 1992 …) Nonlinear energy cascade controlled through energy and enstrophy conservation (finite-volume) A number of properties of differential operators preserved Quadratic conservative finite differencing A number of first order (including momentum) and quadratic quantities conserved Omega-alpha term, transformations between KE and PE Errors associated with representation of orography minimized Mass conserving positive definite monotone Eulerian tracer advection Discretization Principles

6. Zavisa Janjic 6 Coordinates and Grids Global lat-lon Regional rotated lat-lon, more uniform grid size Arakawa B grid h h h v v h h h Pressure-sigma hybrid (Simmons & Burridge 1981, Eckermann 2008) Flow remains on hydrostatic pressure coordinate surfaces in case of nondivergent flow Lorenz vertical grid

7. Zavisa Janjic 7 No splitting (for computational efficiency) Additive splitting inadequate for large Ro Splitting with multi-step iterative schemes too expensive “Stabilized” Adams-Bashforth for horizontal advection of u, v, T and Coriolis force Crank-Nicholson for vertical advection of u, v, T (implicit) because of CFL problems with thin layers Forward-Backward (Ames, 1968; Gadd, 1978; Janjic and Wiin-Nielsen, 1977, JAS) fast waves Implicit for vertically propagating sound waves (Janjic et al., 2001, MWR; Janjic, 2003, MAP) Time Stepping

8. Zavisa Janjic 8 Lateral and Polar Boundaries, Polar Filter Regional domain lateral boundaries Narrow zone with upstream advection, no computational outflow BC for advection Blending zone Conservative global polar boundary conditions Polar filter “Decelerator,” tendencies of T, u, v, Eulerian tracers, divergence, dw/dt Physics not filtered

9. Zavisa Janjic 9 Physics Upgraded NCEP WRF NMM “Unified” physics RRTM, GFDL radiation NOAH, LISS land surface model Mellor-Yamada-Janjic turbulence Elevated subgrid surface drag Cloud topped marine BL GFS Gravity Wave Drag Ferrier, Zhao microphysics Betts-Miller-Janjic convection High resolution enabled Full NEMS GFS physics

10. Zavisa Janjic 10 27 km Global NMMB 12 km NAM NMMB 4 km NAM-nest NMMB 9 km Igor NMMB 9 km Julia NMMB Hypothetical NMMB Simultaneous Run Global [with Igor & Julia] and NAM [with CONUS nest] Courtesy of DiMego et al.

11. Zavisa Janjic 11 2D very high resolution tests Warm bubble, 100 m resolution Cold bubble, 100 m resolution Full compressible NMM Analytical (Boussinesque) ARPS (Boussinesque) Nonlinear mountain wave 400 m resolution Normalized vertical momentum flux, 400 m resolution

12. Zavisa Janjic 12 Mountain waves, 8 km resolution Convection, 1 km resolution

13. Zavisa Janjic 13 Atmospheric Spectrum Numerical models generally generate excessive small scale noise False nonlinear energy cascade (Phillips, 1954; Arakawa, 1966 … ; Sadourny 1975; …) Other computational errors Historically, problem controlled by: Removing spurious small scale energy by numerical filtering, dissipation Preventing excessive noise generation by enstrophy and energy conservation (Arakawa, 1966 …)

14. Zavisa Janjic 14 Atmospheric Spectrum Classical paper by Sadourny, 1975, JAS: One does not even need an atmospheric model for that!

15. Zavisa Janjic 15 Atmospheric Spectrum Instead (Sadourny, 1975, JAS): Philosophy built into the design of the compact nonlinear advection schemes for semi-staggered grids (Janjic, 1984, MWR)

16. Zavisa Janjic 16 Formal accuracy alone does not solve the problem Different nonlinear noise levels (green scheme) with identical formal accuracy and truncation error (Janjic et al. 2011, MWR) Green scheme, still energy & (alternative) enstrophy conserving Second order schemes Fourth order schemes 116 days

17. Zavisa Janjic 17 Atmospheric Spectrum Nastrom-Gage (1985, JAS) 1D spectra in upper troposphere and lower stratosphere from commercial aircraft data No spectral gap Transition at few hundred kilometers from –3 to –5/3 slope in the 10 2 -10 3 km (mesoscale) range Inertial range, 0.01-0.3 m s -1 in the –5/3 range, not to be confused with severe mesoscale phenomena! 10 4 10 8 10 3 10 5 10 2 10 1 10 3 -3 -5/3

18. Zavisa Janjic 18 Atmospheric Spectrum What would the real atmosphere do in a short range run in a limited area domain starting from analysis with truncated spectrum? Short time for statistical equilibrium Large scales cut-off by domain size NMMB well qualified for investigating numerical spectra by design

19. Zavisa Janjic 19 Atmospheric Spectrum Are model simulated spectra forced by sigma coordinate errors? Are model simulated spectra just projections of topography spectrum? -3 -5/3 630 km 63 km Spectrum of topography squared over North America

20. Zavisa Janjic 20 No PhysicsWith Physics -5/3 -3 ● Spectrum in agreement with observed spun-up given physical (or spurious, e.g. sigma) sources on small scales. ●Small scale energy in short regional runs with controlled nonlinear cascade and suppressed noise sources, where from? Flat bottom (Atlantic), NMMB, 15km, 32 levels, 48h run (Loops)

21. Zavisa Janjic 21 No Physics With Physics Where is the small scale energy in the observed spectrum coming from? Atlantic case, NMM-B, 15 km, 32 Levels, 36-48 hour average -5/3 -3 No PhysicsWith Physics

22. Zavisa Janjic 22 Loop, decaying 3D turbulence, Fort Sill storm, 05/20/77. NMM-B, Ferrier microphysics, 1km resolution, 32 levels, 112km by 112km by 16.4km, double periodic, Smagorinsky constant 0.32. Note the hump with active convection Spectrum of w 2 at 700 hPa.

23. Zavisa Janjic 23 -5/3 Hours 3-4 average decaying 3D turbulence, Fort Sill storm, 05/20/77. NMM-B, Ferrier microphysics, 1km resolution, 32 levels, 112km by 112km by 16.4km, double periodic. Smagorinsky constant 0.32. Spectrum of w 2 at 700 hPa.

24. Zavisa Janjic 24 Parallel runs for testing and tuning Initialized from spectral GFS analyses Compatibility issues between grid-point and spectral data (Gibbs phenomenon) Verified against GFS analyses and climatology 500 hPa Height Anomaly Correlation Coefficients Although starting from “same” initial conditions, skill of NMMB and GFS forecasts often disparate Major GFS update on July 28, 2010 One year of parallel global forecasts before and after July 28, 2010 Global Scales

25. Zavisa Janjic 25 Global Resolution “Resolution comparable to that of the GFS” Txxx is the GFS triangular truncation Δλ≈360˚/(2 Txxx) Δφ≈Δλ cos(45˚) Δx≈Δy at 45˚

26. Zavisa Janjic 26 Courtesy Dusan Jovic

27. Zavisa Janjic 27 Global NMMB 769 x 541 x 64 pts. vs. GFS T384 x 64 1 year 500 hPa Height Anomaly Correlation Coefficient vs. forecast time NMMB initialized and verified using GFS analyses and climatology NMMB comparable or lower resolution

28. Zavisa Janjic 28 Global NMMB 769 x 541 x 64 pts. vs. GFS T384 x 64 1 year 500 hPa Height Anomaly Correlation Coefficient vs. forecast time NMMB initialized and verified using GFS analyses and climatology NMMB comparable or lower resolution

29. Zavisa Janjic 29 Average of 32 GFS T574 dropout cases from 2011 (Alpert) ran with NMMB with 769x541x64 points from GFS and ECMWF analyses Above dropout threshold! From July 28 2010 GFS has 3.8 times more points & updated physics 500 hPa ACC Verification using GFS analyses and ERA Interim climatology Courtesy of Dr. Vladimir Djurdjevic

30. Zavisa Janjic 30 NMMB with GFS analysis closer to GFS (14) NMMB with GFS analysis closer to NMMB with ECMWF analysis (14) Ties (2) Anomalous (2) Courtesy of Dr. Vladimir Djurdjevic

31. Zavisa Janjic 31 Courtesy of Yuejian Zhu

32. Zavisa Janjic 32 Courtesy of Yuejian Zhu

33. Zavisa Janjic 33 Aerosol Climatology

34. Zavisa Janjic 34 Global NMMB 769 x 541 x 64 pts. vs. GFS T574 x 64 1 year 500 hPa Height Anomaly Correlation Coefficient vs. forecast time NMMB initialized and verified using GFS analyses and climatology From July 28 2010 GFS has 3.8 times more points & updated

35. Zavisa Janjic 35 Global NMMB 769 x 541 x 64 pts. vs. GFS T574 x 64 1 year 500 hPa Height Anomaly Correlation Coefficient vs. forecast time NMMB initialized and verified using GFS analyses and climatology GFS has 3.8 times more points Global NMMB 769 x 541 x 64 pts. vs. GFS T574 x 64 1 year 500 hPa Height Anomaly Correlation Coefficient vs. forecast time NMMB initialized and verified using GFS analyses and climatology From July 28 2010 GFS has 3.8 times more points & updated physics

36. Zavisa Janjic 36 New testing strategy Insufficient computational resources for sustainable parallel NMMB tests with enhanced resolution Long testing periods required in order to obtain robust representative average scores (a change bringing a noticeable improvement in one season may be detrimental in the other) Increased Resolution

37. Zavisa Janjic 37 A sample of 105 cases over one year at three and a half day intervals created (~ 1/7 of the population) Both 00Z and 12Z initial data equally represented Since the cases are chosen randomly, it is expected that test results over this set of cases would be reasonably close to test results over a year A number of initial data conversion algorithms, resolution settings and physical parameterization sensitivity studies carried out Increased Resolution

38. Zavisa Janjic 38 Plan Increase resolution to become comparable to that of GFS Retune the physics 1.Start from radiation (special kind of parameterization) 2.Convection 3.… Increased Resolution

39. Zavisa Janjic 39 NMMB - RSWIN NMMB - RLWIN ECMWF - RSWIN ECMWF - RLWIN NMMB - (RSWIN-RSWOUT) ECMWF - (RSWIN-RSWOUT) NMMB - RLWTOA ECMWF - RLWTOA Zonal averages of radiation fluxes forecast start: 20111220 48h forecast NMMB vs ECMWF Difference in RLWIN on north hemisphere 50N-90N of about 50W/m^2 Courtesy of Dr. Vladimir Djurdjevic

40. Zavisa Janjic 40 GFS NMMB ECMWF Mean downward long-wave radiation flux 6-12h forecast hour. RRTM w. hydrometeor clouds By far largest fluxes in NMMB! Courtesy of Dr. Vladimir Djurdjevic

41. Zavisa Janjic 41 GFS NMMB RH clouds, no iceHydrometeor clouds, no ice Courtesy of Dr. Vladimir Djurdjevic

42. Zavisa Janjic 42 ECMWF; Vertical section Downward LW flux NMMB; Vertical section NMMB (F_ICEC=0); Vertical section Downward LW flux Problem with radiation- ice interaction, fallback to no ice Zhao microphysics

43. Zavisa Janjic 43 Global NMMB 1149 x 811 x 64 pts. vs. GFS T574 x 64 105 randomly chosen cases from 1 year 500 hPa Height Anomaly Correlation Coefficient vs. forecast time NMMB initialized and verified using GFS analyses and climatology New Linux cluster test Global NMMB 1149 x 811 x 64 pts. vs. GFS T574 x 64 105 randomly chosen 00Z and 12Z cases over 1 year 500 hPa Height Anomaly Correlation Coefficient vs. forecast time NMMB initialized and verified using GFS analyses and climatology

44. Zavisa Janjic 44 New Linux cluster test Global NMMB 1149 x 811 x 64 pts. vs. GFS T574 x 64 105 randomly chosen 00Z and 12Z cases over 1 year 500 hPa Height Anomaly Correlation Coefficient difference vs. forecast time NMMB initialized and verified using GFS analyses and climatology

45. Zavisa Janjic 45 New Linux cluster test Global NMMB 1149 x 811 x 64 pts. vs. GFS T574 x 64 105 randomly chosen 00Z and 12Z cases over 1 year 500 hPa RMSE difference vs. forecast time NMMB initialized and verified using GFS analyses and climatology

46. Zavisa Janjic 46 Tropics What is the truth? Insufficient data, more weight on model in DAS Weak connection between mass and wind Direct circulations Gravity waves Extrapolation underground Different gravity-inertia wave frequency errors and computational dispersion in different models (semi-implicit vs. explicit) Different geostrophic adjustment Each model should have its own DAS for fair comparison

47. Zavisa Janjic 47 Janjic, Z., A. Wiin-Nielsen, 1977: On Geostrophic Adjustment and Numerical Procedures in a Rotating Fluid. J. Atmos. Sci., 34, 297–310. doi: http://dx.doi.org/10.1175/1520- 0469(1977)034 2.0.C O;2http://dx.doi.org/10.1175/1520- 0469(1977)034 2.0.C O;2

48. Zavisa Janjic 48 Semi impliciit, 5 leapfrpg time steps Correct

49. Zavisa Janjic 49 Global, nonhydrostatic, full physics Resolution 1149 x 811 x 64, ~ 22 km 500 processors, 7.9 wall clock min/day 1000 processors, 4.7 wall clock min/day 2000 processors, 3.3 wall clock min/day 4000 processors, 2.8 wall clock min/day, (1 year of simulation in < 17 hours, climate studies?) Scaling on Zeus

50. Zavisa Janjic 50 Global, nonhydrostatic, full physics Resolution 2305 x 1623 x 64, ~ 11 km (8 x more work than 1149 x 811 x 64, ~ 22 km) 3600 processors, 7.6 wall clock min/day Perfect scaling! Technology for ~ 10 km global resolution forecasts already exists! Scaling on Zeus

51. Zavisa Janjic 51 Summary and Conclusions New testing paradigm for high resolution global runs Global NMMB shows good results in medium range forecasting ~10 km resolution global forecasts? Technology exists Resolutions less than <10 km km possible soon