Presentation is loading. Please wait.

Presentation is loading. Please wait.

Inter-Processor communication patterns in weather forecasting models Tomas Wilhelmsson Swedish Meteorological and Hydrological Institute Sixth Annual Workshop.

Published by Modified over 9 years ago

Similar presentations

Presentation on theme: "Inter-Processor communication patterns in weather forecasting models Tomas Wilhelmsson Swedish Meteorological and Hydrological Institute Sixth Annual Workshop."— Presentation transcript:

1 Inter-Processor communication patterns in weather forecasting models Tomas Wilhelmsson Swedish Meteorological and Hydrological Institute Sixth Annual Workshop on Linux Clusters for Super Computing 2005-11-18

2 2015-06-17 Signatur Numerical Weather Prediction Analysis Obtain best estimate of current weather situation from 1.Background, (the last forecast 6 to 12 hours ago) 2.Observations (ground, aircraft, ship, radiosondes, satellites) Variational assimilation in 3D or 4D Most computationally expensive part Forecast Step forward in time (48 hours, 10 days, …) Ensemble forecast Estimate uncertanty by runing many (50-100) forcasts from perturbed analysis

3 2015-06-17 Signatur A 10-day ensemble forecast for Linköping Blue line is unperturbed high resolution forecast Dotted red is unperturbed reduced resolution forecast Bars indicate center 50% of 100 perturbed forecasts at reduced resolution

4 2015-06-17 Signatur HIRLAM at SMHI 48-hour 22 km resolution forecast on a limited domain –Boundaries from global IFS forecast at 40km Also 11 km HIRLAM forecast on a smaller domain 40 minutes elapsed on 32 processor of a Linux cluster – Dual Intel Xeon 3.2 GHz – Infiniband – More info in Torgny Faxén’s talk tomorrow!

5 2015-06-17 Signatur Codes: IFS, ALADIN, HIRLAM, ALADIN IFS – Integrated Forecast System (ECMWF) -Global, Spectral, 2D decomposition, 4D-VAR ALADIN - Aire Limitée Adaptation Dynamique développement InterNaltional –Shares codebase with ARPEGE, the Météo-France version of IFS –Limited area, Spectral, 2D decomposition, 3D-VAR –Future: AROME at 2-3 km scale HIRLAM – High Resolution Limited Area Model –Limited area, Finite difference, 2D decomposition HIRVDA – HIRlam Variational Data Assimilation –Limited area, Spectral, 1D decomposition, 3D-VAR, (and soon 4D-VAR )

6 2015-06-17 Signatur Numerics Longer time steps made possible by Semi-implicit time integration –Advance fast linear modes implitly and slower non-linear modes explicitly –A Helmholz equation has to be solved In HIRLAM by direct FFT + tri-diagonal method Spectral models do it easily in Fourier space –Implications for domain decomposition! Semi-Lagrangian advection –Wide halo zones

7 2015-06-17 Signatur How should we partition the grid? Example: HIRLAM C22 grid (nx = 306, ny = 306, nlev = 40) –Many complex interactions in vertical (the ”physics”). Decomposing the vertical would mean frequent interprocessor communication. –Helmholtz solver FFT part prefers nondecomposed longitudes Tridiagonal solver partpreferes nondecomposed latitudes Similar for spectal models (IFS, ALADIN & HIRVDA) –Transforming from physical space to spectral space means FFTs in both longitiudes and latitudes And physics in vertical

8 2015-06-17 Signatur Grid partitioning in HIRLAM (Jan Boerhout, NEC) 2 1 0 5 4 3 8 7 6 11 10 9 012 345 678 9 11 258 14710 0369 levels longitude latitude TWOD distribution FFT distribution TRI distribution Transpose

9 2015-06-17 Signatur Transforms and transposes in IFS / ALADIN

10 2015-06-17 Signatur Spectral methods in limited area models HIRVDA / ALADIN HIRVDA C22 domain –nx & ny = 306 Extension zone – nxl & nyl = 360 Spectral space – kmax & lmax = 120

11 2015-06-17 Signatur Transposes in HIRVDA (spectral HIRLAM) 1D decomposition

12 2015-06-17 Signatur HIRVDA timings

13 2015-06-17 Signatur Transposes with 2D partitioning

14 2015-06-17 Signatur Load balancing in spectral space Isotropic representation in spectral space requrires an ellipic truncation By accepting an unbalanced y-direction FFT, spectral space can be load balanced

15 2015-06-17 Signatur Number of messages 1D decomposition n=4 => 24 n=64 => 8064 2D decomposition n=4 => 24 n=64 => 2688

16 2015-06-17 Signatur Timings on old cluster (Scali)

17 2015-06-17 Signatur Timings on new cluster (Infiniband)

18 2015-06-17 Signatur Zoom in…

19 2015-06-17 Signatur Minimum time on old cluster

20 2015-06-17 Signatur FFT / Transpose timeline 2D decomposition

21 2015-06-17 Signatur FFT / Transpose timeline 1D decomposition

22 2015-06-17 Signatur Semi-Lagrangian Advection Full cubic interpolation in 3D is 32 points (4x4x4)

23 2015-06-17 Signatur Example: The HIRLAM C22 area (306x306 grid at 22 km resolution) Max wind speed in jet stream 120 m/s Time step 600 s => Distance 72 km = 3.3 grid points) Add stencil width (2) => nhalo= 6 With 64 processors partitioned in 8x8: – 38x38 core points per processor – 50x50 including halo Halo area is 73% of core! But full halo is not needed everywhere!

24 2015-06-17 Signatur IFS & ALADIN – Semi-Lagrangian advection Requesting halo points ’on-demand’

25 2015-06-17 Signatur ’On-demand’ algorithm 1.Exchange full halo for wind components (u,v & w) 2.Calculate departure points 3.Determine halo-points needed for interpolation 4.Send list of halo points to surrounding PE’s 5.Surrounding PE’s send points requested

26 2015-06-17 Signatur Effects on various optimizations on IFS performance Moving from Fujitsu VPP (vector machine) to IBM SP (cluster). Figure from Debora Salmond (ECMWF).

27 2015-06-17 Signatur Conclusion Meteorology and climate sciences provide plenty of fun problems for somebody interested in computational methods and parallelization. Also: –Load balancing observations in data assimilation –Overlapping I/O with computation HIRLAM & HIRVDA will get ’on-demand’ slswap soon

Download ppt "Inter-Processor communication patterns in weather forecasting models Tomas Wilhelmsson Swedish Meteorological and Hydrological Institute Sixth Annual Workshop."

Similar presentations

Assimilation of radar data - research plan

Assimilation of radar data - research plan
An introduction to the memory-distributed aspect of the code Arpege/Ifs/Aladin R. El Khatib Météo-France - CNRM/GMAP September, 2002.

An introduction to the memory-distributed aspect of the code Arpege/Ifs/Aladin R. El Khatib Météo-France - CNRM/GMAP September, 2002.
Introduction to data assimilation in meteorology Pierre Brousseau, Ludovic Auger ATMO 08,Alghero, 15-18 september 2008.

Introduction to data assimilation in meteorology Pierre Brousseau, Ludovic Auger ATMO 08,Alghero, september 2008.
HIRLAM Use of the Hirlam NWP Model at Met Éireann (Irish Meteorological Service) (James Hamilton -- Met Éireann)

HIRLAM Use of the Hirlam NWP Model at Met Éireann (Irish Meteorological Service) (James Hamilton -- Met Éireann)
Assimilation Algorithms: Tangent Linear and Adjoint models Yannick Trémolet ECMWF Data Assimilation Training Course March 2006.

Assimilation Algorithms: Tangent Linear and Adjoint models Yannick Trémolet ECMWF Data Assimilation Training Course March 2006.
Status and performance of HIRLAM 4D-Var Nils Gustafsson.

Status and performance of HIRLAM 4D-Var Nils Gustafsson.
Computational Challenges in Air Pollution Modelling Z. Zlatev National Environmental Research Institute 1. Why air pollution modelling? 2. Major physical.

Computational Challenges in Air Pollution Modelling Z. Zlatev National Environmental Research Institute 1. Why air pollution modelling? 2. Major physical.
WWOSC 2014 Montreal, Canada 2014 - 08 Running operational Canadian NWP models on next-generation supercomputers Michel Desgagné, Abdessamad Qaddouri, Janusz.

WWOSC 2014 Montreal, Canada Running operational Canadian NWP models on next-generation supercomputers Michel Desgagné, Abdessamad Qaddouri, Janusz.
Data assimilation schemes in numerical weather forecasting and their link with ensemble forecasting Gérald Desroziers Météo-France, Toulouse, France.

Data assimilation schemes in numerical weather forecasting and their link with ensemble forecasting Gérald Desroziers Météo-France, Toulouse, France.
Numerical Weather Prediction on Linux-clusters – Operational and research aspects Nils Gustafsson SMHI.

Numerical Weather Prediction on Linux-clusters – Operational and research aspects Nils Gustafsson SMHI.
1/20 Accelerating minimizations in ensemble variational assimilation G. Desroziers, L. Berre Météo-France/CNRS (CNRM/GAME)

1/20 Accelerating minimizations in ensemble variational assimilation G. Desroziers, L. Berre Météo-France/CNRS (CNRM/GAME)
Acknowledgments: Thanks to Professor Nicholas Brummell from UC Santa Cruz for his help on FFTs after class, and also thanks to Professor James Demmel from.

Acknowledgments: Thanks to Professor Nicholas Brummell from UC Santa Cruz for his help on FFTs after class, and also thanks to Professor James Demmel from.
High-resolution numerical modeling and predictability of atmospheric flows M. Ehrendorfer, A. Gohm and G. J. Mayr Institut für Meteorologie und Geophysik.

High-resolution numerical modeling and predictability of atmospheric flows M. Ehrendorfer, A. Gohm and G. J. Mayr Institut für Meteorologie und Geophysik.
Current Status of the Development of the Local Ensemble Transform Kalman Filter at UMD Istvan Szunyogh representing the UMD “Chaos-Weather” Group Ensemble.

Current Status of the Development of the Local Ensemble Transform Kalman Filter at UMD Istvan Szunyogh representing the UMD “Chaos-Weather” Group Ensemble.
WIND ENSEMBLE FORECASTING USING AN ADAPTIVE MASS-CONSISTENT MODEL A. Oliver, E. Rodríguez, G. Montero.

WIND ENSEMBLE FORECASTING USING AN ADAPTIVE MASS-CONSISTENT MODEL A. Oliver, E. Rodríguez, G. Montero.
DMI Update Leif Laursen ( ) Jan Boerhout ( ) CAS2K3, September 7-11, 2003 Annecy, France.

DMI Update Leif Laursen ( ) Jan Boerhout ( ) CAS2K3, September 7-11, 2003 Annecy, France.
Basic Concepts of Numerical Weather Prediction 6 September 2012.

Basic Concepts of Numerical Weather Prediction 6 September 2012.
1 The Spectral Method. 2 Definition where (e m,e n )=δ m,n e n = basis of a Hilbert space (.,.): scalar product in this space In L 2 space where f * :

1 The Spectral Method. 2 Definition where (e m,e n )=δ m,n e n = basis of a Hilbert space (.,.): scalar product in this space In L 2 space where f * :
Operational computing environment at EARS Jure Jerman Meteorological Office Environmental Agency of Slovenia (EARS)

Operational computing environment at EARS Jure Jerman Meteorological Office Environmental Agency of Slovenia (EARS)
1 CCOS Seasonal Modeling: The Computing Environment S.Tonse, N.J.Brown & R. Harley Lawrence Berkeley National Laboratory University Of California at Berkeley.

1 CCOS Seasonal Modeling: The Computing Environment S.Tonse, N.J.Brown & R. Harley Lawrence Berkeley National Laboratory University Of California at Berkeley.

Similar presentations

Ads by Google