1. Designing a cluster for geophysical fluid dynamics applications Göran Broström Dep. of Oceanography, Earth Science Centre, Göteborg University.

2. Our cluster (me and Johan Nilsson, Dep. of Meterology, Stockholm University) Grant from the Knut & Alice Wallenberg foundation (1.4 MSEK) 48 cpu cluster Intel P4 2.26 Ghz 500 Mb 800Mhz Rdram SCI cards Delivered by South Pole Run by NSC (thanks Niclas & Peter)

3. What we study

4. Geophysical fluid dynamics Oceanography Meteorology Climate dynamics

5. Thin fluid layers Large aspect ratio

6. Highly turbulent Gulf stream: Re~10 12

7. Large variety of scales Parameterizations are important in geophysical fluid dynamics

8. Timescales Atmospheric low pressures: 10 days Seasonal/annual cycles: 0.1-1 years Ocean eddies:0.1-1 year El Nino: 2-5 years. North Atlantic Oscillation: 5-50 years. Turnovertime of atmophere: 10 years. Anthropogenic forced climate change: 100 years. Turnover time of the ocean: 4.000 years. Glacial-interglacial timescales: 10.000-200.000 years.

9. Some examples of atmospheric and oceanic low pressures.

10. Timescales Atmospheric low pressures: 10 days Seasonal/annual cycles: 0.1-1 years Ocean eddies:0.1-1 year El Nino: 2-5 years. North Atlantic Oscillation: 5-50 years. Turnovertime of atmophere: 10 years. Anthropogenic forced climate change: 100 years. Turnover time of the ocean: 4.000 years. Glacial-interglacial timescales: 10.000-200.000 years.

11. Normal state

12. Initial ENSO state

13. The ENSO state

15. Timescales Atmospheric low pressures: 10 days Seasonal/annual cycles: 0.1-1 years Ocean eddies:0.1-1 year El Nino: 2-5 years. North Atlantic Oscillation: 5-50 years. Turnovertime of atmophere: 10 years. Anthropogenic forced climate change: 100 years. Turnover time of the ocean: 4.000 years. Glacial-interglacial timescales: 10.000-200.000 years.

16. Positive NAO phase Negative NAO phase

18. Positive NAO phase Negative NAO phase

20. Timescales Atmospheric low pressures: 10 days Seasonal/annual cycles: 0.1-1 years Ocean eddies:0.1-1 year El Nino: 2-5 years. North Atlantic Oscillation: 5-50 years. Turnovertime of atmophere: 10 years. Anthropogenic forced climate change: 100 years. Turnover time of the ocean: 4.000 years. Glacial-interglacial timescales: 10.000-200.000 years.

21. Temperature in the North Atlantic

22. Timescales Atmospheric low pressures: 10 days Seasonal/annual cycles: 0.1-1 years Ocean eddies:0.1-1 year El Nino: 2-5 years. North Atlantic Oscillation: 5-50 years. Turnovertime of atmophere: 10 years. Anthropogenic forced climate change: 100 years. Turnover time of the ocean: 4.000 years. Glacial-interglacial timescales: 10.000-200.000 years.

23. Ice coverage, sea level

24. What model will we use?

25. MIT General circulation model

26. General fluid dynamics solver Atmospheric and ocean physics Sophisticated mixing schemes Biogeochemical modules Efficient solvers Sophisticated coordinate system Automatic adjoint schemes Data assimilation routines Finite difference scheme F77 code Portable

27. MIT General circulation model Spherical coordinates“Cubed sphere”

28. MIT General circulation model General fluid dynamics solver Atmospheric and ocean physics Sophisticated mixing schemes Biogeochemical modules Efficient solvers Sophisticated coordinate system Automatic adjoint schemes Data assimilation routines Finite difference scheme F77 code Portable

29. MIT General circulation model

37. Some computational aspects

38. Some tests in INGVAR (32 AMD 900 Mhz cluster)

39. Experiments with 60*60*20 grid points

42. Experiments with 120*120*20 grid points

43. MM5 Regional atmospheric model

46. Choosing cpu’s, motherboard, memory, connections

47. Specfp (swim)

48. Run time on different nodes

49. Choosing interconnection (requires a cluster to test) Based on earlier experience we use SCI from Dolphinics (SCALI)

50. Our choice Named Otto SCI cards P4 2.26 GHz (single cpus) 800 Mhz Rdram (500 Mb) Intel motherboards (the only available) 48 nodes NSC (nicely in the shadow of Monolith)

51. Otto (P4 2.26 GHz)

52. Scaling Otto (P4 2.26 GHz) Ingvar (AMD 900 MHz)

53. Why do we get this kind of results?

54. Time spent on different “subroutines” 60*60*20120*120*20

55. Relative time Otto/Ingvar

56. Some tests on other machines INGVAR: 32 node, AMD 900 MHz, SCI Idefix: 16 node, Dual PIII 1000 MHz, SCI SGI 3800: 96 Proc. 500 MHz Otto: 48 node, P4 2.26 Mhz, SCI ? MIT, LCS: 32 node, P4 2.26 Mhz, MYRINET

57. Comparing different system (120*120*20 gridpoints)

59. Comparing different system (60*60*20 gridpoints)

60. SCI or Myrinet? 120*120*20 gridpoints

61. SCI or Myrinet? 120*120*20 gridpoints (60*60*20 gripoints) (ooops, I used the ifc Compiler for these tests)

62. SCI or Myrinet? 120*120*20 gridpoints (60*60*20 gripoints) (ooops, I used the ifc Compiler for these tests) (1066Mhz rdram?)

63. SCI or Myrinet? (time spent in pressure calc.) 120*120*20 gridpoints (60*60*20 gripoints) (ooops, I used the ifc Compiler for these tests) (1066Mhz rdram?)

64. Conclusions Linux clusters are useful in computational geophysical fluid dynamics!! SCI cards are necessary for parallel runs >10 nodes. For efficient parallelization: >50*50*20 grid points per node! Few users - great for development. Memory limitations, for 48 proc. a’ 500 Mb, 1200*1200*30 grid points is maximum (eddy resolving North Atlantic, Baltic Sea). For applications similar as ours, go for SCI cards + cpu with fast memory bus and fast memory!!

65. Experiment with low resolution (eddies are parameterized)

69. Thanks for your attention