Computing at Norwegian Meteorological Institute Roar Skålin Director of Information Technology Norwegian Meteorological Institute CAS 2003 – Annecy

Norwegian Meteorological Institute met.no

Norwegian Meteorological Institute Main office in Oslo Regional offices in Bergen and Tromsø Aviation offices at four military airports and Spitsbergen Three arctic stations: Jan Mayen, Bear Island and Hopen 430 employees ( )

Norwegian Meteorological Institute met.no met.no Computing Infrastructure SGI O /512/7.2 SGI O /304/7.2 CXFS Climate Storage 2/8/20 Backup Server SGI O200/2 S-AIT 33 TB DLT 20 TB Production Dell 4/8 Linux Production Dell 2/4 Linux NetApp 790 GB Scali Cluster 20/5/0.3 XX Cluster y/y/y STA 5 TB Storage Server SGI O2000/2 DLT 20 TB met.no - Oslo NTNU - Trondheim Switch Router 2.5 GBit/s 155 MBit/s 1 GBit/s 100 MBit/s x/y/z = processors/GB memory/TB disk

Norwegian Meteorological Institute met.no met.no Local Production Servers Dell PowerEgde servers with two and four CPUs NetApp NAS Linux ECMWF Supervisor Monitor Scheduler (SMS) Perl, shell, Fortran, C++, XML, MySQL, PostgreSQL Cfengine Production Environment November 2003:

Norwegian Meteorological Institute met.no Linux replace proprietary UNIX at met.no Advantages: –Off-the-shelf hardware replace proprietary hardware Reduced cost of new servers Reduced operational costs –Overall increased stability –Easier to fix OS problems –Changing hardware vendor becomes feasible –Become an attractive IT-employer with highly motivated employees Disadvantages: –Cost of porting software –High degree of freedom: a Linux distribution is as many systems as there are users

Norwegian Meteorological Institute met.no Data storage: A critical resource We may loose N-1 production servers and still be up-and-running, but data must be available everywhere all the time We used to duplicate data files, but increased use of databases reduce the value of this strategy met.no replace a SAN by a NetApp NAS because: –availability –Linux support –”sufficient” IO-bandwidth (40-50 MB/s per server)

Norwegian Meteorological Institute met.no HPC in Norway – A national collaboration

Norwegian Meteorological Institute met.no Performance available to met.no CRAY X-MP CRAY Y-MP CRAY T3E SGI O3000

Norwegian Meteorological Institute met.no met.no Production Compute Servers SGI Origin 3800 Embla: 512 MIPS R14K processors 614 Gflops peak 512 GB memory Gridur: 384 MIPS R14K processors 384 Gflops peak 304 GB memory Trix OS / LSF batch system 7.2 TB CXFS filesystem

Norwegian Meteorological Institute met.no Atmospheric models HIRLAM 20HIRLAM 10HIRLAM 5UMMM5 PurposeOperation Exp Air Poll. Resolution20 km 40 layers 360 s 10 km 40 layers 240 s 5 km 40 layers 120 s 3 km 38 layers 75 s 3 km 17 layers 9 s 1 km 17 layers 3 s Grid Points 468*378248*341152*150280*27661x76 Forecast time 60 h48 h Result data 24 h 8 GB1.2 GB0.3 GB2.1 GB0.1 GB

Norwegian Meteorological Institute met.no Oceanograpic models MIPOM22ECOM3DWAVEECOM3D PurposeOperation Exp Resolution4 km 17 layers 150 s km 21/5 layers 600 s 45/8 km4/300m km 17 layers 360/50 s Grid Points 1022x578208x120142x x x250 Forecast time 60 h Result data 24 h 1 GB

Norwegian Meteorological Institute met.no Production timeline HIRLAM20 HIRLAM10 HIRLAM5 UM MM5 ECOM3D/WAVE ECOM3D MIPOM22

Norwegian Meteorological Institute met.no HIRLAM scales, or …? The forecast model without I/O and support programs scales reasonably well up to 512 processors on a SGI O3800 In real life: –data transfer, support programs and I/O has a very limited scaling –there are other users of the system –machine dependent modifications to increase scaling has a high maintenance cost for a shared code such as HIRLAM For cost-efficient operational use, 256 processors seems to be a limit

Norwegian Meteorological Institute met.no How to utilise 898 processors operationally? Split in two systems of 512 and 384 processors and used as primary and backup system Will test a system to overlap model runs based on dependencies: HIRLAM 20 HIRLAM 10 ECOM3D HIRLAM 5 WAVE

Norwegian Meteorological Institute met.no Overlapping Production Timeline HIRLAM20 HIRLAM10 HIRLAM5 UM MM5 ECOM3D/WAVE ECOM3D MIPOM22

Norwegian Meteorological Institute met.no RegClim: Regional Climate Development Under Global Warming Overall aim: Produce scenarios for regional climate change suitable for impact assessment Quantify uncertainties Some keywords: Involve personell from met.no, universities and research organisations Based on global climate scenarios Dynamical and empircal downscaling Regional and global coupled models Atmosphere – ocean – sea-ice

Norwegian Meteorological Institute met.no Climate Computing Infrastructure SGI O /512/7.2 SGI O /304/7.2 CXFS Climate Storage 2/8/20 S-AIT 33 TB Para//ab - Bergen NTNU - Trondheim Router 2.5 GBit/s 155 MBit/s IBM Cluster 64/64/0.58 IBM p690 Regatta 96/320/7 IBM TB x/y/z = processors/GB memory/TB disk

Norwegian Meteorological Institute met.no Climate Storage Server Low-cost solution: Linux server Brocade switch Nexsan AtaBoy/AtaBeast RAID 19.7 TB 34 TB Super-AIT library, tapecapacity 0.5 TB uncompressed

Norwegian Meteorological Institute met.no GRID in Norway Testgrid comprising experimental computers at the four universities Globus 2.4 -> 3.0 Two experimental portals: Bioinformatics and Gaussian Plan testing of large datasets (Storage Resource Broker) and metascheduling autumn 2003 Plan to phase in production supercomputers in 2004