ECMWF Slide 1CAS2K3, Annecy, 7-10 September 2003 Report from ECMWF Walter Zwieflhofer European Centre for Medium-Range Weather Forecasting

ECMWF Slide 2CAS2K3, Annecy, 7-10 September 2003 Topics covered Operational forecasting system Planned improvements High-performance computing system Data handling system Reanalysis activities

ECMWF Slide 3CAS2K3, Annecy, 7-10 September 2003 Operational forecasting system Forecasts of the atmosphere and ocean waves to ten-days ahead -Twice per day -T511 (40km) 60-level deterministic model -T255 (80km) 40-level ensemble prediction system -All of the above are coupled to an ocean-wave model Forecasts to one month ahead -Ensembles using a T159 (125km) L40 atmospheric model coupled to an ocean circulation model (110km) -Currently run experimentally once per fortnight Seasonal forecasts -Once per month, using a T95 (210km) L40 atmospheric model and the same ocean model used for the one-month forecasts

ECMWF Slide 4CAS2K3, Annecy, 7-10 September 2003 Data assimilation Four-dimensional variational (4D-Var) analysis with -12-hour data window -Inner-loop minimizations at up to T159 (125km) L60 resolution -Outer-loop resolution of T511 (40km) L60 Short cut-off analyses -3D-Var with 6-hour data window -4 times per day -Used to supply boundary conditions to several Member States for Limited Area Modelling

ECMWF Slide 5CAS2K3, Annecy, 7-10 September 2003 ECMWF forecast verification (12UTC)

ECMWF Slide 6CAS2K3, Annecy, 7-10 September 2003 Evolution of the number of products disseminated per day Total is now is 1.3 M per day Was 300k in 2000

ECMWF Slide 7CAS2K3, Annecy, 7-10 September 2003 Research directions & operational targets For 2004: -Assimilation of MSG data and additional ENVISAT and AIRS data -Assimilation of rainfall measurements -Increased vertical resolution, particularly in the vicinity of the tropopause -Preparations for upgrades in horizontal resolution Target resolutions for 2005: -Deterministic forecast:T799 (25km) L91 -EPS:T399 (50km) L65 -4D-Var (inner/outer):T255/T799 L91 Compute resources required for the above will be delivered by the IBM Phase 3 system

ECMWF Slide 8CAS2K3, Annecy, 7-10 September 2003 Slide from CAS-2001: Planned changes New DHS New Servers New HPC system New Network

ECMWF Slide 9CAS2K3, Annecy, 7-10 September 2003 Computer configuration (September 2003)

ECMWF Slide 10CAS2K3, Annecy, 7-10 September 2003 Current HPC configuration Two IBM cluster 1600s with 30 p690 servers each Each p690 is partitioned into four 8-way nodes Each of the clusters has 120 nodes -12 of the nodes in each cluster have 32 GB memory -All other nodes have 8 GB memory -Processors run at 1.3 GHz -960 processors per cluster for user work -Dual-plane colony switch with PCI adapters -4.2 terabytes of disk space per cluster Both clusters are configured identically

ECMWF Slide 11CAS2K3, Annecy, 7-10 September 2003 HPC Migration: project milestones June 02Delivery of cluster-A of Phase 1 August 02Delivery of cluster-B of Phase 1 31 October 02Passed the Functional Test (including performance commitments) 08 December 02Passed the Operational Test 17 January 03Full service for external users 07 February 03Passed the Reliability Test (availability was > 99%) 04 March 03Switch-over of the operational suite 31 March 03End of the Fujitsu service

ECMWF Slide 12CAS2K3, Annecy, 7-10 September 2003 Migration: general experience The system needed a larger number of nodes than originally envisaged to meet the contracted performance commitments The in-depth testing performed in autumn 2002 identified a number of problems that had to be addressed before the acceptance process could complete; IBM worked very hard to resolve these in a timely manner A lot of attention went into system related issues such as minimising boot times The migration of the operational suite went smoothly The system is very stable and the reliability is high

ECMWF Slide 13CAS2K3, Annecy, 7-10 September 2003 User availability of the HPC systems

ECMWF Slide 14CAS2K3, Annecy, 7-10 September 2003 HPC upgrade planned for 2004 Phase 3 of the contract with IBM will be a replacement of the currently installed hardware (apart from the reuse of disk space) Based on p690+ servers and the Federation switch Two p690+ servers are already installed at ECMWF for testing purposes The test setup will be enhanced over the next two months to include more nodes and the new switch Main delivery will start in early 2004 Phase 3 will deliver around 2 teraflops of sustained performance on ECMWF’s main codes (around 5x the sustained performance of the final Fujitsu configuration)

ECMWF Slide 15CAS2K3, Annecy, 7-10 September 2003 Architecture change: Fujitsu VPP to IBM Cluster 1600 Peak performance 9.6 Gflops per processor Fujitsu VPP vector processors IBM p690 2 x 960 scalar processors Peak performance 5.2 Gflops per processor 8 processors per shared memory node 8 GB memory per node4 GB memory per CPU

ECMWF Slide 16CAS2K3, Annecy, 7-10 September 2003 T L 1023 ~ 20 km

ECMWF Slide 17CAS2K3, Annecy, 7-10 September 2003 TL511L60 forecast model (26R3)

ECMWF Slide 18CAS2K3, Annecy, 7-10 September 2003 Comparison of 8-CPUs v 7-CPUs (TL511) Speedup in elapsed time using 16 LPARs tasks CPUs/LPAR 128 x 1 thread (8 CPUs) = x 1 thread (7 CPUs) = x 4 threads (8 CPUs) = 1.20 Speedup in elapsed time using 32 LPARs tasks CPUs/LPAR 256 x 1 thread (8 CPUs) = x 1 thread (7 CPUs) = x 4 threads (8 CPUs) = 1.42

ECMWF Slide 19CAS2K3, Annecy, 7-10 September 2003 Data handling system ECMWF has used Tivoli Storage Manager (TSM) for several years but encountered scalability issues In 2001, HPSS was chosen as a replacement Projected data storage volumes (including backups) -mid-2003:1.4 petabytes -mid-2004:2.1 petabytes -mid-2005:3.2 petabytes Experience with HPSS is very positive: -Complex system but very scalable -Excellent support when problems are encountered Timely migration of the data was a major concern at the project planning stage

ECMWF Slide 20CAS2K3, Annecy, 7-10 September 2003 Migration of data from TSM to HPSS

ECMWF Slide 21CAS2K3, Annecy, 7-10 September 2003 ERA- 40 project A global re-analysis of observations covering the period September 1957 to August 2002 Based on cycle 23r4 of ECMWF forecasting system - operational from June 2001 to January 2002 Six-hourly 3D-Var analysis - operations uses 12-hourly 4D-Var T159 horizontal resolution (~125km grid) - operations uses T511 (~39km grid)

ECMWF Slide 22CAS2K3, Annecy, 7-10 September 2003 ERA- 40 status Produced with considerable external support: - Most of the older observations were supplied by NCAR via NCEP - SST and sea-ice analyses were produced by the Met Office and NCEP - EUMETSAT supplied reprocessed satellite winds - Validation partners provided valuable feedback - Practical support from EU, Fujitsu, IAP, JMA, PCMDI, WCRP, GCOS … Production was completed in April 2003 Full set of products is available from ECMWF MARS 2.5 o products are available on a public data server Data is being supplied to NCAR for UCAR use and general use in USA for research and education

Counts of observations accepted by ERA-40 system AIRCRAFT SATELLITE WINDS SATELLITE RADIANCES BUOYS

ECMWF Slide 24CAS2K3, Annecy, 7-10 September 2003 Anomaly correlations of 500hPa height forecasts Ops 1980 Ops 2002/3 Ops 2001 ERA 2001 Australia/New Zealand %