EMC’s Offsite Computing Update HPC RAC 05 January 2012

HPC DetailsAcctsAllocationAllocation UsedProjects GAEA -Cray XT6 -ORNL -Oak Ridge, TN (pendi ng) 6.2% of System resources (5.04M Core Hours per month) <1% Develop GDAS/GFS parallel capability Test GFS Semi-Lagrangian development CFS-L completion GFS coupled atm/ocean testing T-Jet -Linux -GSD -Boulder, CO ~10HWRF – 1380 Cores GDAS – 5400 Cores SL-GFS 3000 Cores, 5 user accounts HWRF – 2011: 4.7 M Corehours GDAS – 3000 Cores 2.0 M Corehours in Q1 and Q2FY12 SL-GFS – User applications have been approved. Still waiting for tokens being delivered to users. HWRF FY12 pre-implementation testing of 3-km moving nest HWRF ensembles GFS Semi-Lagrangian development/testing for FY12 HFIP Demo and FY13 operational implementation Preliminary development of 4d-Var component of Hybrid DA Preliminary testing of ENKF ensembles to week 1 & 2 medium range NWP UCSD San Diego Supercomputer Center -Linux -San Diego, CA 1TBD Develop 30 year Hindcast database using WAVE Model. Reanalysis of winds with high resolution runs in North Sea, Mediterranean Sea, Horn of Africa DOE Computer0n/a Marine Branch working on getting project started here

Accounts: – 31 EMC staff have accounts – Requests for 11 accounts are pending approval by NCO Initial Code Porting finished: – GFS Ops and Semi-Lagrangian low resolution models – GSI Ops – HWRF – NEMS/NMMB – NDAS – RSM – GODAS/MOM4 – NEMS I/O – Numerous production libraries compiled: w3lib, splib, iplib, bacio, bufrlib, and crtm – Two versions of sfcio, sigio and gfsio required to support big or little-endian data. Functional Equivalence Testing on different platforms are still being performed. Codes have NOT been optimized as yet. They are just being ported correctly first. EMC Model Transition Update

Plans are underway for porting the following systems: – HYBRID EnKF – RTOFS – WAVEWATCH III – NEMS GFS and GEFS – SREF – CFSRL Script Porting: – Porting scripts requires significant changes to syntax and takes much longer to port. – Plan is to have a unified script package for use on IBM and GAEA/ZEUS – Global forecast, chgres and post scripts are working – Work is being done on the Global prep, verification and archival scripts No end to end systems have been run. Disk Storage: – Request for 20TB of permanent disk space on fast scratch space to hold 18 months of data ingest to run full assimilation experiments: not yet obtained. EMC Model Transition Update

Problems with Porting: A.Most users have not been given batch job access. B.MPI-Gather does not work. C.Threading does not work. D.Slowness in running codes. Example: – We cannot get T1148L64 SLG to run above 160 tasks: aborts with no traceback – Tests indicate this is NOT a code issue – Cray has not responded to a request for help EMC Model Transition Update

Overall Problems Still Remain: Data transfers to/from Gaea remains a serious issue. It is not possible to transfer large amounts between Gaea and NCEP without using a third party system (Jet) as a relay—not efficient. Restricted Data Policy: – This is a serious issue since we cannot conduct development testing without a policy in place – Requires administrative management action (NCO working the issue and can provide status on request) EMC Model Transition Update

Overall Progress being made: Increase in the number of users that now have access to Gaea More documentation has been provided to NCEP All day user support workshop to be held Nov 1 at NCEP Plans to have reliable help desk support EMC Website set up for Model Transition Team Activities EMC Model Transition Update

Plans for Herc/Zeus being made: List of priority users being prepared for the TDS system Disk configuration plans for Zeus are being prepared, similar to the configuration on the CCS systems (scrub, noscrub, save, etc) Will require direct data transfers from the CCS machines to Zeus. Will require restricted data policy on Zeus. EMC Model Transition Update