CHAP Meeting 4 October 2007 CISL Update Operations and Services CISL HPC Advisory Panel Meeting 4 October 2007 Tom Bettge Director of Operations and Services Computational and Information Systems Laboratory

CHAP Meeting 4 October during the past six months…….. blueice bluevista

CHAP Meeting 4 October 2007 Resource Utilization

CHAP Meeting 4 October 2007 Challenges for CISL and CHAP (a good thing!!) l Witness: 1.7M GAUs available, 1.0M GAUs requested l Witness: NCAR Resources Increasing (another bump in Sept 2008) University use could fall behind NCAR University use could fall behind NCAR l Tapping “Latent” GAUs? (CISL’s challenge) –Special queue access by request for large university users –Consider requiring projects with large allocations to use in a “paced” manner, monthly, quarterly, etc. l Filling the surplus gap for university PIs –Awareness (CISL and NSF) –Mirror the NCC n Only one project selected for BTS in Nov, 5 were submitted –Leave well enough alone – “Build it and they will come.” from April CHAP Meeting

CHAP Meeting 4 October 2007 Large Community Requests (10 6 GAUs)

CHAP Meeting 4 October 2007 Community (non-CSL) GAUs Used

CHAP Meeting 4 October 2007 Breakthrough Science (BTS) Initiative Spring 2007

CHAP Meeting 4 October 2007 Breakthrough Science Projects Usage PIAffiliation NSF Prog Title GAUs Used Eric Chassignet Florida State OCE Building a New Version of CCSM3 with HYCOM as the Ocean Model 452,000 Dave Randall CSUATM Global Climate Modeling with Super Parameterization 325,000 Brian Savage University of Rhode Island EAR 3-D Tomography of the Crust and Upper Mantle Beneath the Gulf Extensional Province and Baja California 1000 Bill Smyth Oregon State OCE Instability and Turbulence in a Sheared, Diffusively Unstable Fluid 279,000 Ron Cohen Carnegie Inst. of Washington EAR Quantum Monte Carlo on Geophysical Materials 521,000 David Yuen University of Minnesota EAR (1) Poro-elastic Wave Propagation in Oil Reservoirs (2) 3-D Tsunami Wave Simulations 39,000 Annick Pouquet NCARNCAR Small Scale Structures in MHD Turbulence: Toward a Better Understanding of the Magnetosphere 233,000 Rich Rotunno NCARNCAR High Resolution Hurricane Simulations 726,000

CHAP Meeting 4 October 2007 University Capability Computing NCAR Capability Computing PIAffiliation NSF Prog Title GAUs Used David Straus COLAATM Detection and Attribution of Climate Change: The Interactive Ensemble 424,000 Chris Davis NCARNCAR Advanced Hurricane WRF Model Trials 227,000

CHAP Meeting 4 October 2007 Program Distribution

CHAP Meeting 4 October 2007 BTS Outcomes l CISL was able to provide large amounts of computing time through the use of special queues and the fair share scheduler l Three university projects have submitted large requests to CHAP because of BTS or UCC simulations –Straus, 423K –Cohen, 250K –Smyth, 70K l Very likely to provide early Capability Computing Opportunity in 2008 with availability of bluefire (some caveats here) –community eligibility TBD –start solicitation earlier –require more detailed proposals –users provide NCAR with more precise benchmarks (NCAR will help in advance) –projects which need resources and are ready-to-go

CHAP Meeting 4 October 2007 ICESS Computing Status

CHAP Meeting 4 October 2007 ICESS System Phase II l Jan 2007 to ~Jun 2008: 1.9 GHz POWER5+ p575 SMP nodes dual-link Federation Switch 4 TB memory, 150 TBytes disk 1600 batch processors – 16 processors/node 4 bluesky equivalents, 12 TFLOPs peak l ~Jun 2008 to ~Jun 2011: 4.7 GHz POWER6 p575+ SMP nodes quad-link Federation Switch 9.2 TB memory, 150 TBytes disk ~3200 batch processors – 32 processors/node 15.5 bluesky equivalents, 57.6 TFLOPs peak Phase I Phase II blueice bluefire

CHAP Meeting 4 October 2007 Sustained Capacity – with ICESS

CHAP Meeting 4 October 2007 ICESS Subcontract Performance Commitments l blueice is 4 bluesky-equivalents l bluefire is to be 15.5 bluesky-equivalents The following table shows committed average code speedups (each code is run on the indicated range of processor counts). Average blueice speedup over bluesky Average est’d bluefire speedup over bluesky Est’d bluefire speedup over blueice CAM (64,256) HD3D (8,16,32,64,128) POP (8,16,24,32,48,64,128) WRF (1,2,4,8,16,32,64,128,256) Weighted Average* *Weighted Average: 45% CAM, 10% HD3D, 20% POP, 25% WRF

CHAP Meeting 4 October 2007 Facility Constraints

CHAP Meeting 4 October 2007 Liquid Cooling !!!! l Power6 uses even more direct methods of heat transfer and reduces reaction time for mechanical systems

CHAP Meeting 4 October 2007 bluefire Schedule l April 2008delivery –Power up enough bluefire to match blueice production workload l May 2008ATP completed on partial bluefire l June 2008blueice decommissioned l July 2008ATP completed on entire bluefire l Aug-Nov 2008Capability Computing Opportunity –UCC and NCC details TBD l October 2008bluevista decommissioned l Dec 2008Full availability of bluefire –e.g., CSL not allocated fully until December 1

CHAP Meeting 4 October 2007 MSS Plans

CHAP Meeting 4 October 2007 MSS Growth

CHAP Meeting 4 October 2007 MSS Observations l Current growth rate is TB/month l Capacity of current archive (6 PB) will be exhausted June-Sept 2008 l Growth rate will be higher with bluefire availability l Estimate capacity of ≥20 PB needed prior to move to NSC l Near Term Plan –Engage vendors with estimated requirements –Request proposals for 4 year contract to partner with NCAR for MSS expansion –Award, install, and begin production by Sept 2008 –Phased decommission of current silos

CHAP Meeting 4 October 2007 CAS2K7 – September 9-13, 2007 l Keynote Speakers –emphasis on petascale computing and interdisciplinary applications and data access l Trends –Power/performance impacts on facilities – huge issues –Grids becoming increasingly useful –Important problems are interdisciplinary –Increasing parallellism, not faster clocks Presentations:

CHAP Meeting 4 October 2007 Questions and Discussion

CHAP Meeting 4 October 2007 Main HPC Systems at NCAR l IBM Power5 p575 l GHz P5pes l 4.7 Tflop Peak l 210 kW l IBM Power6 p575+ l GHz P6pes l 57.6 Tflop Peak l 700 kW l IBM Power5+ p575 l GHz P5pes l 12.0 Tflop Peak l 290 kW IBM Power5 IBM Power6 IBM Power5+ bluevistablueicebluefire

CHAP Meeting 4 October HPC Resource Split NCAR Capability Computing

CHAP Meeting 4 October 2007 Breakthrough Science (BTS) Initiative Spring 2007 l First four months of blueice production use l 75% of blueice devoted to BTS –NSF selected five projects –CHAP selected one projects –NCAR selected two projects l Consultant assigned to each project l Special queues: bts_prm, bts_reg, bts_sby, bts_ded – provided prototype access for capability computing