1. Overview of the New Blue Gene/L Computer Dr. Richard D. Loft Deputy Director of R&D Scientific Computing Division National Center for Atmospheric Research

2. Outline What is Blue Gene/L and why is it interesting? How did one end up at NCAR? What is the objective of the NCAR Blue Gene/L project? What is the status of it? How do I get an account on Blue Gene/L?

3. Why Blue Gene/L is Interesting Features Massive parallelism - fastest in world. (137 Tflops) Achieves high packaging density. (2048 pes/rack) Lower power per processor. (25 KW/rack) Dedicated reduction network. (solver scalability) Puts network interfaces on chip. (embedded tech.) Conventional programming model: xlf90, xlcc compiler MPI

4. Fuel Efficiency: Gflops/Watt Top 20 systems Based on processor power rating only Blue Gene/L Systems

5. BG/L Questions/Limitations Questions High reliability? (1/N effect) Applications for 100k processors? (Amdahl’s Law) System robustness: I/O, scheduling flexibility. Limitations Node Memory Limitation (512 MB/node) Partitioning is quantized (power of two) Simple node kernel - (no: forks-> threads -> OMP) No support for multiple executables.

6. BlueGene/L ASIC

7. The Blue Gene/L Architecture

8. BlueGene/L Has Five Networks 3-Dimensional Torus –interconnects all compute nodes –175 MB/sec/link bidirectional Global Tree –point-to-point, one-to-all broadcast, reduction functionality –1.5 microsecond latency ( @64K node ) Global Interrupts –AND/OR operations for global barriers –1.5 microseconds latency (64K system) Ethernet –incorporated into every node ASIC –active in the I/O nodes (1:64 in LLNL configuration) 1K 1Gbit links –all external comm. (file I/O, control, user interaction, etc.) JTAG (Control)

9. BlueGene/L System Software Architecture User applications execute exclusively in the compute nodes –avoid asynchronous events (e.g., daemons, interrupts) The outside world interacts only with the I/O nodes, an offload engine –standard solution: Linux Machine monitoring and control also offloaded to service nodes: large SP system or Linux cluster.

10. Blue Gene/L system overview

11. Blue Gene/L @ NCAR

12. How did one get to NCAR? MRI proposal in partnership with CU’s Elements of MRI proposal to NSF: proving out an experimental architecture. –Application porting and scalability –System software testing Parallel file systems (Lustre, GPFS) Schedulers (LSF, SLURM, COBALT) –Education

13. BlueGene/L Collaboration NCAR CU Denver CU Boulder Blue Gene/L

14. BlueGene/L Collaborators NCAR –Richard Loft –Janice Coen –Stephen Thomas –Wojciech Grabowski CU Boulder –Henry Tufo –Xiao-Chuan Cai –Charbel Farhat –Thomas Manteuffel –Stephen McCormick CU Denver –Jan Mandel –Andrew Knyazev Blue Gene/L

15. Details of NCAR/CU Blue Gene/L 2048 processors, 5.73 Tflops peak 4.61 Tflops on Linpack Benchmark Unofficially, 33rd fastest system in the world (in one rack!) 6 Tbytes of high performance disk Delivered to Mesa Lab: March 15th Acceptance tests –began March 23rd. –Completed March 28th. –First PI meeting March 30th.

16. BG/L Front-End Architecture

17. Bring-up of Frost BG/L System Criteria for readiness –Scheduler –Fine Grain Partitions –I/O subsystem ready –MSS connection

18. Current “Frost” BG/L Status MSS connections in place. I/O system issues appear to be behind us. Partition definitions (512,256,128, 4x32) in place. Codes ported: POP, WRF, HOMME, BOB, BGC5 (pointwise) Biggest apps issue: memory footprint Establishing relationships with other centers –BG/L Consortium membership –Other BG/L sites: SDSC, Argonne, LLNL, Edinburgh

19. “Frost” BG/L I/O performance

20. Blue Gene/L “Frost” scheduler status IRC chat room scheduler - “hey, get off!” …done LLML SLURM scheduler -testing –has been installed, tested, available for 512 node “midplane” partitions only. –LLNL testbed system will be used to port SLURM to smaller partitions. Argonne Cobalt scheduler - being installed –DB2 Client on the FEN –Python –Elementtree (XML process library for Python) –Xerces (XML parser) –Supporting libraries (Openssl) Platform LSF - development account provided.

21. MRI Investigator Phase MRI Investigator access only –Users related to MRI proposal –Porting/testing evaluation Applications –HOMME atmospheric GCM dycore (Thomas) –Wildfire modeling (Coen) –Scalable solvers - algebraic multigrid (Manteuffel, McCormick) –Numerical Flight Test Simulation (Farhat) –WRF - high resolution (Hacker)

22. User Access to Frost Cycles split –50% UCAR –40% CU Boulder –10% CU Denver Interested users (access policy TBD) –UCAR: contact loft@ucar.eduloft@ucar.edu –CU: contact tufo@ucar.edutufo@ucar.edu

23. Questions?