1. CCS machine development plan for post- peta scale computing and Japanese the next generation supercomputer project Mitsuhisa Sato CCS, University of Tsukuba

2. 2 2010.2.22 #node 2560 node (Intel Xeon 2.8GHz, single core /node) peak performance 14.34 TF memory 5 TB network 250MB/s/link x 3 (3D-HXB by GbE) #node 2560 node (Intel Xeon 2.8GHz, single core /node) peak performance 14.34 TF memory 5 TB network 250MB/s/link x 3 (3D-HXB by GbE) Designed by T2K Open Supercomputer Alliance (U. Tokyo and Kyoto U) Spec ； 648 nodes (quad Opteron, 4sockets/node) 10000 cores Peak performance 95.4TF total memory 20TB total disk capacity 800TB ( 20th in top 500, June, 2008) A Special-purpose system to Astrophysics simulation by hybrid computation of radiation and N-body. Each node is equipped by GRAPE-6, which is an accelerator specialized for N-body Gravity calculation. 256 nodes performance: cluster 3.5TFLOPS ＋ Grape-6 35TFLOPS PACS-CS FIRST T2K-tsukuba GRAPE-6 (2006 ～） (2007 ～） (2008 ～） Computing resources in CCS

3. 3 2010.2.22 System installation and future plans 200420122011201020092005200620072008 FCS-IV FCS-V HA-PACS (planned) PACS-CS CP-PACS T2K FIRST H16H17H18H19H20H21H22H23H24 （計画） 2011-2013 VPP suspended 2013 NGS (10PF) FCS: Front-end system the next system to T2K

4. 4 2010.2.22 Issues for Post-peta scale systems (not exa?) System to enable strong-scaling the current petascale system enabled by weak-scaling We need more powerful node & network GPGPU is one of solution More specialized architecture we need a sharp science target All applications cannot use More difficult to program Need supports from CS-side Collaboration with computer science and computational science CCS's mission

5. 5 2010.2.22 HA-PACS: Highly Accelerated Parallel Advanced system for Computational Sciences (planned) Objective: to investigate acceleration technologies for post-petascale computing and its software, algorithms and computational science applications, and demonstrate by building a prototype system Design and deploy a GPGPU-based Cluster system Research on programming model and languages, environment for parallel system with accelerators. Design of Algorithms and applications for parallel system with accelerators. Research on architectures for parallel system with accelerators. examples

6. 6 2010.2.22 IB switch.............. 12 node IB switch.............. 12 node IB switch.............. 12 node HA-PACS/NG powered by PEARL Link CPU GPGPU CPU GPGPU Infiniband QDR.............. GPGPU.............. PEARL Link PCIe Infiniband QDR PEARL: PCI-Express Adaptive and Reliable Link  Use PCI-Express as a high-speed link  Connect CPU and devices including GPGPU through a router chip, PEACH (PCI-Express Adaptive Communication Hub)

7. 7 2010.2.22 Strategic target computational sciences of HA-PACS ① Bio-physics : high performance QM/MM hybrid simulation for mechanisms of high- efficiency enzymatic reactions, electronic and 3D structures of biomacromolecules Speedup of QM is a key for this simulations ② astrophysics: full Hydrodynamics and radiative- transfer simulation for the Universe and Formation of Astronomical objects Full 6 dimensional simulation is required ③ Particle physics: full-lattice QCD simulation

8. Japanese the next generation supercomputer project

9. 9 2010.2.22 background: Japanese government plan The 3 rd Science and Technology Basic Plan (FY2006-FY2010) “ Next-generation super computing technology ” is selected as one of key technologies of national importance  Development and installation of the advanced high performance supercomputer system (10petaflops) → the Next-Generation Supercomputer  Development application software  Establishment of “ Advanced Computational Science and Technology Center ” (tentative name) The 4th Science and Technology Basic Plan (FY2011-FY2015) (Now under discussion)  Exaflops class HPC Technology New chip device, software, hardware … After the election of the House of Representatives in the last summer, ….  In the November of the last year, the new government party have decided to freeze the plan of the development at the screening of government projects!!!  In January of this year, the cabinet have made a decision to resume the super computer project.

10. 10 2010.2.22 The System Overview of NGS Ultra high-speed/ high-reliable CPU Advanced 45nm process technology 8cores/CPU, 128GFLOPS Error recovery ( ECC, Instruction retry, etc.) High performance/highly reliable network Direct interconnection network by multi-dimensional mesh/torus network Expandability and reliablity System Software Linux OS Fortran, C, and MPI libraries Distributed parallel file system 【 Massively Parallel/Distributed Memory Supercomputer 】 Logical 3-dimensional torus network Courtesy of FUJITSU

11. 11 2010.2.22 Configuration of Compute Nodes Number of nodes > 80k Number of CPUs > 80k Number of cores > 640k Peak Performance > 10PFLOPS Total Memory Capacity > 1PB ( 16GB/node ) Multi-dimensional mesh/torus network Peak bandwidth: 5GB/s x 2 for each direction of logical 3-dimensional torus network Peak bi-sectional bandwidth: > 30TB/s ノード CPU: 128GFLOPS (8 Core) Core SIMD(4FMA) 16GFlops Core SIMD(4FMA) 16GFlops Core SIMD(4FMA) 16GFlops Core SIMD(4FMA) 16GFlops Core SIMD(4FMA) 16GFlops Core SIMD(4FMA) 16GFlops Core SIMD(4FMA) 16GFlops L2$: 5MB 64GB/s Core SIMD(4FMA) 16GFLOPS MEM: 16GB Logical 3-dimensional torus network for programming x y z 5GB/s x 2

12. 12 2010.2.22 The Next-Generation Supercomputer Project FY2008FY2009FY2010FY2011 Computer building Research building FY2007FY2006FY2012 Next-Generation Integrated Nanoscience Simulation Next-Generation Integrated Life Simulation Verification Development, production, and evaluation Tuning and improvement Tuning and improvement Verification Production, installation, and adjustment Production, installation, and adjustment Construction Design Construction Design Prototype and evaluation Detailed design Conceptual design Development, production, and evaluation System Buildings Applications ○Schedule open to users

13. 13 2010.2.22 The categories of users of NGS 1. Strategic Use: MEXT selected 5 strategic fields from national viewpoint. Field 1: Life science/Drug manufacture Field 2: New material/energy creation Field 3: Global change prediction for disaster prevention/mitigation Field 4: Mono-zukuri (Manufacturing technology) Field 5: The origin of matters and the universe 2. General Use: The use for the needs of the researchers in many science and technology fields including industrial use and educational use

14. 14 2010.2.22 Organization for NGS  “ Advanced Computational Science and Technology Center ” (ACSTC) (tentative name) will be organized at NGS.  MEXT selects 5 core organizations that lead research activities in 5 strategic fields ACSTC → Core research center Conducts advanced and basic R&D in computational science Leads cooperation among strategic fields Provides key knowledge to 5 organizations in strategic fields and another research organizations 5 core organizations → Research center in each field Conducts advanced R&D in each field CCS was selected as a core organization for "Field 5: The origin of matters and the universe" particle physics, Astrophysics, nuclear physics Collaboration with KEK and National Observatory