1. Possible foreseeable measures for tera-scale data handling Kazutoshi Horiuchi *1 Keiko Takahashi *1 Hirofumi Sakuma *1 Shigemune Kitawaki *2 *1 Frontier Research System for Global Change *2 Earth Simulator Research and Development Center

2. Global Change Prediction by an Integrated Three-in-one Research Observation Numerical Simulation Process Study & Modeling Accurate & spatially representative data Optimal monitoring plan Sophisticated high resolution model High performance computing Assimilation data for validation Accurate & spatially representative data ESRDC FRSGC FORSGC

3. Frontier Research System for Global Change (Project) Funding Bodies Japan Marine Science and Technology Center (JAMSTEC) National Space Development Agency (NASDA) Activities Process Study Model Development (Common) Goal Global Change Prediction

4. On-Going Process Studies §Climate Variations Research §Hydrological Cycle Research §Global Warming Research §Atmospheric Composition Research §Ecosystem Change Research §Research of International Pacific Research Center §Research of International Arctic Research Center

5. Current Target of Model Development Group Coupled Model (now based on CCSR/NIES, MOM3) for Climate Change Experiment Cloud Cluster Resolving Ultra High Resolution Model for Prediction of Typhoon/Baiu Evolution Coupled Chemistry - Global Climate Model for Prediction of Atmospheric Composition Change Next Generation Model (Cubic/Icosahedral Grid, CIP method) 4DVAR Ocean Data Assimilation Model (based on MOM3)

6. Current Target of Coupled Model Development on ES §High Resolution: l Atmosphere Model part: T213L50 l Ocean Model part: 1/10 deg. 53 layers §High Performance Estimation of Acceleration ratio Atmosphere Model: (under estimation) Ocean Model: 300 ~ 400 times (480PEs;60Ns) 5 days for 100 years integration

7. Earth Simulator Research Development Center (Project) Funding Bodies Japan Atomic Energy Research Institute (JAERI) National Space Development Agency (NASDA) Japan Marine Science and Technology Center (JAMSTEC) Activities Development of High Speed Parallel Computer Understanding and Prediction of Global Change (Common) Goal Global Change Prediction

8. Characteristics of Earth Simulator §Peak Performance: 40TFLOPS §Number of Processor Nodes: 640 §Number of PEs: 5120 (8PEs/Node) §Interconnection Network: 16GB/s §Total Memory: 10TB §Total Secondary Storage: 600-700TB §Total Mass Storage: 1PB (84Drives)

9. 16 Nodes Architecture of Earth Simulator 16 Nodes Interconnection Network TSS Cluster *1Batch Cluster *39 … Fiber Channel Switch … MM VP 0 VP 1VP 7 16 Nodes … MM VP 0 VP 1VP 7 … MM VP 0 VP 1VP 7 … … Fiber Channel Switch … MM VP 0 VP 1VP 7 16 Nodes … MM VP 0 VP 1VP 7 … MM VP 0 VP 1VP 7 … Mass Storage System WAN WS FS … 84 Drives …

10. I/O Model in Distributed Memory Parallel Computer PPP … communication PPP … PPP … (Unix) File(Data-Distributed Unix) Files Parallel File

11. Parallel File System on ES To handle distributed data as a logically single file has advantages to develop application softwares and to process post processings. 1 2 3 4 5 6 7 123 456 789 147 258 369 M1M1 M2M2 8 9 File Image P P P Processors Disks Proc no.: Np Dist. Size: Sd Dist. Pattern: P= BLOCK/CYCLIC Disk no.: Nd Striping Size: Ss Distribution Mechanism D1D1 D2D2 D3D3 P1P1 P2P2 P3P3 123 456 789

12. Support for Parallel File on Several Levels …… Unix FileParallel File Operating System (with PFS) FAL MPI-IO F90 HPF-RTP F90-RTP Library Compiler C User Program PFSUFSPFSUFSPFS Hardware HPF

13. Review of Model Development Flow Improvement of Model Execution with Model Evaluation of Results Input Data Output Data Resources for Process Study Results of Process Study Analysis Visualization

14. Is it Satisfactory about I/O performance? §The faster super computers are, the larger the amount of the output data generated by large-scale simulations. §The large amount of data is stored to secondary storages and/or mass storages whose devices are slower. §Is it satisfactory about I/O performance ?

15. Amount of Input/Output Data - Coupled Model §To answer the question, the following cases are investigated. l Case I: 1000 Years Integration for the Prediction of Global Warming, Decadal Variability, etc. l Case II: 50 Years Integration for the Analysis of El nino, Dipole Mode Events, Asian Monsoon, etc.

16. Amount of Input/Output Data - Coupled Model §Atmosphere Model part NOTE: The amount of output data is estimated as 2 byte integer elements §Ocean Model part

17. Estimated I/O time - Coupled Model NOTE: Time is estimated only on drive’s I/O rate. Multiple drives are assumed to be independent.

18. Summary of I/O Performance (from the viewpoint of Model Development) §Disk I/O time might be satisfactory. l 0.2% of the simulation time l Less than 2 hours as a total §Tape I/O time might be conspicuous. l 11-35% of the simulation time for 8 tape drives l 1-6 days for 8 tape drives This inefficiency might be critical for iterative works such as model development

19. How to Shorten Turn Around Time of Model Development §Give up outputting numerical data. §Output necessary minimum data. §Output full data, with executing tape I/O and simulations in parallel, and with tape I/O library being able to extract necessary minimum data for post processing.

20. Shortening of TAT by Giving up Outputting Numerical Data － 3.0 1.9 67.5 68.1 68.0 － 0.98 0.79 Elapsed time (h) Time increase (%) Elapsed time (h) Time increase (%) CFD solver only + fixed camera + moving camera 1.6M grid (169x92x101)6.2M grid (337x183x101) 14.5 14.7 14.9 Elapsed time for the concurrent visualization with RVSLIB in the batch processing mode on SX-4 *The number of computational time steps was 10000. *Contour and tracers were displayed at every 10 time steps and visualized animation was stored in a file. *Time integration for moving the tracers was done at every time step for greater accuracy. * This result was provided by NEC

21. Shortening of TAT by Outputting Necessary Minimum Data §“Browse sets”, into which the large amount of output data is abstracted (spatially and/or temporally) within simulations, should be stored. §Specific regions of output data should be stored. This may be Know-Hows of using ES

22. Shortening of TAT by Enhancement of Tape I/O Library for Full Output Data §Tape I/O should be executed with simulations in parallel. §In mass storages, output data should be re- organized, and small subsets which are needed for post processing should be able to be extracted. This may be requirements for the improvement of ES

23. Conclusion §I/O performance was roughly estimated and I/O problem was apprehended. §I/O problem would be avoided with concurrent visualization and/or know-how of the usage. §However we would like to examine the efficient technique for handling the large amount of data continuously to realize comfortable environment for global change prediction.