HPC User Forum 9/10/08 Managed by UT-Battelle for the Department of Energy 1 HPC User Forum 9/10/2008 Scott Klasky S. Ethier, S. Hodson, C. Jin, Z. Lin, J. Lofstead, R. Oldfeld, M. Parashar,K. Schwan, A. Shoshani, M. Wolf, Y. Xiao, F. Zheng

HPC User Forum 9/10/08 Managed by UT-Battelle for the Department of Energy 2  GTC  EFFIS  ADIOS.  Workflow.  Dashboard.  Conclusions.

HPC User Forum 9/10/08 Managed by UT-Battelle for the Department of Energy  Compute node (7,832; 35.2 GF/node) –1 socket (AM2/HT1) per node –4 cores per socket (31,328 cores total) –Core CPU: 2.2 GHz AMD Opteron –Memory per core: 2 GB (DDR2-800)  232 service & I/O nodes  Local storage: ~750 TB, 41 GB/s  Interconnect: 3D torus, SeaStar 2.1 NIC  Aggregate memory: 63 TB  Peak performance: 275 TF Compute node (13,888; 73.6 GF/node) –2 sockets per node (F/HT1) –4 cores per socket (111,104 cores total) –Core CPU: 2.3 GHz AMD Opteron –Memory per core: 2 GB (DDR2-800) 256 service & I/O nodes Local storage: ~10 PB, 200+ GB/s Interconnect: 3D torus, SeaStar 2.1 NIC Aggregate memory: 222 TB Peak performance: 1.0 PF 150 cabinets, 3400 ft MW power.

HPC User Forum 9/10/08 Managed by UT-Battelle for the Department of Energy 4 Big Simulations for early 2008: GTC Science Goals and Impact Science Goals  Use GTC (classic) to analyze cascades and propagation in Collisionless Trapped Electron Mode (CTEM) turbulence –Resolve the critical question of ρ* scaling of confinement in large tokamaks such as ITER; what are consequences of departure from this scaling? –Avalanches and turbulence spreading tend to break Gyro-Bohm scaling but zonal flows tend to restore it by shearing apart extended eddies: a competition  Use GTC-S (shaped) to study electron temperature gradient (ETG) drift turbulence & compare against NSTX experiments –NSTX has a spherical torus with a very low major to minor radius aspect ratio and a strongly-shaped cross-section –NSTX exps have produced very interesting high frequency short wavelength modes - are these kinetic electron modes? –ETG is a likely candidate but only a fully nonlinear kinetic simulation with the exact shape & exp profiles can address this Science Impact  Further the understanding of CTEM turbulence by validation against modulated ECH heat pulse propagation studies on the DIII-D, JET & Tore Supra tokamaks –Is CTEM the key mechanism for electron thermal transport? –Electron temperature fluctuation measurements will shed light –Understand the role of nonlinear dynamics of precession drift resonance in CTEM turbulence  First-time for direct comparison between simulation & experiment on ETG drift turbulence –GTC-S possesses right geometry and right nonlinear physics to possibly resolve this –Help to pinpoint micro-turbulence activities responsible for energy loss through the electron channel in NSTX plasmas

HPC User Forum 9/10/08 Managed by UT-Battelle for the Department of Energy 5 GTC Early Application: electron microturbulence in Fusion Plasma “Scientific Discovery” - Transition to favorable scaling of confinement for both ions and electrons now observed in simulations for ITER plasmas Electron transport less understood but more important in ITER since fusion products first heat the electrons Simulation of electron turbulence is more demanding due to shorter time scales and smaller spatial scales Recent GTC simulation of electron turbulence used 28,000 cores for 42 hours in a dedicated run on Jaguar at ORNL producing 60 TB of data currently being analyzed. This run pushes 15 billion particles for 4800 major time cycles Good news for ITER! Good news for ITER! Ion transport Electron transport

HPC User Forum 9/10/08 Managed by UT-Battelle for the Department of Energy 6  3D fluid data analysis provides critical information to characterize microturbulence, such as radial eddy size, eddy auto-correlation time  Flux Surface Electrostatic Potential demonstrates a ballooning structure  Radial Turbulence eddies have average size ~ 5 ion gyroradius

HPC User Forum 9/10/08 Managed by UT-Battelle for the Department of Energy 7  From SDM center* –Workflow engine – Kepler –Provenance support –Wide-area data movement  From universities –Code coupling (Rutgers) –Visualization (Rutgers)  Newly developed technologies –Adaptable I/O (ADIOS) (with Georgia Tech) –Dashboard (with SDM center) Visualization Code Coupling Wide-area Data Movement Dashboard Workflow Adaptable I/O Provenance and Metadata Foundation Technologies Enabling Technologies Approach: place highly annotated, fast, easy-to-use I/O methods in the code, which can be monitored and controlled, have a workflow engine record all of the information, visualize this on a dashboard, move desired data to user’s site, and have everything reported to a database.

HPC User Forum 9/10/08 Managed by UT-Battelle for the Department of Energy 8  GTC  EFFIS  ADIOS.  Conclusions.

HPC User Forum 9/10/08 Managed by UT-Battelle for the Department of Energy 9  “Those fine fort.* files!”  Multiple HPC architectures –BlueGene, Cray, IB-based clusters  Multiple Parallel Filesystems –Lustre, PVFS2, GPFS, Panasas, PNFS  Many different APIs –MPI-IO, POSIX, HDF5, netCDF –GTC (fusion) has changed IO routines 8 times so far based on performance when moving to different platforms.  Different IO patterns –Restarts, analysis, diagnostics –Different combinations provide different levels of IO performance  Compensate for inefficiencies in the current IO infrastructures to improve overall performance

HPC User Forum 9/10/08 Managed by UT-Battelle for the Department of Energy 10  Allows plug-ins for different I/O implementations.  Abstracts the API from the method used for I/O.  Simple API, almost as easy as F90 write statement.  Best practices/optimize IO routines for all supported transports “for free”  Componentization.  Thin API  XML file –data groupings with annotation –IO method selection –buffer sizes  Common tools –Buffering –Scheduling  Pluggable IO routines External Metadata (XML file) Scientific Codes ADIOS API MPI-CIOLIVE/DataTapMPI-IOPOSIX IOpHDF-5pnetCDFViz EnginesOthers (plug-in) bufferingschedulefeedback

HPC User Forum 9/10/08 Managed by UT-Battelle for the Department of Energy 11  Simple API very similar to standard Fortran or C POSIX IO calls. –As close to identical as possible for C and Fortran API –open, read/write, close is the core –set_path, end_iteration, begin/end_computation, init/finalize are the auxiliaries  No changes in the API for different transport methods.  Metadata and configuration defined in an external XML file parsed once on startup. –Describe the various IO grouping including attributes and hierarchical path structures for elements as an adios-group –Define the transport method used for each adios-group and give parameters for communication/writing/reading –Change on a per element basis what is written –Change on a per adios-group basis how the IO is handled

HPC User Forum 9/10/08 Managed by UT-Battelle for the Department of Energy 12 ADIOS is an IO componentization, which allows us to –Abstract the API from the IO implementation. –Switch from synchronous to asynchronous IO at runtime. –Change from real-time visualization to fast IO at runtime. Combines. –Fast I/O routines. –Easy to use. –Scalable architecture (100s cores) millions of procs. –QoS. –Metadata rich output. –Visualization applied during simulations. –Analysis, compression techniques applied during simulations. –Provenance tracking.

HPC User Forum 9/10/08 Managed by UT-Battelle for the Department of Energy 13  ADIOS Fortran and C based API almost as simple as standard POSIX IO  External configuration to describe metadata and control IO settings  Take advantage of existing IO techniques (no new native IO methods) Fast, simple-to-write, efficient IO for multiple platforms without changing the source code

HPC User Forum 9/10/08 Managed by UT-Battelle for the Department of Energy 14  Data groupings –logical groups of related items written at the same time.  Not necessarily one group per writing event  IO Methods –Choose what works best for each grouping –Vetted, improved, and/or written by experts for each  POSIX (Wei-keng Liao, Northwestern)  MPI-IO (Steve Hodson, ORNL)  MPI-IO Collective (Wei-keng Liao, Northwestern)  NULL (Jay Lofstead, GT)  Ga Tech DataTap Asynchronous (Hasan Abbasi, GT)  phdf5  others.. (pnetcdf on the way).

HPC User Forum 9/10/08 Managed by UT-Battelle for the Department of Energy 15  Specialty APIs –HDF-5 – complex API –Parallel netCDF – no structure  File system aware middleware –MPI ADIO layer – File system connection, complex API  Parallel File systems –Lustre – Metadata server issues –PVFS2 – client complexity –LWFS – client complexity –GPFS, pNFS, Panasas – may have other issues

HPC User Forum 9/10/08 Managed by UT-Battelle for the Department of Energy 16  Platforms tested –Cray CNL (ORNL Jaguar) –Cray Catamount (SNL Redstorm) –Linux Infiniband/Gigabit (ORNL Ewok) –BlueGene P now being tested/debugged. –Looking for future OSX support.  Native IO Methods –MPI-IO independent, MPI-IO collective, POSIX, NULL, Ga Tech DataTap asynchronous, Rutgers DART asynchronous, Posix-NxM, phdf5, pnetcdf, kepler-db

HPC User Forum 9/10/08 Managed by UT-Battelle for the Department of Energy 17  MPI-IO method. –GTC and GTS codes have achieved over 20 GB/sec on Cray XT at ORNL.  30GB diagnostic files every 3 minutes, 1.2 TB restart files every 30 minutes, 300MB other diagnostic files every 3 minutes.  DART: <2% overhead for writing 2 TB/hour with XGC code.  DataTap vs. Posix –1 file per process (Posix). –5 secs for GTC computation. –~25 seconds for Posix IO –~4 seconds with DataTap

HPC User Forum 9/10/08 Managed by UT-Battelle for the Department of Energy 18  June 7, 2008: 24 hour GTC run on Jaguar at ORNL –93% of machine (28,672 cores) –MPI-OpenMP mixed model on quad-core nodes (7168 MPI procs) –three interruptions total (simple node failure) with hour runs –Wrote 65 TB of data at >20 GB/sec (25 TB for post analysis) –IO overhead ~3% of wall clock time. –Mixed IO methods of synchronous MPI-IO and POSIX IO configured in the XML file

HPC User Forum 9/10/08 Managed by UT-Battelle for the Department of Energy 19 Chimera IO Performance (Supernova code) 2x scaling Plot minimum value from 5 runs with 9 restarts/run Error bars show maximum time for the method.

HPC User Forum 9/10/08 Managed by UT-Battelle for the Department of Energy 20 Chimera Benchmark Results  Why ADIOS is better than pHDF5? ADIOS_MPI_IO vs. pHDF5 w/ MPI Indep. IO driver ADIOS_MPI_IO Function# of callsTime write MPI_File_open MPI_Recv buffer_write fopen bp_calsize_stringtag other--~40 pHDF5 Function# of callsTime write MPI_Bcast(sync) MPI_File_open MPI_File_set_size MPI_Comm_dup H5P,H5D,etc other--~20 Use 512 cores, 5 restart dumps. Conversion time on 1 processor for the 2048 core job = 3.6s (read) + 5.6s (write) (other) = 18.8 s Number above are sum among all PEs (parallelism not shown)

HPC User Forum 9/10/08 Managed by UT-Battelle for the Department of Energy 21  A research transport to study asynchronous data movement  Uses server directed I/O to maintain high bandwidth, low overhead for data extraction  I/O scheduling is performed to the perturbation caused by asynchronous I/O

HPC User Forum 9/10/08 Managed by UT-Battelle for the Department of Energy 22  Due to perturbations caused by asynchronous I/O, the overall performance of the application may actually get worse  We schedule the data movement using application state information to prevent asynchronous I/O from interfering with MPI communication  800 GB of data. – Schedule I/O takes 2x longer to move data. Overhead is 2x less.

HPC User Forum 9/10/08 Managed by UT-Battelle for the Department of Energy 23 XML configuration file: … Fortan90 code: ! initialize the system loading the configuration file adios_init (“config.xml”, err) ! open a write path for that type adios_open (h1, “output”, “restart.n1”, “w”, err) adios_group_size (h1, size, total_size, comm, err) ! write the data items adios_write (h1, “g_NX”, 1000, err) adios_write (h1, “g_NY”, 800, err) adios_write (h1, “lo_x”, x_offset, err) adios_write (h1, “lo_y”, y_offset, err) adios_write (h1, “l_NX”, x_size, err) adios_write (h1, “l_NY”, y_size, err) adios_write (h1, “temperature”, u, err) ! commit the writes for asynchronous transmission adios_close (h1, err) … ! do more work ! shutdown the system at the end of my run adios_finalize (mype, err)

HPC User Forum 9/10/08 Managed by UT-Battelle for the Department of Energy 24 C code: // parse the XML file and determine buffer sizes adios_init (“config.xml”); // open and write the retrieved type adios_open (&h1, “restart”, “restart.n1”, “w”); adios_group_size (h1, size, &total_size, comm); adios_write (h1, “n”, n); // int n; adios_write (h1, “mi”, mi); // int mi; adios_write (h1, “zion”, zion); // float zion [10][20][30][40]; // write more variables... // commit the writes for synchronous transmission or // generally initiate the write for asynchronous transmission adios_close (h1); // do more work... // shutdown the system at the end of my run adios_finalize (mype); XML configuration file: … … srv=ewok001.ccs.ornl.gov

HPC User Forum 9/10/08 Managed by UT-Battelle for the Department of Energy 25  Petascale GTC runs will produce 1PB per simulation.  Couple GTC with Edge core (core-edge coupling). –4 PB of data per run. –Can’t store all of GTC runs at ORNL unless we go to tape. ( 12 days to grab data from tape if we get 1GB/sec). –1.5 FTE looking at the the data. –Need more ‘real-time’ analysis of data. –Workflows, data-in-transit (IO graphs), …?  Can we create a staging area with “fat-nodes” –Move data from computational nodes to fat nodes using network of HPC resource. –Reduce data on fat-nodes. –Allow users to “plug-in” analysis routines on “fat-nodes” –How Fat?  Shared memory helps (don’t have to paralyze parallelize-all analysis codes.  Typical upper bound of codes we studied write 1/20 th of memory/core for analysis. Want 1/20 th of resources (5% overhead). Need 2x memory per core for analysis (2x overhead for memory we need (in data + out data).  On Cray at ORNL this means we will have roughly 750 sockets (quad core) for fat memory with shared memory of 34 GB of shared memory.  Also useful for codes which require memory but not as many nodes.  Can we have shared memory on this portion?  What are the other solutions?

HPC User Forum 9/10/08 Managed by UT-Battelle for the Department of Energy 26  GTC is a code which is scaling to the petascale computers.  GBP, Cray XT.  New changes are new science and new IO (ADIOS).  Major challenge in the future is speeding up the data analysis.  ADIOS is an IO componentization. –ADIOS is being integrated integrated into Kepler. –Achieved over 50% peak IO performance for several codes on Jaguar. –Can change IO implementations at runtime. –Metadata is contained in XML file. –Petascale science starts with petascale applications. –Need enabling technologies to scale. –Need to rethink ways to do science.