HPCMP Benchmarking and Performance Analysis Mark Cowan USACE ERDC ITL in support of DoD HPCMP Tuesday, April 17, 2012

What is the HPCMP? Initiated in 1992 Congressional mandate to modernize DoD’s HPC capabilities Assembled from collection of HPC departments across Army, Air Force, and Navy labs and test centers

What is the HPCMP? FOCUS Solve military and security problems using HPC hardware and software Assess technical and management risks Performance Time Available resources Cost Schedule Supports DoD objectives through research, development, test and evaluation

Where we benchmark

Migrate to a 2-year acquisition cycle Why the radical change? Entice more vendors into the competition Vendor feedback  remove or alleviate disincentives Review the entirety of the TI acquisition process Line-by-line justification of benchmarking rules document Address both HPC community and vendor concerns Comprehensive reevaluation of how we benchmark Analyze the codes Justify the test cases

Migrate to a 2-year acquisition cycle Dangers? Time the milestones poorly on the calendar and miss out on release of cutting-edge technology Difficult problem How to schedule activities to maximize likelihood of hitting publicly-available products months in advance, while being blind to intricacies of chip fabrication schedules and unforeseen recalls?

Codes considered for TI-11/12 ABAQUS COBALT ICEPIC ABINIT CP2K LAMMPS ACES CPMD LS-DYNA ADCIRC CTH MATLAB ADH ETA OOCORE ALE3D FDTD OVERFLOW ALEGRA FLAPW SHAMRC AMR FLUENT SIERRA AVUS GAMESS STAR CCM+ CFD++ GASP VASP CFDSHIP-IOWA GAUSSIAN WRF COAMPS HYCOM XPATCH

TI-11/12 benchmarking applications ADCIRC – Coastal Circulation and Storm Surge model 100% Fortran, MPI Uses METIS library (C) 205K LOC ALEGRA – Hydrodynamic and solid dynamics plus magnetic field and thermal transport 96% C, 4% Fortran, MPI 978K LOC AVUS (Cobalt-60) – Turbulent flow CFD code Fortran, MPI, 29K LOC CTH – Shock physics code ~58% Fortran/~42% C, MPI, 900K LOC GAMESS – Quantum chemistry code Fortran, MPI, 330K LOC HYCOM – Ocean circulation modeling code Fortran, MPI, 31K LOC ICEPIC – Particle-in-cell magnetohydrodynamics code C, MPI, 350K LOC LAMMPS – Molecular dynamics code C++, MPI, 45K LOC █ Predicted █ Benchmarked

Components of testing packages Applications tested on representative input sets CODE CASE Distinguished Core Count Time (sec) on DIAMOND Core Counts ADCIRC baroclinic 1024 8959 512, 768, 1024, 1280, 1536, 1792, 2048 hurricane 1280 2082 ALEGRA obliqueImp 1536 1640 1024, 1280, 1536, 1792, 2048 explWire 256 944 256, 384, 512, 768, 1024 AVUS waverider 941 384, 512, 768, 1024, 1536 turret-td 1332 768, 1024, 1280, 1536, 2048 CTH fixed-grid 3399 amr 2535 GAMESS DFT-grad 4701 128, 192, 256, 384, 512 MP2-grad 512 2536 128, 256, 512, 768, 1024 CC-energy 3658 512, 768, 1024, 1536, 2048 HYCOM lrg 1353 3020 1001, 1353, 1516, 1770, 2045 ICEPIC magnetron 384 2559 256, 384, 512, 768, 1024 gyrotron 2048 3639 1536, 1792, 2048, 2304, 2560 LAMMPS Au 3182 128, 256, 384, 512, 1024, 1280, 1536, 2048

Some components of HPC procurement cycle

Some components of HPC procurement cycle Acquire new versions of codes Port codes to various machines Acquire test cases Develop or acquire accuracy checks Test codes, get times to compare Assemble package for vendors

Some components of HPC procurement cycle Run codes with test cases on installed DSRC machines Optimize! How fast can we go?

Some components of HPC procurement cycle We review vendor submittal Anything suspicious? How do vendor times compare to ours? How did vendors optimize? How risky is vendor’s proposal? Present our results

Components of testing packages continued Timers measure the elapsed running times Accuracy checks ensure validity of output files Often requires determination of acceptable error bounds

How the test packages are used Run all test cases on 5 different DSRC machines to acquire times Debug test packages Quantify variation across/within machines Compare times to proposed systems

Machine attributes DSRC Name Make Model Chip Set Processor Speed (GHz)   Architectures Used in Study DSRC Name Make Model Chip Set Processor Speed (GHz) Interconnect Number of Cores Cores per Node Operating System ERDC Diamond SGI Altix ICE Intel Xeon QC 2.8 DDR4 InfiniBand 15360 8 SUSE Linux MHPCC Mana Dell PowerEdge M610 DDR InfiniBand 9216 Linux NAVY DaVinci IBM Power6 IBM P6 DC 4.7 DDR Infiniband 4800 32 AIX Einstein Cray XT5 Cray Opteron QC 2.3 SeaStar2+ 12736 CNL Garnet XE6 AMD Opteron 64-bit 2.4 Cray Gemini 20224 16 CLE

RESULTS! Graphs of runtimes

Risk Assessment: Major Areas Assessed Compliance assessment Ability to follow benchmark rules Number of test case results provided Results within accuracy criteria Assessment of risk in meeting proposed times in acceptance tests Differences between benchmarked and proposed system Processor , interconnect, and I/O system differences Quality of estimation procedure Quality of explanation and soundness of estimation procedure Aggressiveness of final estimate Comparison with measured benchmark system times Comparison with predicted times Assessment of likelihood of users and/or developers using proposed code modifications Acceptability of proposed code modifications

Benchmarking website URL: http://www.benchmarking.hpc.mil/

Benchmarking website continued Narrative of website purpose, codes tested Heatmap of systems best suited for applications

Brief description of application Brief description of test cases Benchmarking website continued Brief description of application Brief description of test cases

Benchmarking website continued An example of how we made the heatmap for allocation choices

Want to suggest an improvement? Benchmarking website continued Got a question? Want to suggest an improvement? Contact us.

Performance Team Members Mark Cowan – ERDC – Chair Larry Davis – HPCMPO Lloyd Slonaker – AFRL Tim Sell – AFRL Laura Brown – ERDC Mahbubur Rashid – ERDC Christine Cuicchi – NAVO Matt Grismer – AFRL Jerry Boatz – AFRL

Performance Team Advisors William Ward – HPCMPO Steve Finn – DTRA Carrie Leach – ERDC Paul Bennett – ERDC Tom Oppe – ERDC Henry Newman – Instrumental Michael Laurenzano – SDSC Bronis de Supinski – LLNL Joseph Swartz – LM Allan Snavely – SDSC Laura Carrington – SDSC Robert Pennington – NSF Nick Wright – NERSC James Ianni – ARL

Questions?

Contact me… Mark Cowan USACE ERDC ITL Computational Analysis Branch 3909 Halls Ferry Road Building 8000, Room 1255 Vicksburg, MS 39180 (601) 634-2665 Mark.A.Cowan@usace.army.mil

ADDENDA

AVUS: Code description CFD code, formerly COBALT_60 Simulates 3-D turbulent viscous flow over irregular geometries Grid-based, reads a large grid file AVUS: 29K lines of Fortran 90 code Uses ParMETIS: 12K lines of C code Parallelism via MPI, no OpenMP Runs on Cray XT, IBM Power, SGI Altix, Linux clusters

CTH: Code description CTA: CSM (Computational Structural Mechanics) Shock Physics Two-step, 2nd order accurate Eulerian algorithm is used to solve the mass, momentum, and energy conservation equations An explicit approach that does not require solving a linear system Has both static and adaptive mesh capabilities Parallelism via MPI 900K LOC, 58% FORTRAN and 42% C Uses NetCDF, supplied with distribution

GAMESS: Code description CTA: CCM (Computational Chemistry, Biology, and Materials Science) Ab Initio Quantum chemistry Computes many energy integrals with molecular data in form of atom positions and electron orbitals Communication depends on platform LAPI, Sockets, SHMEM, MPI Code composition: 99% FORTRAN, 1% C

HYCOM: Code description CTA: Climate/Weather/Ocean Modeling and Simulation (CWO) A primitive equation ocean general circulation model Communication is MPI (MPI-2 is available) 100% FORTRAN Version 2.2.27

HYCOM: MPI-2 details HYCOM may be run with MPI or MPI-2 MPI-2 is MPI with additional features such as parallel I/O, dynamic process management and remote memory operations HYCOM utilizes parallel I/O feature Parallel I/O times required starting with TI-10

ICEPIC: Code description CTA: Computational Electromagnetics and Acoustics (CEA) Particle-in-cell plasma physics code Ions and electrons move under influence of electromagnetic fields Particles are updated in a grid-free manner; grouped in cells which are periodically adjusted to preserve load balance Fields calculated on a structured, static grid and dual grid according to Maxwell's Equations Can simulate plasmas contained in complex geometries Used in electromagnetic device design ~350K lines of code, 100% C++, C

LAMMPS: Code description CTA: CCM (Comp Chemistry, Biology, & Material Science) Classical molecular dynamics code that models particles in a liquid, solid, or gaseous state Calculates atomic velocities, positions, system energy, and temperature After equilibration: surface tension, radial pressure, and phase change Post-processing: pair-correlation function and diffusion coefficients All actions occur within box (usually orthogonal) Distributed-memory message-passing parallelism (MPI) Highly-portable C++ Libraries needed: MPI and single-processor FFT

ADCIRC: Code description ADCIRC Coastal Circulation and Storm Surge Model Solves time dependent, free surface circulation and transport problems in 2 and 3 dimensions. Use the finite element method in space, which permits highly flexible, unstructured grids. Typical ADCIRC applications have included: Modeling tides and wind driven circulation, Analysis of hurricane storm surge and flooding, Dredging feasibility and material disposal studies, Larval transport studies, and Near shore marine operations

Pressure and temperature during formation of jet from shaped charge “BASE” ALEGRA: Code description ALE code -- Arbitrary Lagrangian-Eulerian -- provides flexibility, accuracy and reduced numerical dissipation over pure Eulerian code; modern remeshing technology allows for robust mesh smoothing and control. Hydrodynamic and solid dynamics Models large distortions and strong shock propagation in multiple-materials Finite element code; descendent of PRONTO and uses some CTH Eulerian technology Energy deposition and explosive burn models Geometry -- 2D/3D Cartesian, 2D cylindrical   Material Models in ALEGRA: Equations of State Elastic-Plastic Models Fracture Models Pressure and temperature during formation of jet from shaped charge

“BASE” ALEGRA: Code description

ALEGRA_MHD: Code description All hydrodynamics/solid dynamic modules of "base" ALEGRA PLUS magnetic field and thermal transport effects Lorentz forces, Joule heating, thermal transport and simple models for radiating excess energy 2D and 3D versions 2D modeling with the magnetic flux density vector components in or out of the plane with the corresponding current density out of or in the plane, respectively. 3D uses a magnetic diffusion solution based on edge and face elements which maintains the discrete flux divergence-free property during magnetic solve and constrained transport remap stage Lumped element coupled circuit equations Magnetic and thermal conduction Advanced models for thermal and electrical conductivity Emission model radiates excess energy when medium is optically thin while accounting for reabsorption