Parallel Computational Steering and Analysis for HPC Applications using a ParaView Interface and the HDF5 DSM Virtual File Driver J. Biddiscombe, J. Soumagne, G. Oger, D. Guibert, J.G. Piccinali Parallel Computational Steering and Analysis for HPC Applications using a ParaView InterfaceMonday, 11 April 20111

Project/Funding Monday, 11 April 20112Parallel Computational Steering and Analysis for HPC Applications using a ParaView Interface NextMuSE ( Next generation Multi-mechanics Simulation Environment European Community’s Seventh Framework Programme (FP7/ ) Information and Communication Technologies (ICT) Small or medium-scale focused research project - Specific Targeted Research Project (STREP) Future and Emerging Technologies Open Scheme (FET-Open)

Project : Naval Partners (branching out) Monday, 11 April 20113Parallel Computational Steering and Analysis for HPC Applications using a ParaView Interface

leaflets hinges closed open  25 mm Project : Mechanical heart valves Reynolds number  6000 Design static since early 1980s Monday, 11 April 20114Parallel Computational Steering and Analysis for HPC Applications using a ParaView Interface

Project : TurboMachinery Partners Monday, 11 April 20115Parallel Computational Steering and Analysis for HPC Applications using a ParaView Interface

Requirements  In-Situ Visualization  Computational Steering  Two way exchange of potentially large data (simulation has no meshing capability)  Multiple codes/languages  Treat Cluster/Supercomputer like workstation extension Constraints  Partners who are far away ... and who can’t program their way out of a paper bag (because they’re physicists really) Monday, 11 April 20116Parallel Computational Steering and Analysis for HPC Applications using a ParaView Interface

Ideal Solution Sim Node Vis Code Sim Node M Nodes Vis Code Visit/ParaView Use this approach for in- situ visualization Memory pressures (especially for time- dependent data) If Vis/Analysis code doesn’t scale as well as simulation code, possible problem Render Server Data Server Monday, 11 April 20117Parallel Computational Steering and Analysis for HPC Applications using a ParaView Interface

Parallel IO (c.f. Lustre) Sim Node OSS Node Sim Node M NodesN OSS Nodes All IO goes through OSS nodes HDF5 Model already looks like this Monday, 11 April 20118Parallel Computational Steering and Analysis for HPC Applications using a ParaView Interface

DSM Implementation Sim Node Vis Node DSM Sim Node M NodesN Nodes DSM HDF5/MPI HDF IO intercepted and routed to DSM HDF5/MPI DSM Model could look like this Just replace the IO layer in HDF5 IO Forwarding c.f. ADIOS etc IO Forwarding c.f. ADIOS etc Monday, 11 April 20119Parallel Computational Steering and Analysis for HPC Applications using a ParaView Interface

Analysis/Vis Machine Simulation Machine (Infiniband) Switch Simulation Nodes Analysis/Vis Nodes DSM DSM may be on either side Render Workstation GUI GUI may be separate Workstation DSM Simulation Machine DSM GUI Render GUI Render Analysis/Vis Workstation GUI Render Monday, 11 April Parallel Computational Steering and Analysis for HPC Applications using a ParaView Interface Configurations

H5FDdsm Bandwidth J. Soumagne, J. Biddiscombe, “Computational Steering and Parallel Online Monitoring Using RMA through the HDF5 DSM Virtual File Driver”, International Conference on Computational Science, ICCS 2011 J. Soumagne, J. Biddiscombe, J. Clarke, “An HDF5 MPI virtual file driver for parallel in-situ post-processing”, EuroMPI 2010, Sep vol of Lecture Notes in Computer Science. Springer Berlin/Heidelberg, 2010, pp. 62–71 Talk to Jerome for further info on numbers Monday, 11 April Parallel Computational Steering and Analysis for HPC Applications using a ParaView Interface

 Our driver: H5FDdsm  Data sent into HDF5 is then automatically redirected to this driver  Currently uses flat memory model (may be improved by scattering) more links = more BW HDF5 – Virtual File Drivers Monday, 11 April Parallel Computational Steering and Analysis for HPC Applications using a ParaView Interface … start end of file (eof) eof + metadata = total length of DSM eof + metadata = total length of DSM Network network layer = plugin TCP / MPI / MPI_RMA (Gemini/DMAPP) network layer = plugin TCP / MPI / MPI_RMA (Gemini/DMAPP)

Performance – Write test with a 20GB DSM distributed among 8 post-processing nodes on a Cray XT5 Monday, 11 April Parallel Computational Steering and Analysis for HPC Applications using a ParaView Interface Saturation of the network

Performance – Steering overhead Monday, 11 April Parallel Computational Steering and Analysis for HPC Applications using a ParaView Interface

Current developments  Scalability on large systems  Dynamic MPI communication not supported  Use MPMD job instead of two separate SPMD jobs  Additional communicator designed for Gemini systems  RMA here does not perform so well  DMAPP communicator (reachable bandwidth of 5GB/s between nodes) Monday, 11 April Parallel Computational Steering and Analysis for HPC Applications using a ParaView Interface

In-Situ Visualization – How to use the driver Create a file access property list Instead of using the MPI-IO driver Insert our DSM driver Then carry on as usual Comment out any MPIO_COLLECTIVE dataset transfer properties Monday, 11 April Parallel Computational Steering and Analysis for HPC Applications using a ParaView Interface H5Pset_fapl_dsm(fapl_id, MPI_COMM_WORLD, NULL); H5Pset_fapl_mpio(fapl_id, MPI_COMM_WORLD, MPI_INFO_NULL); file_id = H5Fcreate(filename, H5F_ACC_TRUNC, H5P_DEFAULT, acc_plist_id); H5Pclose(acc_plist_id); H5Fclose(file_id); fapl_id = H5Pcreate(H5P_FILE_ACCESS);

In-Situ Visualization – Create Template Monday, 11 April Parallel Computational Steering and Analysis for HPC Applications using a ParaView Interface  Built on top of Xdmf model and format  Generate Xdmf files to drive Xdmf reader in plugin  Uses in-memory H5dump to get the missing (metadata) information  Much simpler to write  Allows variable number of elements /fluid/position /fluid/density /fluid/velocity /fluid/vmob /fluid/w

In-Situ Visualization – Turbine Monday, 11 April Parallel Computational Steering and Analysis for HPC Applications using a ParaView Interface

Selectable Arrays ParaView doesn’t have ‘contracts’, so from the template, we can generate on/off switches for each exported dataset/array Simulation can check if an array is ‘enabled’ and not send it. (better than turning it off in the XdmfReader) We don’t use HDF5 for this, just a metadata flag maintained by the H5FDdsm library. Monday, 11 April Parallel Computational Steering and Analysis for HPC Applications using a ParaView Interface H5FD_dsm_steering_is_enabled(const char *name); // /grid/group/name

Note on Steering Architecture Monday, 11 April Parallel Computational Steering and Analysis for HPC Applications using a ParaView Interface ParaView Servers … Set Steering Commands and Disabled Objects in metadata section Read simulation data – write user defined steering commands and data into HDF5 “file” section Parallel Simulation Write output data – Read user defined steering commands and data Get Steering Commands and Disabled Objects

Computational Steering Each simulation requires custom controls And different analysis requirements Monday, 11 April Parallel Computational Steering and Analysis for HPC Applications using a ParaView Interface Heart Valve Sloshing Ship Turbine Deforming Hull

Automatic GUI Generation : Interactions XML <DataExportProperty name="ModifiedBodyNodes" command="SetSteeringArray" label="Modified Body Node Data"> <DataExportDomain name="data_export" full_path="/Mesh_DataSet" geometry_path="/Mesh_Nodes#1/NewXYZ" topology_path="/Mesh_Nodes#1/NewCo..." command_property="ReloadFreeBodyMesh"> Monday, 11 April Parallel Computational Steering and Analysis for HPC Applications using a ParaView Interface Reuse ServerManager syntax from ParaView Added some extra Properties Can also link GUI 3D widgets using ‘hints’

No ‘app-specific’ knowledge required Generate ServerManager XML from template parse hints/commands and other stuff Register it internally on the fly No need to (re)compile paraview plugin Unless you have some custom heart valve adjustment widget that is specially designed for that app Engineer only modifies simulation ‘Names’ and XML template Monday, 11 April Parallel Computational Steering and Analysis for HPC Applications using a ParaView Interface

H5FDdsm Steering API  Initialize the steering extension (buffer is optional)  Get steering commands and disabled objects  Test if “name” is a disabled object  Get/Set user defined scalar parameter  Get/Set user defined vector parameter Monday, 11 April Parallel Computational Steering and Analysis for HPC Applications using a ParaView Interface H5FD_dsm_steering_init( MPI_Comm comm, void *buffer); H5FD_dsm_steering_update(); H5FD_dsm_steering_is_enabled(const char *name); H5FD_dsm_steering_scalar_get(const char *name, hid_t mem_type, void *data); H5FD_dsm_steering_vector_get(const char *name, hid_t mem_type, hsize_t number_of_elements, void *data);

H5FDdsm Steering API  Test if a user defined parameter is set  Wait for an incoming “play” command  Begin/End query and avoid multiple metadata accesses  Get/Free dataset handle for data specific parallel read Monday, 11 April Parallel Computational Steering and Analysis for HPC Applications using a ParaView Interface H5FD_dsm_steering_is_set(const char *name, int *set); H5FD_dsm_steering_begin_query(); H5FD_dsm_steering_get_handle(const char *name, hid_t *handle); H5FD_dsm_steering_free_handle(hid_t handle); H5FD_dsm_steering_wait();

Can switch between modes arbitrarily (or every N iterations) Can switch between modes arbitrarily (or every N iterations) Calculation Write Initialization Analysis + GUI Interaction Analysis + GUI Interaction Calculation Wait mode Analysis + GUI Interaction Analysis + GUI Interaction File Wait mode Wait Mode Monday, 11 April Parallel Computational Steering and Analysis for HPC Applications using a ParaView Interface Computation/Analysis not overlapped Define some pipeline in the ParaView GUI which does something interesting (to send data back) Define some pipeline in the ParaView GUI which does something interesting (to send data back) H5FD_dsm_steering_wait();

GUI Write Initialization Analysis File Calculation Analysis Free Mode Monday, 11 April Parallel Computational Steering and Analysis for HPC Applications using a ParaView Interface Computation/Analysis overlapped Can switch between modes arbitrarily (or every N iterations) Can switch between modes arbitrarily (or every N iterations) File Create wipes DSM clean So may want to add more sync options File Create wipes DSM clean So may want to add more sync options Can store N time steps

Steering demo (recorded) Monday, 11 April Parallel Computational Steering and Analysis for HPC Applications using a ParaView Interface

Monday, 11 April Parallel Computational Steering and Analysis for HPC Applications using a ParaView Interface Steering demo (recorded)

Animated Wedge Monday, 11 April Parallel Computational Steering and Analysis for HPC Applications using a ParaView Interface

Conclusion  Minimally intrusive way to steer applications  Supports Parallel for Sim and Vis/Analysis  Allows remeshing/exchange of large data Can do stuff that we couldn’t do before! Further work  Now developing new MPI mode for large core counts  DMAPP/Gemini layer  Turbine mesh next target for manipulation Monday, 11 April Parallel Computational Steering and Analysis for HPC Applications using a ParaView Interface

Software available on HPCforge Monday, 11 April Parallel Computational Steering and Analysis for HPC Applications using a ParaView Interface H5FDdsm:  ICARUS: (ParaView plugin)  XdmfGenerator:  Questions :