1. Parallel Computational Steering and Analysis for HPC Applications using a ParaView Interface and the HDF5 DSM Virtual File Driver J. Biddiscombe, J. Soumagne, G. Oger, D. Guibert, J.G. Piccinali Parallel Computational Steering and Analysis for HPC Applications using a ParaView InterfaceMonday, 11 April 20111

2. Project/Funding Monday, 11 April 20112Parallel Computational Steering and Analysis for HPC Applications using a ParaView Interface NextMuSE (http://nextmuse.cscs.ch)http://nextmuse.cscs.ch Next generation Multi-mechanics Simulation Environment European Community’s Seventh Framework Programme (FP7/2007-2013) Information and Communication Technologies (ICT) Small or medium-scale focused research project - Specific Targeted Research Project (STREP) Future and Emerging Technologies Open Scheme (FET-Open)

3. Project : Naval Partners (branching out) Monday, 11 April 20113Parallel Computational Steering and Analysis for HPC Applications using a ParaView Interface

4. leaflets hinges closed open  25 mm Project : Mechanical heart valves Reynolds number  6000 Design static since early 1980s Monday, 11 April 20114Parallel Computational Steering and Analysis for HPC Applications using a ParaView Interface

5. Project : TurboMachinery Partners Monday, 11 April 20115Parallel Computational Steering and Analysis for HPC Applications using a ParaView Interface

6. Requirements  In-Situ Visualization  Computational Steering  Two way exchange of potentially large data (simulation has no meshing capability)  Multiple codes/languages  Treat Cluster/Supercomputer like workstation extension Constraints  Partners who are far away ... and who can’t program their way out of a paper bag (because they’re physicists really) Monday, 11 April 20116Parallel Computational Steering and Analysis for HPC Applications using a ParaView Interface

7. Ideal Solution Sim Node Vis Code Sim Node M Nodes Vis Code Visit/ParaView Use this approach for in- situ visualization Memory pressures (especially for time- dependent data) If Vis/Analysis code doesn’t scale as well as simulation code, possible problem Render Server Data Server Monday, 11 April 20117Parallel Computational Steering and Analysis for HPC Applications using a ParaView Interface

8. Parallel IO (c.f. Lustre) Sim Node OSS Node Sim Node M NodesN OSS Nodes All IO goes through OSS nodes HDF5 Model already looks like this Monday, 11 April 20118Parallel Computational Steering and Analysis for HPC Applications using a ParaView Interface

9. DSM Implementation Sim Node Vis Node DSM Sim Node M NodesN Nodes DSM HDF5/MPI HDF IO intercepted and routed to DSM HDF5/MPI DSM Model could look like this Just replace the IO layer in HDF5 IO Forwarding c.f. ADIOS etc IO Forwarding c.f. ADIOS etc Monday, 11 April 20119Parallel Computational Steering and Analysis for HPC Applications using a ParaView Interface

10. Analysis/Vis Machine Simulation Machine (Infiniband) Switch Simulation Nodes Analysis/Vis Nodes DSM DSM may be on either side Render Workstation GUI GUI may be separate Workstation DSM Simulation Machine DSM GUI Render GUI Render Analysis/Vis Workstation GUI Render 3 3 2 2 2 2 Monday, 11 April 201110Parallel Computational Steering and Analysis for HPC Applications using a ParaView Interface Configurations

11. H5FDdsm Bandwidth J. Soumagne, J. Biddiscombe, “Computational Steering and Parallel Online Monitoring Using RMA through the HDF5 DSM Virtual File Driver”, International Conference on Computational Science, ICCS 2011 J. Soumagne, J. Biddiscombe, J. Clarke, “An HDF5 MPI virtual file driver for parallel in-situ post-processing”, EuroMPI 2010, Sep 2010. vol. 6305 of Lecture Notes in Computer Science. Springer Berlin/Heidelberg, 2010, pp. 62–71 Talk to Jerome for further info on numbers Monday, 11 April 201111Parallel Computational Steering and Analysis for HPC Applications using a ParaView Interface

12.  Our driver: H5FDdsm  Data sent into HDF5 is then automatically redirected to this driver  Currently uses flat memory model (may be improved by scattering) more links = more BW HDF5 – Virtual File Drivers Monday, 11 April 201112Parallel Computational Steering and Analysis for HPC Applications using a ParaView Interface … start end of file (eof) eof + metadata = total length of DSM eof + metadata = total length of DSM Network network layer = plugin TCP / MPI / MPI_RMA (Gemini/DMAPP) network layer = plugin TCP / MPI / MPI_RMA (Gemini/DMAPP)

13. Performance – Write test with a 20GB DSM distributed among 8 post-processing nodes on a Cray XT5 Monday, 11 April 201113Parallel Computational Steering and Analysis for HPC Applications using a ParaView Interface Saturation of the network

14. Performance – Steering overhead Monday, 11 April 201114Parallel Computational Steering and Analysis for HPC Applications using a ParaView Interface

15. Current developments  Scalability on large systems  Dynamic MPI communication not supported  Use MPMD job instead of two separate SPMD jobs  Additional communicator designed for Gemini systems  RMA here does not perform so well  DMAPP communicator (reachable bandwidth of 5GB/s between nodes) Monday, 11 April 201115Parallel Computational Steering and Analysis for HPC Applications using a ParaView Interface

16. In-Situ Visualization – How to use the driver Create a file access property list Instead of using the MPI-IO driver Insert our DSM driver Then carry on as usual Comment out any MPIO_COLLECTIVE dataset transfer properties Monday, 11 April 201116Parallel Computational Steering and Analysis for HPC Applications using a ParaView Interface H5Pset_fapl_dsm(fapl_id, MPI_COMM_WORLD, NULL); H5Pset_fapl_mpio(fapl_id, MPI_COMM_WORLD, MPI_INFO_NULL); file_id = H5Fcreate(filename, H5F_ACC_TRUNC, H5P_DEFAULT, acc_plist_id); H5Pclose(acc_plist_id); H5Fclose(file_id); fapl_id = H5Pcreate(H5P_FILE_ACCESS);

17. In-Situ Visualization – Create Template Monday, 11 April 201117Parallel Computational Steering and Analysis for HPC Applications using a ParaView Interface  Built on top of Xdmf model and format  Generate Xdmf files to drive Xdmf reader in plugin  Uses in-memory H5dump to get the missing (metadata) information  Much simpler to write  Allows variable number of elements /fluid/position /fluid/density /fluid/velocity /fluid/vmob /fluid/w

18. In-Situ Visualization – Turbine Monday, 11 April 201118Parallel Computational Steering and Analysis for HPC Applications using a ParaView Interface

19. Selectable Arrays ParaView doesn’t have ‘contracts’, so from the template, we can generate on/off switches for each exported dataset/array Simulation can check if an array is ‘enabled’ and not send it. (better than turning it off in the XdmfReader) We don’t use HDF5 for this, just a metadata flag maintained by the H5FDdsm library. Monday, 11 April 201119Parallel Computational Steering and Analysis for HPC Applications using a ParaView Interface H5FD_dsm_steering_is_enabled(const char *name); // /grid/group/name

20. Note on Steering Architecture Monday, 11 April 201120Parallel Computational Steering and Analysis for HPC Applications using a ParaView Interface ParaView Servers … Set Steering Commands and Disabled Objects in metadata section Read simulation data – write user defined steering commands and data into HDF5 “file” section Parallel Simulation Write output data – Read user defined steering commands and data Get Steering Commands and Disabled Objects

21. Computational Steering Each simulation requires custom controls And different analysis requirements Monday, 11 April 201121Parallel Computational Steering and Analysis for HPC Applications using a ParaView Interface Heart Valve Sloshing Ship Turbine Deforming Hull

22. Automatic GUI Generation : Interactions XML <DataExportProperty name="ModifiedBodyNodes" command="SetSteeringArray" label="Modified Body Node Data"> <DataExportDomain name="data_export" full_path="/Mesh_DataSet" geometry_path="/Mesh_Nodes#1/NewXYZ" topology_path="/Mesh_Nodes#1/NewCo..." command_property="ReloadFreeBodyMesh"> Monday, 11 April 201122Parallel Computational Steering and Analysis for HPC Applications using a ParaView Interface Reuse ServerManager syntax from ParaView Added some extra Properties Can also link GUI 3D widgets using ‘hints’

23. No ‘app-specific’ knowledge required Generate ServerManager XML from template parse hints/commands and other stuff Register it internally on the fly No need to (re)compile paraview plugin Unless you have some custom heart valve adjustment widget that is specially designed for that app Engineer only modifies simulation ‘Names’ and XML template Monday, 11 April 201123Parallel Computational Steering and Analysis for HPC Applications using a ParaView Interface

24. H5FDdsm Steering API  Initialize the steering extension (buffer is optional)  Get steering commands and disabled objects  Test if “name” is a disabled object  Get/Set user defined scalar parameter  Get/Set user defined vector parameter Monday, 11 April 201124Parallel Computational Steering and Analysis for HPC Applications using a ParaView Interface H5FD_dsm_steering_init( MPI_Comm comm, void *buffer); H5FD_dsm_steering_update(); H5FD_dsm_steering_is_enabled(const char *name); H5FD_dsm_steering_scalar_get(const char *name, hid_t mem_type, void *data); H5FD_dsm_steering_vector_get(const char *name, hid_t mem_type, hsize_t number_of_elements, void *data);

25. H5FDdsm Steering API  Test if a user defined parameter is set  Wait for an incoming “play” command  Begin/End query and avoid multiple metadata accesses  Get/Free dataset handle for data specific parallel read Monday, 11 April 201125Parallel Computational Steering and Analysis for HPC Applications using a ParaView Interface H5FD_dsm_steering_is_set(const char *name, int *set); H5FD_dsm_steering_begin_query(); H5FD_dsm_steering_get_handle(const char *name, hid_t *handle); H5FD_dsm_steering_free_handle(hid_t handle); H5FD_dsm_steering_wait();

26. Can switch between modes arbitrarily (or every N iterations) Can switch between modes arbitrarily (or every N iterations) Calculation Write Initialization Analysis + GUI Interaction Analysis + GUI Interaction Calculation Wait mode Analysis + GUI Interaction Analysis + GUI Interaction File Wait mode Wait Mode Monday, 11 April 201126Parallel Computational Steering and Analysis for HPC Applications using a ParaView Interface Computation/Analysis not overlapped Define some pipeline in the ParaView GUI which does something interesting (to send data back) Define some pipeline in the ParaView GUI which does something interesting (to send data back) H5FD_dsm_steering_wait();

27. GUI Write Initialization Analysis File Calculation Analysis Free Mode Monday, 11 April 201127Parallel Computational Steering and Analysis for HPC Applications using a ParaView Interface Computation/Analysis overlapped Can switch between modes arbitrarily (or every N iterations) Can switch between modes arbitrarily (or every N iterations) File Create wipes DSM clean So may want to add more sync options File Create wipes DSM clean So may want to add more sync options Can store N time steps

28. Steering demo (recorded) Monday, 11 April 201128Parallel Computational Steering and Analysis for HPC Applications using a ParaView Interface

29. Monday, 11 April 201129Parallel Computational Steering and Analysis for HPC Applications using a ParaView Interface Steering demo (recorded)

30. Animated Wedge Monday, 11 April 201130Parallel Computational Steering and Analysis for HPC Applications using a ParaView Interface

31. Conclusion  Minimally intrusive way to steer applications  Supports Parallel for Sim and Vis/Analysis  Allows remeshing/exchange of large data Can do stuff that we couldn’t do before! Further work  Now developing new MPI mode for large core counts  DMAPP/Gemini layer  Turbine mesh next target for manipulation Monday, 11 April 201131Parallel Computational Steering and Analysis for HPC Applications using a ParaView Interface

32. Software available on HPCforge Monday, 11 April 201132Parallel Computational Steering and Analysis for HPC Applications using a ParaView Interface H5FDdsm:  https://hpcforge.org/projects/h5fddsm https://hpcforge.org/projects/h5fddsm ICARUS: (ParaView plugin)  https://hpcforge.org/projects/icarus https://hpcforge.org/projects/icarus XdmfGenerator:  https://hpcforge.org/projects/xdmfgenerator https://hpcforge.org/projects/xdmfgenerator Questions :