1. EUFORIA FP7-INFRASTRUCTURES-2007-1, Grant 211804 JRA4 Overview and plans M. Haefele, E. Sonnendrücker Euforia kick-off meeting 22 January 2008 Gothenburg

2. EUFORIA FP7-INFRASTRUCTURES-2007-1, Grant 211804 Outline Presentation of JRA4 tasks Task 1: Existing visualization tools Task 2: Visualization & Kepler platform Task 3: Lossy data compression format Task 4: 4D and 5D visualization

3. EUFORIA FP7-INFRASTRUCTURES-2007-1, Grant 211804 Outline Presentation of JRA4 tasks Task 1: Existing visualization tools Task 2: Visualization & Kepler platform Task 3: Lossy data compression format Task 4: 4D and 5D visualization

4. EUFORIA FP7-INFRASTRUCTURES-2007-1, Grant 211804 Purpose Provide visualization support for project Unified visualization framework for all codes to be integrated  Post-processing visualization Integration of visualization tools in Kepler workflow  Monitoring  Steering

5. EUFORIA FP7-INFRASTRUCTURES-2007-1, Grant 211804 Outline Presentation of JRA4 tasks Task 1: Existing visualization tools Task 2: Visualization & Kepler platform Task 3: Lossy data compression format Task 4: 4D and 5D visualization

6. EUFORIA FP7-INFRASTRUCTURES-2007-1, Grant 211804 Python Matplotlib Pure Python library based on numpy Can be used in  python scripts  web application server that can call python Syntax close to Matlab visualization tools => Replace Matlab visualization for free ! Only 2D plots are available 3D plots will use the VTK library

7. EUFORIA FP7-INFRASTRUCTURES-2007-1, Grant 211804 Python matplotlib

8. EUFORIA FP7-INFRASTRUCTURES-2007-1, Grant 211804 VTK : Visualization Toolkit Open source C++ library Community of more than 1000 developers Architecture managed by a private company : Kitware Exists and evolves since 1998 => powerful and stable tool

9. EUFORIA FP7-INFRASTRUCTURES-2007-1, Grant 211804 ParaView Software based on parallel VTK Can be seen as a GUI for VTK The user designs the VTK pipeline by himself  The user must know the different pieces of the pipeline  Great flexibility Parallelization of both processing and rendering Plugin system based on the use of new VTK classes Open source software designed by Los Alamos Lab and Kitware

10. EUFORIA FP7-INFRASTRUCTURES-2007-1, Grant 211804 Paraview

11. EUFORIA FP7-INFRASTRUCTURES-2007-1, Grant 211804 VisIt Software based on parallel VTK Tool to visualize different kind of data The VTK pipeline is hidden from the user  More user friendly  Less flexibility Parallelization of processing (remote processing) The plugin system proposes an intermediate layer on top of VTK Open source software designed by Lawrence Livermore Lab

12. EUFORIA FP7-INFRASTRUCTURES-2007-1, Grant 211804 VisIt

13. EUFORIA FP7-INFRASTRUCTURES-2007-1, Grant 211804 Task1: Work plan List visualization requirements of codes in platform Develop data (MDS+, UAL) import plugin for  Matplotlib  VisIt Develop sample visualizations for typical data in Matplotlib and VisIt Write tutorials

14. EUFORIA FP7-INFRASTRUCTURES-2007-1, Grant 211804 Outline Presentation of JRA4 tasks Task 1: Existing visualization tools Task 2: Visualization & Kepler platform Task 3: Lossy data compression format Task 4: 4D and 5D visualization

15. EUFORIA FP7-INFRASTRUCTURES-2007-1, Grant 211804 Monitoring tools Visualization during the simulation Check if the simulation is still OK 1D, 2D (light) dataset are enough Visualization can be performed on the gateway Visualization must be integrated to Kepler Which quantities are reliable for code monitoring ? Which tools have to be integrated into kepler ?

16. EUFORIA FP7-INFRASTRUCTURES-2007-1, Grant 211804 Interpretation tools Visualization after the simulation Huge dataset generated by distributed simulations 1. Copy every dataset to a central visualization resource  Requires big storage and network bandwidth  Requires a single visualization resource 2. Leave dataset on the remote simulation resource  Requires no extra storage or network bandwidth  Requires remote visualization resource

17. EUFORIA FP7-INFRASTRUCTURES-2007-1, Grant 211804 Visualization strategy In both case, we suggest to use an existing tool based on VTK: VisIt (or ParaView) Example of CSCS (J. Favre) Shared remote visualization resource strategy Data reading, processing and rendering are performed at the computer center Only resulting pictures go through the network Interactive visualization performed with ParaView

18. EUFORIA FP7-INFRASTRUCTURES-2007-1, Grant 211804 Remaining questions Build a link between visualization tool and Kepler in order to keep track of generated pictures ? What are physicists looking for in the data ?

19. EUFORIA FP7-INFRASTRUCTURES-2007-1, Grant 211804 Outline Presentation of JRA4 tasks Task 1: Existing visualization tools Task 2: Visualization & Kepler platform Task 3: Lossy data compression format Task 4: 4D and 5D visualization

20. EUFORIA FP7-INFRASTRUCTURES-2007-1, Grant 211804 Large dataset management Size S of a uniformly discretized function in Bytes d=2, N=128 => 128 KB d=3, N=128 => 16 MB d=4, N=128 => 2 GB d=4, N=256 => 32 GB d=6, N=128 => 32 TB

21. EUFORIA FP7-INFRASTRUCTURES-2007-1, Grant 211804 Large dataset management Strategy : Use data compression to reduce the size of the numerical problem Hierarchical bases of finite elements Local compression algorithm Compression integrated into the parallel simulation Efficient algorithm for slice reconstruction

22. EUFORIA FP7-INFRASTRUCTURES-2007-1, Grant 211804 Results for a 32 GB function Particles beam result

23. EUFORIA FP7-INFRASTRUCTURES-2007-1, Grant 211804 Results for a 32 GB function Laser-plasma interaction result

24. EUFORIA FP7-INFRASTRUCTURES-2007-1, Grant 211804 Present status Interactive exploration of 4D data with 2D slices Integration of the compression process into the parallel simulation => Allow the export of the distribution function => Without damaging the speedup of the simulation Data reduction with a factor between 8 and 150

25. EUFORIA FP7-INFRASTRUCTURES-2007-1, Grant 211804 Task 3: Work plan Enhance compression algorithm Parallelization of slice reconstruction algorithm Integration of these tools into VisIt, Kepler... Handle 5D dataset

26. EUFORIA FP7-INFRASTRUCTURES-2007-1, Grant 211804 Outline Presentation of JRA4 tasks Task 1: Existing visualization tools Task 2: Visualization & Kepler platform Task 3: Lossy data compression format Task 4: 4D and 5D visualization

27. EUFORIA FP7-INFRASTRUCTURES-2007-1, Grant 211804 Specific difficulties Visualization of data coming from kinetic simulations Particle distribution function lives in phase space The full model requires 6D + time RR Objective : visualize functions f: R 4  R How to visualize a 4D function ? How to represent the large dataset (2-32GB) for an interactive visualization ?

28. EUFORIA FP7-INFRASTRUCTURES-2007-1, Grant 211804 Multidimensional visualization 4D distribution function f(r,  v // ) Dimension reduction Representation of a set of slices/projections Data coming from Gysela5D (CEA Cadarache)

29. EUFORIA FP7-INFRASTRUCTURES-2007-1, Grant 211804 Demonstration Demo time !