1. Advanced Scientific Visualization Paul Navrátil 28 May 2009

2. Scientific Visualization “The purpose of computing is insight not numbers.” -- R. W. Hamming (1961) “The purpose of computing is insight not numbers.” -- R. W. Hamming (1961)

6. Visualization Allows Us to “See” the Science Application Render Geometric Primitives Pixels Raw Data 01001101011001 11001010010101 00101010100110 11101101011011 00110010111010 01001101011001 11001010010101 00101010100110 11101101011011 00110010111010

7. But what about large, distributed data? 01001101011001 11001010010101 00101010100110 11101101011011 00110010111010 01001101011001 11001010010101 00101010100110 11101101011011 00110010111010 01001101011001 11001010010101 00101010100110 11101101011011 00110010111010 01001101011001 11001010010101 00101010100110 11101101011011 00110010111010 01001101011001 11001010010101 00101010100110 11101101011011 00110010111010 01001101011001 11001010010101 00101010100110 11101101011011 00110010111010 01001101011001 11001010010101 00101010100110 11101101011011 00110010111010 01001101011001 11001010010101 00101010100110 11101101011011 00110010111010 01001101011001 11001010010101 00101010100110 11101101011011 00110010111010 01001101011001 11001010010101 00101010100110 11101101011011 00110010111010 01001101011001 11001010010101 00101010100110 11101101011011 00110010111010 01001101011001 11001010010101 00101010100110 11101101011011 00110010111010 01001101011001 11001010010101 00101010100110 11101101011011 00110010111010 01001101011001 11001010010101 00101010100110 11101101011011 00110010111010 01001101011001 11001010010101 00101010100110 11101101011011 00110010111010 01001101011001 11001010010101 00101010100110 11101101011011 00110010111010 01001101011001 11001010010101 00101010100110 11101101011011 00110010111010 01001101011001 11001010010101 00101010100110 11101101011011 00110010111010 01001101011001 11001010010101 00101010100110 11101101011011 00110010111010 01001101011001 11001010010101 00101010100110 11101101011011 00110010111010 01001101011001 11001010010101 00101010100110 11101101011011 00110010111010 01001101011001 11001010010101 00101010100110 11101101011011 00110010111010 01001101011001 11001010010101 00101010100110 11101101011011 00110010111010 01001101011001 11001010010101 00101010100110 11101101011011 00110010111010

8. Or distributed rendering? 01001101011001 11001010010101 00101010100110 11101101011011 00110010111010 01001101011001 11001010010101 00101010100110 11101101011011 00110010111010

9. Or distributed displays? 01001101011001 11001010010101 00101010100110 11101101011011 00110010111010 01001101011001 11001010010101 00101010100110 11101101011011 00110010111010

10. Or all three? 01001101011001 11001010010101 00101010100110 11101101011011 00110010111010 01001101011001 11001010010101 00101010100110 11101101011011 00110010111010 01001101011001 11001010010101 00101010100110 11101101011011 00110010111010 01001101011001 11001010010101 00101010100110 11101101011011 00110010111010 01001101011001 11001010010101 00101010100110 11101101011011 00110010111010 01001101011001 11001010010101 00101010100110 11101101011011 00110010111010 01001101011001 11001010010101 00101010100110 11101101011011 00110010111010 01001101011001 11001010010101 00101010100110 11101101011011 00110010111010 01001101011001 11001010010101 00101010100110 11101101011011 00110010111010 01001101011001 11001010010101 00101010100110 11101101011011 00110010111010 01001101011001 11001010010101 00101010100110 11101101011011 00110010111010 01001101011001 11001010010101 00101010100110 11101101011011 00110010111010 01001101011001 11001010010101 00101010100110 11101101011011 00110010111010 01001101011001 11001010010101 00101010100110 11101101011011 00110010111010 01001101011001 11001010010101 00101010100110 11101101011011 00110010111010 01001101011001 11001010010101 00101010100110 11101101011011 00110010111010 01001101011001 11001010010101 00101010100110 11101101011011 00110010111010 01001101011001 11001010010101 00101010100110 11101101011011 00110010111010 01001101011001 11001010010101 00101010100110 11101101011011 00110010111010 01001101011001 11001010010101 00101010100110 11101101011011 00110010111010 01001101011001 11001010010101 00101010100110 11101101011011 00110010111010 01001101011001 11001010010101 00101010100110 11101101011011 00110010111010 01001101011001 11001010010101 00101010100110 11101101011011 00110010111010 01001101011001 11001010010101 00101010100110 11101101011011 00110010111010

11. Schedule Advanced Scientific Visualization Topics –Large Data Visualization Challenges –Potential Solutions –Demonstrations Lab –Remote Visualization on Spur and Ranger –Scripting Visualization using Python in VisIt or Paraview

12. Visualization must scale with HPC Large data produced by large simulations require large visualization machines and produce large visualization results Terabytes of Data AT LEAST Terabytes of Vis Gigapixel Images Resampling, Application, … Resolution to Capture Feature Detail

13. Visualization Scaling Challenges Moving data to the visualization machine Most applications built for shared memory machines, not distributed clusters Image resolution limits in some software cannot capture feature details Displays cannot show entire high-resolution images at their native resolution

14. Moving Data How much time do you have? File Size10 Gbps54 Mbps 1 GB1 sec2.5 min 1 TB~17 min~43 hours 1 PB~12 days~5 years

15. Analyzing Data Visualization programs only beginning to efficiently handle ultrascale data –650 GB dataset -> 3 TB memory footprint –Allocate HPC nodes for RAM not cores –N-1 idle processors per node! Stability across many distributed nodes –Rendering clusters typically number N <= 64 –Data must be dividable onto N cores Remember this when resampling!

16. Imaging Data 4096 x 2160 PNG ~ 10 MB x 360 degrees ~ 3.6 GB x 30 days ~ 108 GB x 12 months ~ 1.3 TB @ 10 fps 3.6 hours @ 60 fps 36 min Image: NASA Blue Marble Project

17. Displaying Data Dell 30” flat-panel LCD 4 Megapixel display 2560 x 1600 resolution

18. Displaying Data Stallion – currently world’s highest-resolution tiled display 307 Megapixels 38400 x 8000 pixel resolution Dell 30” LCD

19. Displaying Data Dell 30” LCD – 4 Mpixel (2560 x 1600) Stallion – 307 Mpixel (38400 x 8000) NASA Blue Marble 0.5 km 2 per pixel 3732 Mpixel (86400 x 43200)

20. What’s the solution?

21. Solution by Partial Sums Moving data – integrate vis machine into simulation machine. Move the machine to data! –Ranger + Spur: shared file system and interconnect Analyzing data – create larger vis machines and develop more efficient vis apps –Smaller memory footprint –More stable across many distributed nodes Until then, the simulation machine is the vis machine!

22. Solution by Partial Sums Imaging data – focus vis effort on interesting features parallelize image creation –Feature detection to determine visualization targets but can miss “unknown unknowns” –Distribute image rendering across cluster Displaying data – high resolution displays multi-resolution image navigation –Large displays need large spaces –Physical navigation of display provides better insights

23. Old Model (No Remote Capability) Local Visualization Resource Local Visualization Resource HPC System HPC System Data Archive Data Archive Pixels Mouse Display Remote Site Wide-Area Network Wide-Area Network Local Site

24. New Model Remote Capability Large-Scale Visualization Resource Large-Scale Visualization Resource HPC System HPC System Data Archive Data Archive Display Remote Site Wide-Area Network Wide-Area Network Local Site Pixels Mouse

25. Spur / Ranger topology spur login3.ranger Login Nodes login4.ranger Compute Nodes Vis nodes ivis[1-7|big] HPC nodes ixxx-xxx vis queue normal development queues File System $HOME $WORK $SCRATCH

26. Parallel Visualization –Task parallelism – passing results to 1 process for rendering 12345 1Read file 1Isosurface 1Cut Plane 1 2Read file 2Streamlines 2Render 3Read file 3Triangulate 3Decimate 3Glyph 3 Timesteps Processes

27. Parallel Visualization Pipeline parallelism Useful when processes have access to separate resources or when an operation requires many steps. 12345 1Read file 1Read file 2Read File 3 2Isosurface 1Isosurface 2Isosurface 3 3Render 1Render 2Render 3 Timesteps Processes

28. Parallel Visualization Data parallelism Data set is partitioned among the processes and all processes execute same operations on the data. Scales well as long as the data and operations can be decomposed. 123 1Read partition 1 Isosurface partition 1 Render partition 1 2Read partition 2 Isosurface partition 2 Render partition 2 3Read partition 3 Isosurface partition 2 Render partition 3 Timesteps Processes

29. Parallel Visualization Libraries Chromium – http://chromium.sourceforge.nethttp://chromium.sourceforge.net –Sits between application and native OpenGL –Intercepts OpenGL calls, distribute across cluster –Can do either sort-first or sort-last (sort-first is simpler, sort-last can be better for large data) –Last update 31 Aug 2006, no new GL goodies IceT – http://www.cs.unm.edu/~kmorel/IceT/ SAGE – http://www.evl.uic.edu/cavern/sage/ CGLX – http://vis.ucsd.edu/~cglx/http://www.cs.unm.edu/~kmorel/IceT/http://www.evl.uic.edu/cavern/sage/http://vis.ucsd.edu/~cglx/ –specifically for large tiled displays –Must use IceT / SAGE / CGLX API in code Mesa – http://www.mesa3d.org/http://www.mesa3d.org/ –Software rendering library –Enables OpenGL rendering on machines without GPUs

30. Open-Source Parallel Vis Apps VisIt – https://wci.llnl.gov/codes/visit/https://wci.llnl.gov/codes/visit/ –Good scaling to hundreds of cores –Integrated job launching mechanism for rendering engines –Good documentation and user community –GUI not as polished ParaView – http://www.paraview.org/http://www.paraview.org/ –Polished GUI, easier to navigate –Less stable across hundreds of cores –Official documentation must be purchased, though rich knowledge base on web (via Google)

31. CUDA – coding for GPUs C / C++ interface plus GPU-based extensions Can use both for accelerating visualization operations and for general-purpose computing (GPGPU) Special GPU libraries for math, FFT, BLAS Image: Tom Halfhill, Microprocessor Report

32. GPU layout Image: Tom Halfhill, Microprocessor Report

33. GPU Considerations Parallelism – kernel should be highly SIMD –Switching kernels is expensive! Job size – high workload per thread –amortize thread initialization and memory transfer costs Memory footprint – task must decompose well –local store per GPU core is low (16 KB on G80) –card-local RAM is limited (~1GB on G8x) –access to system RAM is slow (treat like disk access) GPU N-body study (in GPU Gems 3): http://www.nvidia.com/object/io_1195170003876.html http://www.nvidia.com/object/io_1195170003876.html

34. Summary Challenges at every stage of visualization when operating on large data Partial solutions exist, though not integrated Problem sizes continue to grow at every stage Vis software community must keep pace with hardware innovations

35. Obrigado! pnav@tacc.utexas.edu