1. 1/40 Illustrative Volume Rendering on Consumer Graphics Hardware Roy van Pelt Exam Committee: dr. A. Vilanova (BMT) dr. ir. H.M.M. van de Wetering (CSE) dr. ir. M. Chaudron (CSE)

2. 2/40 Project motivation  Improve comprehensibility  Useful for context visualisations  Similar to illustrations in physiology books

3. 3/40 Introduction: Illustrative Volume Rendering  Volume Rendering  3D dataset, i.e.: CT / MRI Images by kind courtesy of R. Brecheisen (2007) (MVR framework)

4. 4/40 Introduction: Illustrative Volume Rendering  Illustrative Volume Rendering  Depict dataset as a drawing/sketch StipplingHatchingContours

5. 5/40 Introduction: Illustrative Volume Rendering  Existing framework: VolumeFlies  Traditional illustration techniques  Based on particle systems

6. 6/40 Project objectives  Design and implement VolumeFlies, using consumer graphics hardware, such that real-time interaction with the data is possible.  Integrate VolumeFlies with a direct volume rendering approach.

7. 7/40 Background Info: GPU pipeline Massively parallel – Adjust algorithms to use parallelism – Unified shader design Programmability – Stream programming model using shaders Rasterization Geometry Processing FragmentOperationsVertexShaderGeometryShaderFragmentShader

8. 8/40 Background Info: GPU pipeline Massively parallel – Adjust algorithms to use parallelism – Unified shader design Programmability – Stream programming model using shaders Rasterization Geometry Processing FragmentOperationsVertexShaderGeometryShaderFragmentShader

9. 9/40 Background Info: General GPU approach Rasterization Geometry Processing FragmentOperationsVertexShaderGeometryShaderFragmentShader Proxy geometry = “Array index” Source buffer = “Input array”Destination buffer = “Output array” Transform feedback Discard fragment shader

10. 10/40 Background Info: Particle system  Particle: an element described by properties For example particle position in the volume Movie source: http://www.runevision.com

11. 11/40 Framework modules Initialiser 1 2 3 4 Behaviou r Filter Visualise r Initialises particle set Alters particle set Removes unwanted particles Illustratively depicts particles

12. 12/40 Framework modules Initialiser 1 Initialize particles near the ISO-surface Related to marching cubes

13. 13/40 Framework modules Initialiser 1VertexShaderGeometryShaderFragmentShader 3D Texture Volume Proxy geometry Particle positions (x,y,z) = (r,g,b)

14. 14/40 Framework modules Initialiser 1 2 3 4 Behaviou r Filter Visualise r

15. 15/40 Framework modules 2 Behaviou r 0.20.40.60.81 20 40 60 Redistribute particles evenly on the surface Energy minimisation “Robust particle systems for curvature dependent sampling of implicit surfaces” – Meyer et al. (SMI 2005) Minimise the total energy within the particle system

16. 16/40 Framework modules 2 Behaviou r Redistribute particles evenly on the surface Two step particle displacement iso-surface tangent plane “Robust particle systems for curvature dependent sampling of implicit surfaces” – Meyer et al. (SMI 2005)

17. 17/40 Framework modules 2 Behaviou r 1. Sort particles by their bin number Odd-even merge sort 2. Create a lookup table for the bins Binary search 3. Redistribute particles by repulsion Energy minimisation approach 1 1 1 2 2 2 3 3 3 4 4 4 5 5 5 6 6 6 7 7 7 8 8 8 9 9 9 10 11 12 1 1 1 3 3 3 4 4 4 5 5 5 2 2 2 6 6 6 8 8 8 7 7 7 10 12 9 9 9 11 Redistribution hard to port to GPU: Addressing all neighbours is computationally expensive Solution: Create a bin-structure to compute energy minimisation locally.

18. 18/40 Framework modules 2 Behaviou r VolumeFlies applied a fixed number of iterations A new stop-criterion is introduced Verify the change of the total energy Apply reduction operation on the GPU Total system energy + + +

19. 19/40 Framework modules 2 Behaviou r Odd-Even Merge Sort Binary Search Energy Minimization Verify stop-criterion GPU approach more elaborate. Still much faster than CPU approach. Particle system is generic.

20. 20/40 Framework modules Initializer 12 3 4 Behaviou r Filter Visualize r

21. 21/40 Framework modules 3Filter a) Create the ISO surface Cone splatting: Cones directed to the viewing plane Cones are scaled to prevent edge overlap

22. 22/40 Framework modules 3Filter b) Hidden surface removal Off-screen buffer: Splat rgb = Particle xyz Visible when particle position occurs as a colour  

23. 23/40 Framework modules Initialiser 1 2 3 4 Behaviou r Filter Visualise r

24. 24/40 Density-based stippling 4 Visualise r Vary particle density to change tone Initial density resembles darkest tone Per particle random threshold Based on basic diffuse lighting Executed by vertex shader User can control contrast

25. 25/40 Scale-based stippling 4 Visualise r Vary point size to change tone Larger points create darker areas Scale point representation Based on basic diffuse lighting Executed by vertex shader User can control contrast

26. 26/40 Direction-based hatching 4 Visualise r Hatch tracing in a single direction Single and cross hatching Single hatches Cross hatches

27. 27/40 Direction-based hatching 4 Visualise r iso-surface Hatch tracing in a single direction Segments projected in tangent plane

28. 28/40 Direction-based hatching 4 Visualise r VertexShaderGeometryShaderFragmentShader Particle positions Generate hatches in fixed direction 3D Texture: Volume Proxy geometry Hatch segments 1 VertexShaderGeometryShaderFragmentShader Visualise generated hatchesProxy geometry Hatch segments 2

29. 29/40 Direction-based hatching 4 Visualise r Again apply basic diffuse lighting Two-level threshold 2 nd level 1 st level

30. 30/40 Curvature-based hatching 4 Visualise r Emphasize curvature with hatches Fast curvature calculation approach 2. Optimize for GPU by using interpolation capabilities Differential structure in volume required “Fast Third-Order Texture Filtering” – Sigg, Hadwiger (GPU Gems 2) 1. Filtering approach for derivative reconstruction 3. Compute principal curvature Re-implemented in high-level shading language Based on general GPU-approach

31. 31/40 Curvature-based hatching 4 Visualise r Emphasize curvature with hatches Defining the principal curvature Image by Eric Gaba Variation of the normal vector when moving a small distance iso-surface tangent plane Maximum and minimum change describe the principal curvature

32. 32/40 Curvature-based hatching 4 Visualise r Emphasize curvature with curvature Calculate principal curvature  Measure first partial derivates:gradient “Curvature-Based Transfer Functions for Direct Volume Rendering Methods and Applications” – Kindlmann et al. (IEEE Visualization 2003) iso-surface tangent plane  Measure second partial derivates:Hessian  Extract curvatures by eigen analysis on the shape operator

33. 33/40 Curvature-based hatching 4 Visualise r 1D Transfer function -1 0 1

34. 34/40 Curvature-based hatching 4 Visualise r Messy hatch results using curvature directions Smooth the field directions and weight the trace Fixed direction when surface is unreliable

35. 35/40 Curvature-based hatching 4 Visualise r Principal curvature can be calculated in real-time Demonstrated by real-time curvature colour mapping

36. 36/40 Contours 4 Visualise r Draw ‘hatches’ for particles near the contour Segments trace the direction of the contour Contour: locations where normal is perpendicular to the view

37. 37/40 Results: Performance 253.34 52.65 53.05 Load Volume Brute-force initialiser Redistribution Smooth Field Hatch generation (Direction) Hatch generation (Scale) Hatch visualisation Contours

38. 38/40 Results: Demonstration movie

39. 39/40 Conclusions & Future work  Completely GPU-based  General GPU approach  Two generic components:  Particle system  Real-time curvature estimator  Integration with direct volume rendering  More elaborate memory management  Improvement of modules / Zooming  New applications and styles: DTI, animation, ghosting, exploded view…

40. 40/40 Thank you for your attention! Questions ?

41. 41/40 Particle repulsion  Energy minimisation  Two step particle displacement

42. 42/40 Stippling  Density-based  Scale-based

43. 43/40 Hatching  Smooth field

44. 44/40 Hatching  Tracing

45. 45/40 Contours Image by D. DeCarlo

46. 46/40 Curvature estimation 1/3  Measure partial derivatives by convolution filtering with cubic B-spline

47. 47/40 Curvature estimation 2/3  Optimise for the GPU by using interpolation x ii+1i-1i+2 Weights lookup texture Colour texture

48. 48/40 Curvature estimation 3/3  Compute principal curvature  Eigen analysis on S