1/40 Illustrative Volume Rendering on Consumer Graphics Hardware Roy van Pelt Exam Committee: dr. A. Vilanova (BMT) dr. ir. H.M.M. van de Wetering (CSE) dr. ir. M. Chaudron (CSE)

2/40 Project motivation  Improve comprehensibility  Useful for context visualisations  Similar to illustrations in physiology books

3/40 Introduction: Illustrative Volume Rendering  Volume Rendering  3D dataset, i.e.: CT / MRI Images by kind courtesy of R. Brecheisen (2007) (MVR framework)

4/40 Introduction: Illustrative Volume Rendering  Illustrative Volume Rendering  Depict dataset as a drawing/sketch StipplingHatchingContours

5/40 Introduction: Illustrative Volume Rendering  Existing framework: VolumeFlies  Traditional illustration techniques  Based on particle systems

6/40 Project objectives  Design and implement VolumeFlies, using consumer graphics hardware, such that real-time interaction with the data is possible.  Integrate VolumeFlies with a direct volume rendering approach.

7/40 Background Info: GPU pipeline Massively parallel – Adjust algorithms to use parallelism – Unified shader design Programmability – Stream programming model using shaders Rasterization Geometry Processing FragmentOperationsVertexShaderGeometryShaderFragmentShader

8/40 Background Info: GPU pipeline Massively parallel – Adjust algorithms to use parallelism – Unified shader design Programmability – Stream programming model using shaders Rasterization Geometry Processing FragmentOperationsVertexShaderGeometryShaderFragmentShader

9/40 Background Info: General GPU approach Rasterization Geometry Processing FragmentOperationsVertexShaderGeometryShaderFragmentShader Proxy geometry = “Array index” Source buffer = “Input array”Destination buffer = “Output array” Transform feedback Discard fragment shader

10/40 Background Info: Particle system  Particle: an element described by properties For example particle position in the volume Movie source:

11/40 Framework modules Initialiser Behaviou r Filter Visualise r Initialises particle set Alters particle set Removes unwanted particles Illustratively depicts particles

12/40 Framework modules Initialiser 1 Initialize particles near the ISO-surface Related to marching cubes

13/40 Framework modules Initialiser 1VertexShaderGeometryShaderFragmentShader 3D Texture Volume Proxy geometry Particle positions (x,y,z) = (r,g,b)

14/40 Framework modules Initialiser Behaviou r Filter Visualise r

15/40 Framework modules 2 Behaviou r Redistribute particles evenly on the surface Energy minimisation “Robust particle systems for curvature dependent sampling of implicit surfaces” – Meyer et al. (SMI 2005) Minimise the total energy within the particle system

16/40 Framework modules 2 Behaviou r Redistribute particles evenly on the surface Two step particle displacement iso-surface tangent plane “Robust particle systems for curvature dependent sampling of implicit surfaces” – Meyer et al. (SMI 2005)

17/40 Framework modules 2 Behaviou r 1. Sort particles by their bin number Odd-even merge sort 2. Create a lookup table for the bins Binary search 3. Redistribute particles by repulsion Energy minimisation approach Redistribution hard to port to GPU: Addressing all neighbours is computationally expensive Solution: Create a bin-structure to compute energy minimisation locally.

18/40 Framework modules 2 Behaviou r VolumeFlies applied a fixed number of iterations A new stop-criterion is introduced Verify the change of the total energy Apply reduction operation on the GPU Total system energy + + +

19/40 Framework modules 2 Behaviou r Odd-Even Merge Sort Binary Search Energy Minimization Verify stop-criterion GPU approach more elaborate. Still much faster than CPU approach. Particle system is generic.

20/40 Framework modules Initializer Behaviou r Filter Visualize r

21/40 Framework modules 3Filter a) Create the ISO surface Cone splatting: Cones directed to the viewing plane Cones are scaled to prevent edge overlap

22/40 Framework modules 3Filter b) Hidden surface removal Off-screen buffer: Splat rgb = Particle xyz Visible when particle position occurs as a colour  

23/40 Framework modules Initialiser Behaviou r Filter Visualise r

24/40 Density-based stippling 4 Visualise r Vary particle density to change tone Initial density resembles darkest tone Per particle random threshold Based on basic diffuse lighting Executed by vertex shader User can control contrast

25/40 Scale-based stippling 4 Visualise r Vary point size to change tone Larger points create darker areas Scale point representation Based on basic diffuse lighting Executed by vertex shader User can control contrast

26/40 Direction-based hatching 4 Visualise r Hatch tracing in a single direction Single and cross hatching Single hatches Cross hatches

27/40 Direction-based hatching 4 Visualise r iso-surface Hatch tracing in a single direction Segments projected in tangent plane

28/40 Direction-based hatching 4 Visualise r VertexShaderGeometryShaderFragmentShader Particle positions Generate hatches in fixed direction 3D Texture: Volume Proxy geometry Hatch segments 1 VertexShaderGeometryShaderFragmentShader Visualise generated hatchesProxy geometry Hatch segments 2

29/40 Direction-based hatching 4 Visualise r Again apply basic diffuse lighting Two-level threshold 2 nd level 1 st level

30/40 Curvature-based hatching 4 Visualise r Emphasize curvature with hatches Fast curvature calculation approach 2. Optimize for GPU by using interpolation capabilities Differential structure in volume required “Fast Third-Order Texture Filtering” – Sigg, Hadwiger (GPU Gems 2) 1. Filtering approach for derivative reconstruction 3. Compute principal curvature Re-implemented in high-level shading language Based on general GPU-approach

31/40 Curvature-based hatching 4 Visualise r Emphasize curvature with hatches Defining the principal curvature Image by Eric Gaba Variation of the normal vector when moving a small distance iso-surface tangent plane Maximum and minimum change describe the principal curvature

32/40 Curvature-based hatching 4 Visualise r Emphasize curvature with curvature Calculate principal curvature  Measure first partial derivates:gradient “Curvature-Based Transfer Functions for Direct Volume Rendering Methods and Applications” – Kindlmann et al. (IEEE Visualization 2003) iso-surface tangent plane  Measure second partial derivates:Hessian  Extract curvatures by eigen analysis on the shape operator

33/40 Curvature-based hatching 4 Visualise r 1D Transfer function

34/40 Curvature-based hatching 4 Visualise r Messy hatch results using curvature directions Smooth the field directions and weight the trace Fixed direction when surface is unreliable

35/40 Curvature-based hatching 4 Visualise r Principal curvature can be calculated in real-time Demonstrated by real-time curvature colour mapping

36/40 Contours 4 Visualise r Draw ‘hatches’ for particles near the contour Segments trace the direction of the contour Contour: locations where normal is perpendicular to the view

37/40 Results: Performance Load Volume Brute-force initialiser Redistribution Smooth Field Hatch generation (Direction) Hatch generation (Scale) Hatch visualisation Contours

38/40 Results: Demonstration movie

39/40 Conclusions & Future work  Completely GPU-based  General GPU approach  Two generic components:  Particle system  Real-time curvature estimator  Integration with direct volume rendering  More elaborate memory management  Improvement of modules / Zooming  New applications and styles: DTI, animation, ghosting, exploded view…

40/40 Thank you for your attention! Questions ?

41/40 Particle repulsion  Energy minimisation  Two step particle displacement

42/40 Stippling  Density-based  Scale-based

43/40 Hatching  Smooth field

44/40 Hatching  Tracing

45/40 Contours Image by D. DeCarlo

46/40 Curvature estimation 1/3  Measure partial derivatives by convolution filtering with cubic B-spline

47/40 Curvature estimation 2/3  Optimise for the GPU by using interpolation x ii+1i-1i+2 Weights lookup texture Colour texture

48/40 Curvature estimation 3/3  Compute principal curvature  Eigen analysis on S