1. Volume Visualization Huamin QU Computer Science Department Hong Kong University of Science and Technology Hong Kong University of Science and Technology

2. https://www.youtube.com/watch?v=- aCuxW6saGYhttps://www.youtube.com/watch?v=- aCuxW6saGY

3. Today Volume visualization Data acquisition Iso-surface visualization Direct volume rendering

4. Volume Visualization Volume visualization creates images from scalar and vector datasets defined on multiple dimensional grids to gain insight into the structure contained within the data.

5. Volumetric Data Volumetric data is a set V of samples (x, y, z, v) - v is property, may be vector - (x, y, z) is 3D location The samples are located at the grid points

6. Grid Types: Structured Grids Regular Grids Curvilinear Grids

7. Grid Types: Structured Grids Can use 3D array to store data - Voxel [i, j, k] Grid points may be implicitly defined Neighboring voxels can be easily located

8. Grid Types: Unstructured Grids

9. Store as a list of voxels Grid positions need to be explicitly stored Neighboring voxels are not easy to locate

10. Volume Data Acquisition Methods X-Rays Computer Tomography (CT or CAT) MRI (or NMR) PET / SPECT Ultrasound Microscopy Computational Synthetic

11. CT or CAT - Principles Computerized (Axial) Tomography - introduced in 1972 by Hounsfield and Cormack - natural progression from X-rays - based on the principle that a three-dimensional object can be reconstructed from its two dimensional projections - based on the Radon transform (a map from an n dimensional space to an (n-1)-dimensional space)

12. Computed Tomography (CT) - Overview

13. CT or CAT - Methods measures the attenuation of X-rays from many different angles - a computer reconstructs the organ under study in a series of cross sections or planes - combine X-ray pictures from various angles to reconstruct 3D structures

14. CT or CAT - Advantages significantly more data is collected - superior to single X-ray scans - far easier to separate soft tissues other than bone from one another (e.g. liver, kidney) - data exist in digital form -> can be analyzed quantitatively - adds enormously to the diagnostic information - used in many large hospitals and medical centers throughout the world

15. CT or CAT - Disadvantages significantly more data is collected - soft tissue X-ray absorption still relatively similar - still a health risk MRI is used for a detailed imaging of anatomy

16. MRI Nuclear Magnetic Resonance (NMR) (or Magnetic Resonance Imaging - MRI) - most detailed anatomical information - high-energy radiation is not used, i.e. “ save ” - based on the principle of nuclear resonance - (medicine) uses resonance properties of protons MRI is good for tissues, but not for bone

17. Magnetic Spin Resonance (MRI) - Overview

18. PET/SPECT Positron Emission Tomography Single Photon Emission Computerized Tomography - recent technique - involves the emission of particles of antimatter by compounds injected into the body being scanned - follow the movements of the injected compound and its metabolism - reconstruction techniques similar to CT - Filter Back Projection & iterative schemes

19. SPECT Emit (any) gamma rays collected with gamma camera cheap

20. PET positrons collides with electron to emit photons in 1800 angle both annihilation photons detected in coincidence higher sensitivity more expensive tracer has shorter half-live

21. PET Images

22. Comparison “ CT and MRI show that you have a brain; PET and SPECT show that you use it! ”

23. Ultrasound the use of high-frequency sound (ultrasonic) waves to produce images of structures within the human body above the range of sound audible to humans (typically above 1MHz) piezoelectric crystal creates sound waves aimed at a specific area of the body change in tissue density reflects waves echoes are recorded

24. Ultrasound (2) Delay of reflected signal and amplitude determines the position of the tissue still images or a moving picture of the inside of the body there are no known examples of tissue damage from conventional ultrasound imaging commonly used to examine fetuses in utero in order to ascertain size, position, or abnormalities also for heart, liver, kidneys, gallbladder, breast, eye, and major blood vessels

25. Ultrasound (3) by far least expensive very safe very noisy 1D, 2D, 3D scanners irregular sampling - reconstruction problems

26. Ultrasound Image

27. Comparison

28. Computational Methods (CM) Computational Field Simulations Computational Fluid Dynamics - Flow simulations Computational Chemistry - Electron-electron interactions, Molecular surfaces Computational Mechanics - Fracture Computational Manufacturing – Die-casting

29. CM - Approach (Continuous) physical model – Partial/Ordinary Differential Equation (ODE/PDE) – e.g. Navier-Stokes equation for fluid flow – e.g. Hosted Equations – e.g. Schrödinger Equation - for waves/quantum Continuous solution doesn ’ t exist (for most part) Numerical Approximation/Solution 1. Discretize solution space - Grid generation explicit 2. Replace continuous operators with discrete ones 3. Solve for physical quantities

30. CM - Methods Grid Generation – non-elliptical methods: algebraic, conformal, hyperbolic, parabolic, biharmonic – elliptical methods (based on elliptical PDE ’ s) Numerical Methods – Newton – Runge-Kutta – Finite Element – Finite Differences Time Varying

31. CM - Solutions (Structured) PDE usually constrained/given at boundary map from computational to physical space polar maps, elliptical, non-elliptical structured grids

32. CM - Solutions (Unstructured) usually scattered data set Delaunay Triangulation Element Size Optimization – start with initial tetrahedral base grid – interactively insert grid points – insertion guided by curvature and distance to surface Advancing Front Method – start with boundary – advance boundary towards inside until “ filled ”

33. Data Representation

34. Data Representation (2) Polygon mesh, Curvilinear, Unstructured – Vertex list – Adjacency list of cells (not for curvilinear) – No convenient structure Compressed Grids - RLE, JPEG, Wavelets Multi-resolution Grids

35. Data Characteristics Large Data Sets – Modest Head 256 3 = 16Mbytes !! – Visible Human - MRI (256 2 ) + CT (512 2 ) + photo – Male - 15GB (1mm slice dist.); Female - 40GB (0.33mm) Noisy - Ultrasound (How about CFD, CAGD ?) Band-limited - Textured Images (Mandrill)

36. Data Characteristics Limited Dynamic Range - values between min, max populated with unequal probability Non-Uniform Spatial Occupancy - – Quadtree/Octree – Rectangular spatial subdivision

37. Volume Generation Capture original function accurately – Sampling theorems – Sufficient resolution Should not create – Noise - medical – Small triangles - CAGD – Flaws (cracks) - CAGD For computational simulations – Capture geometry – Adapt to solution – Time varying, vector fields

38. Volume Visualization Method Rendering via Geometric Primitives - triangle meshes Directive volume rendering

39. Isosurface Extraction by contouring – closed contours – continuous – determined by iso-value several methods – marching cubes – dividing cubes – surface tracking – span space

40. Marching Cubes Cell consists of 4(8) pixel (voxel) values: (i+[01], j+[01], k+[01]) 1. Consider a Cell 2. Classify each vertex as inside or outside 3. Build an index 4. Get edge list from table[index] 5. Interpolate the edge location 6. Go to next cell

41. MC 1: Create a Cube Consider a Cube defined by eight data values:

42. MC 2: Classify Each Voxel Classify each voxel according to whether it lies outside the surface (value > iso-surface value) inside the surface (value <= iso- surface value)

43. Build An Index Use the binary labeling of each voxel to create an index

44. MC 4: Lookup Edge List For a given index, access an array storing a list of edges all 256 cases can be derived from 15 base cases

45. MC 4: Example Index = 10110001 triangle 1 = e4,e7,e12 triangle 2 = e1, e7, e4 triangle 3 = e1, e6, e7 triangle 4 = e1, e10, e6

46. MC 5: Interp. Triangle Vertex For each triangle edge, find the vertex location along the edge using linear interpolation of the voxel values

47. MC 6: Compute Normals Calculate the normal at each cube vertex Use linear interpolation to compute the polygon vertex normal

48. Direct Volume Rendering Ray casting (Ray tracing) Splatting Shear-warp 3D texture mapping

49. Direct Volume Rendering Ray casting (Ray tracing) Splatting Shear-warp 3D texture mapping

50. Direct Volume Rendering The following slides are from Huang Jian (www.cs.utk.edu/~huangj/CS594S02/raycasting.ppt ) Klaus Mueller (http://www.cs.sunysb.edu/~mueller/teaching/cse564/fullVolRen.pdf)http://www.cs.sunysb.edu/~mueller/teaching/cse564/fullVolRen.pdf

51. Full Volume Rendering - Overview Here we consider the volume as a transparent gel The rays accumulate colors and opacities (transparencies) as they step across the volume Once the opacity has reached a value close to 1.0 we can stop the ray... everything further back is hidden by the cumulative opaque structures in front

52. Transfer Function

53. The Volume Rendering Integral Consider a volume consisting of particles: - each has color C and light attenuating density μ controlled by a transfer function

54. Volume Rendering Integral (I) A rendering ray accumulates attenuated colors This process can be simulated by the volume rendering integral, which is generally not solvable analytically)

55. The Volume Rendering Integral (II) We can approximate it by discretizing it into sampling intervals of width ∆s: After simplification, the discretized volume rendering integral can be solved by stepwise compositing samples along the ray

56. What Exactly is Compositing? It is the accumulation of colors weighted by opacities Compositing is commonplace in cartoon animations - blend images of static objects with images of dynamic characters (so called cel-animations) Colors and opacities of back pixels are attenuated by opacities of front pixels: rgb = RGB back · α back (1 - α front ) + RGB front · α front α = α back (1 - α front ) + α front

57. Compositing By using: rgb back = RGB back · α back rgb front = RGB front · α front we get 2 recursive equations that can be used to composite any number of objects front-to-back: rgb front = rgb back (1 - α front ) + rgb front α front = α back (1 - α front ) + α front Volume rendering uses this recursive expression to combine (=composite) the samples taken along the ray

58. Compositing - Example

59. Papers Tuy and Tuy, 1984, IEEE CG & A (one of the earliest volume rendering techniques) Levoy, 1988 IEEE CG&A, and later improvements Drebin, Carpenter, Hanrahan, 1988, SIGGRAPH Direct: No conversion to surface geometry

60. Basic Idea Based on the idea of ray tracing Trace from eat each pixel as a ray into object space Compute color value along the ray Assign the value to the pixel

61. Data Representation 3D volume data are represented by a finite number of cross sectional slices (hence a 3D raster) On each volume element (voxel), stores a data value (if it uses only a single bit, then it is a binary data set. Normally, we see a gray value of 8 to 16 bits on each voxel.) N x 2D arraies = 3D array

62. Data Representation (2) What is a Voxel? – Two definitions A voxel is a cubic cell, which has a single value cover the entire cubic region A voxel is a data point at a corner of the cubic cell The value of a point inside the cell is determined by interpolation

63. Viewing Ray Casting Where to position the volume and image plane What is a ‘ ray ’ How to march a ray

64. Viewing (1) 1. Position the volume Assuming the volume dimensions is w x w x w We position the center of the volume at the world origin y (0,0,0)x z Volume center = [w/2,w/2,w/2] (local space) Translate T(-w/2,-w/2,-w/2) (data to world matrix? world to data matrix )

65. Viewing (2) 2. Position the image plane Assuming the distance between the image plane and the volume center is D, and initially the center of the image plane is (0,0,-D) y (0,0,0)x z Image plane

66. Viewing (3) 3. Rotate the image plane A new position of the image plane can be defined in terms of three rotation angle  with respect to x,y,z axes Assuming the original view vector is [0,0,1], then the new view vector g becomes: cos  sin  cos  sin  g = [0,0,1] 0 1 0 0 cos  sin  sin  cos  sin  0 cos  sin  cos 

67. Viewing (4) y (0,0,0)x z u v E S E0u0 v0 + S0 B B = [0,0,0] S 0 = [0,0,-D] u 0 = [1,0,0] v 0 = [0,1,0] Now, R: the rotation matrix S = B – D x g U = [1,0,0] x R V = [0,1,0] x R

68. Viewing (5) R: the rotation matrix S = B – D x g U = [1,0,0] x R V = [0,1,0] x R + S Image Plane: L x L pixels E Then E = S – L/2 x u – L/2 x v So Each pixel (i,j) has coordinates P = E + i x u + j x v u v We enumerate the pixels by changing i and j (0..L-1)

69. Viewing (6) 4. Cast rays Remember for each pixel on the image plane P = E + i x u + j x v and the view vector g = [0,0,1] x R So the ray has the equation: Q = P + k (d x g) d: the sampling distance at each step x x d x x p Q K = 0,1,2,…

70. Early Methods Before 1988 Did not consider transparency did not consider sophisticated light transportation theory were concerned with quick solutions hence more or less applied to binary data non-binary data - require sophisticated classification/compositing methods!

71. Ray Tracing -> Ray Casting “ another ” typical method from traditional graphics Typically we only deal with primary rays - hence: ray-casting a natural image-order technique as opposed to surface graphics - how do we calculate the ray/surface intersection??? Since we have no surfaces - we need to carefully step through the volume

72. Ray Casting Stepping through the volume: a ray is cast into the volume, sampling the volume at certain intervals The sampling intervals are usually equi-distant, but don ’ t have to be (e.g. importance sampling) At each sampling location, a sample is interpolated / reconstructed from the grid voxels popular filters are: nearest neighbor (box), trilinear (tent), Gaussian, cubic spline Along the ray - what are we looking for?

73. Example: Using the nearest neighbor kernel In tuys ’ paper Q = P + K x V (v=dxg) At each step k, Q is rounded off to the nearest voxel (like the DDA algorithm) Check if the voxel is on the boundary or not (compare against a threshold) If yes, perform shading

74. Basic Idea of Ray-casting Pipeline - Data are defined at the corners of each cell (voxel) - The data value inside the voxel is determined using interpolation (e.g. tri-linear) - Composite colors and opacities along the ray path - Can use other ray-traversal schemes as well c1 c2 c3

75. Ray Traversal Schemes Depth Intensity Max Average Accumulate First

76. Ray Traversal - First Depth Intensity First First: extracts iso-surfaces (again!) done by Tuy&Tuy ’ 84

77. Ray Traversal - Average Depth Intensity Average Average: produces basically an X-ray picture

78. Ray Traversal - MIP Depth Intensity Max Max: Maximum Intensity Projection used for Magnetic Resonance Angiogram

79. Ray Traversal - Accumulate Depth Intensity Accumulate Accumulate opacity while compositing colors: make transparent layers visible! Levoy ‘ 88

80. Raycasting color opacity 1.0 volumetric compositing object (color, opacity)

81. Raycasting color opacity Interpolation kernel 1.0 object (color, opacity) volumetric compositing

82. Raycasting color c = c s  s (1 -  ) + c opacity  =  s (1 -  ) +  1.0 object (color, opacity) volumetric compositing Interpolation kernel

83. Raycasting color opacity 1.0 object (color, opacity) volumetric compositing

84. Raycasting color opacity 1.0 object (color, opacity) volumetric compositing

85. Raycasting color opacity 1.0 object (color, opacity) volumetric compositing

86. Raycasting color opacity 1.0 object (color, opacity) volumetric compositing

87. Raycasting color opacity object (color, opacity) volumetric compositing

88. Volume Rendering Pipeline Acquired values Data preparation Prepared values classificationshading Voxel colors Ray-tracing / resampling Sample colors compositing Voxel opacities Sample opacities Image Pixels

89. DCH DVR Pipeline

90. DCH - Pipeline Original data Material percentage volumes Color volumeOpacity volumeDensity volume GradientShaded volume Transformed volume Final image Classification Normals Shading shears compositing

91. Common Components of General Pipeline Interpolation/reconstruction Classification or transfer function Gradient/normal estimation for shading Question: are normals also interpolated?

92. Levoy - Interpolation

93. Levoy - Interpolation (2) Closest valueWeighted average

94. Levoy – Gradient/Normals Central difference per voxel X+1 y-1, Y+1 z-1

95. Levoy - Compositing Image order back-to-front using the over operator

96. Levoy - Shading Phong Shading + Depth Cueing C p = color of parallel light source k a / k d / k s = ambient / diffuse / specular light coefficient k 1, k 2 = fall-off constants d(x) = distance to picture plane L = normalized vector to light H = normalized vector for maximum highlight N(x i ) = surface normal at voxel x i

97. Classification Classification: Mapping from data to opacities Region of interest: high opacity (more opaque) Rest: translucent or transparent The opacity function is typically specified by the user Levoy came up with two formula to compute opacity 1. Isosurface 2. Region boundary (e.g. between bone and fresh)

98. Classification/Transfer Function Maps raw voxel value into presentable entities: color, intensity, opacity, etc. Raw-data  material (R, G, B, , K a, K d, K s,...) May require probabilistic methods (Drebin). Derive material volume from input. Estimate % of each material in all voxels. Pre-computed. AKA segmentation. Often use look-up tables (LUT) to store the transfer function that are discovered

99. Levoy - Classification Usually not only interested in a particular iso- surface but also in regions of “ change ” Feature extraction - High value of opacity exists in regions of change Transfer function (Levoy) - Saliency Surface “ strength ”

100. Levoy - Classification Chemistry Data only iso-value - loose information of layers iso-range - could be too narrow or too wide thickness of region should be constant hence linear fall off of opacity wider fall off for larger gradient Medical Data assume at most two tissues meet linear transition between opacities of “ neighboring ” tissues reflects linear combination of tissues within one voxel

101. Opacity function (1) Goal: visualize voxels that have a selected threshold value fv - No intermediate geometry is extracted - The idea is to assign voxels that have value fv the maximum opacity (say  ) - And then create a smooth transition for the surrounding area from 1 to 0 -Levoy wants to maintain a constant thickness for the transition area.

102. Opacity function (2) Maintain a constant isosurface thickness opacity =  opacity = 0 Can we assign opacity based on function value instead of distance? (local operation: we don ’ t know where the isosurface is) Yes – we can based on the value distance f – fv but we need to take into account the local gradient

103. Opacity function (3) Assign opacity based on value difference (f-fv) and local gradient gradient: the value fall-off rate grad =  f  s Assuming a region has a constant gradient and the isosurface transition has a thickness R opacity =  F = fv opacity = 0 F = fv – grad * R thickness = R F = f(x) Then we interpolate the opacity opacity =  –  * ( fv-f(x))/ (grad * R)

104. Levoy - Classification A

105. Levoy - Classification B

106. DCH - Material Percentage V. Probabilistic classifier probability that a voxel has intensity I: p i - percentage of material P i (I) - prob. that material i has value I P i (I) given through statistics/physics p i then given by:

107. DCH - Classification Like Levoy - assumes only two materials per voxel that will lead to material percentage volumes from them we conclude color/opacity: where C i =(  i R i,  i G i,  i B i,  i )

108. DCH- Classification

109. Levoy - Improvements Levoy 1990 front-to-back with early ray termination  = 0.95 hierarchical oct-tree data structure skip empty cells efficiently

110. Volumetric Ray Integration color opacity object (color, opacity) 1.0