Progressive Transmission and Rendering of Foveated Volume Data Chen Chen, Zhiyong Huang, Hang Yu, Ee-Chien Chang School of Computing National University of Singapore

Introduction - motivation Problem Large volume data provided by modern scanners like CT and MRI Slow network speed comparing to the large size of volume data Solution Wavelet foveation Progressive transmission and rendering coarse-to-fine region-based 2018/11/19 grapp 2006

Introduction - wavelet foveated volume Foveated volume is a non-uniform sampled volume whose resolution is highest at the fovea (user specified ROI) and falls off as the distance to the fovea increases – priority selection Wavelet foveated volume is constructed by select ROI at each level of details in the volume’s wavelet form - compression 2018/11/19 grapp 2006

Proposed method - overview Transform to Wavelet Form Divide into 83 block, run-length encode (RLE) each block Client request = ROI {(x,y,z),s} Generate Fw Reorder data blocks of Fw Progressive transmission Progressive Rendering 2018/11/19 grapp 2006

Proposed method - overview Client-server volume rendering system Server Side: Input: Cw & ROI{(x,y,z),s} Output: Fw Client Side: Input: Fw Output: Iw1 -> Iw2 -> … -> Iwfinal Two transmission and rendering schemes 2018/11/19 grapp 2006

Proposed method - overview Server Site Client Site Wavelet Transform Cw request Generate Fw from Cw and ROI User Input ROI={(x,y,z),r} Reorder data blocks in Fw according to the schemes Progressive Render RB or CTF Progressive transmission Progressive rendering Display 2018/11/19 grapp 2006

Wavelet transform Divide the volume into blocks of size 2k3 Apply generic Haar wavelet transform on each block to obtain 7 detail blocks and 1 average block Group 8 adjacent blocks to again obtain a block of size 2k3 Repeat this procedure until the average volume reaches size m3 2018/11/19 grapp 2006

Wavelet transform - run-length encoding (RLE) A more compact representation RLE is performed block by block Compressed block will be a list of value(v) and length(l) pairs v= Integer value of a coefficient l = Number of successive v Example: L=1,1,1,1,0,0,5,0,0,0,0,0,0,0,0 R=(1,4),(0,2),(5,1),(0,8) 2018/11/19 grapp 2006

Derive foveated volume Cw, ROI{(x,y,z),s} => Fw Layer 0 image = average image of Cw Layer i image = IWT(layer i-1 image, 7 layer i-1 detail images) ROI in ith layer = ROI in Layer i image 2018/11/19 grapp 2006

Derive foveated volume Center of ROI in ith layer is Size of ROI in ith layer is s3 Boundary of ROI in ith layer can be defined as 2018/11/19 grapp 2006

Derive foveated volume 4th ROI = {(9,9),6} = {[6,11],[6,11]} Layer 3 details (i,j) =([3,5],[3,5]) 2018/11/19 grapp 2006

Progressive transmission - motivation Data size = 10243, Fovea size = 483, Average image size = 163 Maxlayer=Log2(1024/16)=6 Total amount of data = 243*(5*7+1)=497,664 coefficients Reorder data blocks Progressive transfer 2018/11/19 grapp 2006

Progressive transmission - two schemes Coarse-to-fine Description: Transmit a rough average image first and refine the fovea from outer ROIi to inner ROIi+1 Implementation: Send the wavelet foveated data from inner layer to outer layer progressively Region-based Description: Transmit full resolution ROImaxlayer first and expand the image from ROImaxlayer-1 to ROI0 with decreasing resolution Implementation: Data blocks and parent data blocks of ROImaxlayer-i+1 will be sent in ith iteration 2018/11/19 grapp 2006

Progressive transmission - illustration in 2D 2D image size= 128*128, m=16, k=8, ROI={(86,86),45}, maxlayer=3 ROI0={(10,10),45}=[0,15] blk_index=[0,1] ROI1={(21,21),45}=[0,31] blk_index=[0,1] ROI2={(43,43),45}=[20,63] blk_index=[1,3] ROI3={(86,86),45}=[64,107] blk_index=[4,6] 2018/11/19 grapp 2006

Progressive transmission - illustration in 2D 2018/11/19 grapp 2006

Progressive transmission - coarse-to-fine 2018/11/19 grapp 2006

Progressive transmission - coarse-to-fine 2018/11/19 grapp 2006

Progressive transmission - region-based 2018/11/19 grapp 2006

Progressive rendering - rendering equation For each voxel: intensity value opacity value The final intensity result reaches the viewer of each pixel will be Eulerian Sum over accumulated opacity Rendering equation to render a subvolume with same intensity and opacity value 2018/11/19 grapp 2006

Progressive rendering - rendering equation For 2 subvolumes Sa and Sb with size na and nb, voxels in Sa and Sb have , , and , respectively. Sa is in front of Sb, that is, Sa is “over” Sb. The intensity of Sa and Sb and Sfinal can be defined as 2018/11/19 grapp 2006

Region-based - partial volume reconstruction Starting coefficient: Ending coefficient: Wavelet coefficients at ith layer detail image begin: end: Reconstruction starts from layer 0, that is, the average image and move out to higher level of details until the desired resolution is reached 2018/11/19 grapp 2006

Region-based Center fovea is reconstructed using partial volume reconstruction and rendered at iteration 1 6 subvolumes are rendered: Top, Bottom, Left, Right, Front, Back with resolution scale 2(1-i), i is the number of iteration 6 sub volumes are rendered separately and combine the rendered result to the previous rendering image 2018/11/19 grapp 2006

Region-based - image composition Top, bottom, left and right are added to the previous rendering result image at their right position Front and back image needs image composition with the previous rendering result image: (front) over (inner) over (back) “over” operator can be defined as: given image pixel A(da, αa) and B(db, αb) C = A over B dc = da + αa*db αc = αa* αb 2018/11/19 grapp 2006

Region-based - image composition 2018/11/19 grapp 2006

Coarse-to-fine Start rendering from ROI0 to ROImaxlayer Each ROIi except ROImaxlayer can be divided into 7 subvolumes: top, bottom, left, right, front, back and center Center subvolume is the average image of the inner ROI and its rendering result, accumulated density value and opacity value will be kept for next iteration 2018/11/19 grapp 2006

Coarse-to-fine ROIi is constructed by center part of ROIi-1 and level i detail coefficients Number of voxels each coefficient in ROIi represent is 2maxlayer-i Rendering equation for each voxel in ROIi is Rendering result of front and back subvolume RFi and RBi will be take down for next iteration Rendering result of iteration i+1 will be combined with RFi and RBi in iteration i+1 2018/11/19 grapp 2006

Results - direct rendering 2018/11/19 grapp 2006

Results - region-based 2018/11/19 grapp 2006

Results - region-based 2018/11/19 grapp 2006

Results - coarse-to-fine 2018/11/19 grapp 2006

Results - coarse-to-fine 2018/11/19 grapp 2006

Results - time 2018/11/19 grapp 2006

Results - coarse-to-fine 2018/11/19 grapp 2006

Results - coarse-to-fine 2018/11/19 grapp 2006

Results - region-based 2018/11/19 grapp 2006

Results - region-based 2018/11/19 grapp 2006

Results - coarse-to-fine 2018/11/19 grapp 2006

Results - coarse-to-fine 2018/11/19 grapp 2006

Results - region-based 2018/11/19 grapp 2006

Results - region-based 2018/11/19 grapp 2006

Conclusion and future work Two progressive transmission and rendering schemes, region-based and coarse-to-fine Blockwise RLE used in wavelet compression Wavelet foveation Image composition Large memory is still needed cut the data set exceeding a certain amount into smaller data sets and build location map to indicate actual position of each data segments Cache can be used to store frequently used blocks, for example, those high level details 2018/11/19 grapp 2006