3D multimodal visualization techniques for tumor resection in neurosurgery Ralph Brecheisen Graduation committee: Prof. dr. ir. Bart ter Haar Romeny Prof. dr. ir. Jack van Wijk Dr. ir. Anna Vilanova i Bartroli Dr. ir. Bram Platel 18 december 2007 Supervisors: Dr. ir. Anna Vilanova i Bartroli Dr. ir. Bram Platel

2 Introduction Polestar N20 i-MRI Medtronic StealthStation Image-guided Surgery TU/e (Dr. Bram Platel) Department of Neurosurgery azM Enhance i-MRI with preoperative data Improve surgery planning Intraoperative navigation Resection verification Tumor resections

3 Brain tumor resections Tumor Functional activations Blood vessels Fiber tracts High-risk Procedures Image: Yale University

4 Investigate requirements Images: University of Florida, Music Therapy World, King Edward Memorial Hospital, Neuro Imaging UCLA, National Institutes of Health Tumor Functional activations Blood vessels Fiber tracts Anatomical context ? - 2-dimensional - Not spatially related Too difficult!

5 What do we want to do? Combine in single visualization Tumor Functional activations Blood vessels Fiber tracts Anatomical context 3D! Images: university hospital Maastricht, Stanford University, M. Meissner Viatronix Inc

6 What else do we need? Semi- transparency Hide/show Cut outs (clipping) Intraoperative navigation Surgical tools3D pointers Grid lines Reduce occlusions and clutter Geometric models

7 How to deal with the volume data?  Reviewed existing implementations  Main points:  Data representation  Rendering

8 Data representation A B Different scan times Different volume of interest Intraoperative data Interpolation inaccuracies Memory waste Resampling

9 Rendering screen Ray castingTexture slicing - Adaptive sampling - Programmable GPU - Works for old GPU’s - Not very flexible D D D

10 Review conclusion  Nobody does combination of  Ray casting  Render without resampling  Intersecting semi-transparent geometry How do WE render multiple volumes?

11 How to render a single volume? x z y Polygon Viewing window 2D projection Vertex - Color - Position Bounding box

12 Rasterization I Fragment - Color - Depth - ScreenCoord (i,j) Viewing window D frag (0,0) i j Screen Cube

13 Rasterization II nearest fragments written to screen Default: depth test LESS Other depth tests: EQUAL, GREATER

14 How to link bounding box with dataset? (0,0,0) (1,1,1) 3D texture = dataset (in GPU memory) Fragment - Color/depth/screenpos - Interpolated texture coordinate Vertex

15 Programmable fragment shaders Fragment shader Fragment User-defined parameters - E.g. texture ID’s Updated color Looping Boolean logic, math functions Random texture lookup Tex ID

16 Hardware-accelerated ray casting 3D texture Fragment shader: Take sequence of texture samples along viewing direction v Screen v* Projection image Mix samples frag Nearest fragments

17 How to mix texture samples? opacity bone image courtesy: Washington University Medical Center Transfer function 0 1 Blending equation α1α1 α2α2 α3α3 … skin Color samples

18 How could we render multiple volumes? Screen 3D Texture 13D Texture 2 A B P Check position at every step! Use single pass P’ P’’

19 Multi-volume depth peeling Nearest fragments Second-nearest fragments C ACC 2D accumulation texture Screen DEPTH LAYERS Subdivide volumes in regions

20 Initialization C ACC = (0,0,0,0) C NEAR D NEAR 2D textures Screen Depth test LESS

21 Depth peeling I Screen C FAR D FAR if (D frag ≤ D near ) then Discard() endif D NEAR Peeling shader second-nearest fragments depth test LESS

22 Ray casting I Raycasting shader: (1) Compute start/end positions in region (2) Take C acc (3) Blend with C near (4) Blend with texture samples (5) Output color C ACC C NEAR C ACC D NEAR D FAR depth test EQUAL D NEAR

23 Depth peeling II Screen C FAR D FAR if (D frag ≤ D near ) then Discard() endif D NEAR Peeling shader third-nearest fragments D NEAR  D FAR C NEAR  C FAR Swap textures

24 Ray casting II Raycast shader depth test EQUAL D NEAR C ACC C NEAR D NEAR D FAR Screen

25 Intersecting geometric models ScreenSphere Volume bounding box Raycasting shader: … (2) Take C acc (3) Blend with C near … C near = blue C near = 0,0,0,0 Almost for Free!

26 Cut outs (clipping) Screen Clipping sphere D NEAR D FAR

27 Results A picture says… DEMO!

28 Performance Dataset(s)FPS (1 mm)FPS (0.5 mm) CT head (256x256x225) CT head (256x256x225) + FMRI finger tapping (64x64x64) + DTI fiber tracts (geometry) + Tumor (64x64x41) CT body (512x512x174) CT body (512x512x174) + Engine (256x256x128) + Sphere (geometry) CT body + engine + clipping (with zoom)27.8 (13.5)13.8 (8.0) NVIDIA GeForce 8800 GTX, 768MB (screen: 500x500)

29 Conclusions  Investigated surgery planning requirements  Developed new multi-volume rendering algorithm based on depth peeling  N volumes, N transfer functions  Translucent geometry intersections  Convex clipping  Interactive displacement of volumes and geometry  Basic user interface

30 Future work  Build neurosurgery application  Integrate with i-MRI and navigation system  Combine gray value TF’s with object segmentation  Integrate DTI tool Anna  Enhance structures using multimodal info  Memory handling and performance

31 Questions?