1. Stereoscopic Video Overlay with Deformable Registration Balazs Vagvolgyi Prof. Gregory Hager CISST ERC Dr. David Yuh, M.D. Department of Surgery Johns Hopkins University

2. The CASA Project Today’s Surgical Assistant: A Simple Information Channel

3. The CASA Project Stereo surface tracking Stereo tool tracking Virtual fixtures with da Vinci Robot Task graph execution system HMM-based Intent Recognition Information Fusion with da Vinci Display Ultrasound Capabilities of a Context-Aware Surgical Assistant (CASA) Tissue Classification Preoperative Imagery

4. The CASA Project Stereo surface tracking Stereo tool tracking Information Fusion with da Vinci Display Developing a Context-Aware Surgical Assistant (CASA) Preoperative Imagery

5. Information Overlay Problem setting: –Given pre-operative scan data from a suitable imaging modality –Video sequence from a stereo endoscope Add value –Overlay underlying anatomy on the stereo video stream (x-ray vision) –Include annotations or other information tied to imagery [[ add kidney picture ]] Key Problem: Nonrigid registration of organ surface to data

6. Inputs: What Do We Know? 1.Pre-operative 3D model - most probably volumetric - only a portion of it will be visible on the endoscope - anatomy will be deformed during the surgical procedure 2.Camera system properties can be measured - optical & stereo calibration - local brightness/contrast/color response 3.Stereo image stream - 3D surface can be reconstructed - texture information 4.A guesstimate of model–endoscope 3D relationship - We can guess where to start searching [i.e. patient position]

7. Outputs: What Do We Generate? 1.Position of 3D model registered to stereo image 2.Model deformed to the current shape of anatomy 3.Rendering a synthetic 3D view on the stereo stream 4.Everything done real-time Original ImageStereo DataDeformed Mesh

8. 2D3D All this in a flow chart Stereo image pre-processing Building and optimizing disparity map Deformable Registration to 3D surface 3D texture tracking Recognizing deformations optical parameters stereo video stream Image overlay disparity 3D data image data parameters 3D model

9. Classical Stereo Vision: The Problem Blocks of each image are compared using SAD Optimization for each block independently on entire depth range +Very fast implementation (GPU) ¬Lousy results Small Vision System from Videre Design (w/o structured light):

10. Input images downsized to several scale levels (½, ¼, …) Each scale processed with the same algorithm –Propagate coarse search results to the finer scale +Quality of disparity map is better +Even faster than single scale computation ¬Requires structured light Solution #1: Lighting and Multi-Scale SVL implementation (using structured light):

11. Solve a (spatially) global optimization with regularization –O(D) = min SAD(D) + Smooth(D) GLOBAL optimum found in polynomial time Solution #2: Dynamic Programming

12. 1.Defining the recursive cost function 2.Memoization 3.Finding lowest cost path, which is the disparity map (D M in red) SmoothnessError Solution #2: Dynamic Programming

13. Dynamic Programming on Images Minor issue: previous approach applies to scanline Approximate DP applied to entire image - 3D disparity space (D): - Cost function (C): - Memoization (P):

14. Dynamic Programming: Results

15. Dynamic Programming: In Vivo Results Stereo recordings from the da Vinci robot Focal length of ~ 700 pixels ~5mm baseline Distance to surface of 55mm to 154mm. Raw Disparity Map Textured 3D Model

16. Surface to 3D Model Registration Inputs: –point cloud from the stereo surface modeler –point cloud generated from a model or volume image Outputs: - transformation to register the 3D model to the 3D surface

17. Results: Rigid Registration Complete system (stereo plus registration) operates at 5 frames/second Current algorithm uses IPC with modifications to account for occlusions due to viewpoint (z-buffer)

18. From Rigid to Deformable Calculate residual errors in z direction Define a spring-mass system Perform local gradient descent

19. Deformable Registration Results Final registration error of < 1mm except for the area where the tool enters the image

20. Coming in CASA The Language of Surgery Tool Tracking Tissue Surface Classification

21. Thank you!

22. Telemanipulation with Integrated Laparoscopic Ultrasound for Hepatic Surgery Ultrasound probe examining artificial lesion in porcine liver with registered 2D ultrasound overlay Registered 3D ultrasound volume swept w/autonomous robot motion Needle insertion demonstrates alignment Collaboration between JHU and Intuitive Surgical, Inc.