1. Video Overlay Advanced Computer Integrated Surgery (600.446) Jeff Hsin, Cyrus Moon, Anand Viswanathan

2. Outline Synopsis of Project Present Work Accomplished Exposures, Dependencies Future Plans

3. Image Overlay

4. Image/Video Overlay Image Overlay: The ability to see bone/tissue/target superimposed onto the patient during surgery through a half-silver mirror. Video Overlay: The ability to superimpose a computer generated image on a desired object within a live video feed.

5. Original Plan Tracker code: –Testing3/18 Networking Code (SPREAD/CORBA): –Integration/Testing3/18-3/25 Graphics Code: –diagram, frames and transformations3/11 –framework3/24 –ability to track/render sphere:3/25 –tool object creation3/25 –tool calibration4/? –video overlay4/7

6. Plan Changes Doing video overlay (instead of image overlay) is definite Using CORBA instead of SPREAD (using CIS tracker class) Extended schedule

7. Updated Plan Tracker code: –Testing3/18 Networking Code (SPREAD/CORBA): –Integration/Testing4/13* Graphics Code: –diagram, frames and transformations3/11 –framework3/25 –primitive tracking/rendering:4/15* –tool object creation3/25 –tool calibration4/? –video input4/15* –camera calibration? –Working video overlay system4/22* Blue – Completed * – Changed Date

8. Software Components Tracking Networking (CORBA) Video Input Rendering

9. Software Flow Chart Video Software Network Software Tracking Software Graphics Software Tracker Input Camera Input Overlay Image

10. Tracking Functional Optotracker class Misc. Programs: –RigidBody Definition procedure –Temporary Pointer Calibration –DefineCurve function

11. OPTOTRAK measurement system that tracks small infrared (IRED) markers that are attached to a subject or object. active marker based technology. Real-time data that is very accurate.

12. Tracking Libraries Objective: Create an object oriented library for the OptoTrack for the CIS libraries. OptoTracker class implements the Tracker Class. OptoTracker class can track multiple rigid bodies as well as individual markers and returns frames and 3D positions in the OptoTrack coordinate system.

13. Optotracker class OptoTracker() ~OptoTracker() void updateToolSamples(char* save_filename) cis_status_t setConfiguration (char* config_filename) void DefineRigidBody(char* save_filename) cis_status_t Initialize() cis_status_t DestructOpto()

14. CORBA Networking Reasons for choosing CORBA –Ability to use on any computer –use C++ for the implementation An Optotracker object is created on the server, then the object is transported to the client. This can then be used for multiple clients: –one client  controller –other clients  monitors

15. Video Implementation Calibrate Camera lens (Dewarp Camera data) Access the SGI Video Stream in order to acquire a continuous video buffer. Integrate the video buffer with the graphics code.

16. Graphics Software Outline Input-Frames of tracked objects (from tracker software), stored video data For each object, calculates object-to-lens transform; renders the object in the appropriate position/orientation in camera-space renders video image at a constant position/orientation in camera space –Texture mapping, frame buffer Outputs Video/Graphical image

17. Coordinate Graph (Video Overlay) Patient markers Model (from Imaging) World (Tracker) Tool 1Tool k.... Camera Lens Screen *Dynamic Transforms (determined by Optotrak)

18. Current/Future Problems Integration of software modules –Synchronization of Video and Graphics Learning/Using existing libraries Camera calibration Generalization of the graphics software –Tree implementation of the scene graph Class work

19. Dependencies Logistical help – ERC staff CORBA software – Sadi Mentors – Dr. Cohen, Dr. Taylor Calibration rigid body – rigid corner, dimple Object to be tracked – pelvis