1. Seminar in Computer-Assisted Surgery Medical Robotics Medical Image Processing Lecture 2: Elements of CAS systems technical elements clinical applications example CAS systems in detail

2. Computer-Assisted Surgery Medical Robotics Medical Image Processing Lecture 2: Elements of CAS systems technical elements clinical applications example CAS systems in detail

3. CAS, Srping 2002 © L. Joskowicz 3 CAS clinical applications Neurosurgery Orthopaedics Maxillofacial, craneofacial, and dental surgery Laparoscopic and endoscopic surgeries Radiotherapy Specific procedures in ophtalmology, othorhinolaringology, etc.

4. CAS, Srping 2002 © L. Joskowicz 4 Elements of CAS systems

5. CAS, Srping 2002 © L. Joskowicz 5 Technical elements of CAS systems 1. Medical images 2. Medical image visualization 3. Segmentation and modeling 4. Virtual and augmented reality, tele-surgery 5. Preoperative analysis and planning 6. Image and robot registration 7. Medical mechanical and robotics systems 8. Real-time tracking 9. Safety, man-machine interface, human factors

6. CAS, Srping 2002 © L. Joskowicz 6 Key parameters for understanding and comparing solutions How many procedures are performed yearly? What is the rate of complications? What are their causes? In what aspects can a CAS system help? Does it address part of a clinically important problem? What stage is the system in: in-vitro, cadaver, clinical trials?

7. 1. Medical Images

8. CAS, Srping 2002 © L. Joskowicz 8 Most common imaging modalities Film X-ray, Digital X-ray, Fluoroscopy, Digital Substraction Angiography (DSA) Ultrasound -- 2D and 2.5D (stack of slices) Computed Tomography (CT) Magnetic Resonance Imaging (MRI) Nuclear Medicine (NM) –PET -- Positron Emission Tomography –SPECT -- Single Photon Emission Tomography

9. CAS, Srping 2002 © L. Joskowicz 9 Medical images: characteristics (1) Preoperative or intraoperative use –depends on the size and location of imaging machine Dimensionality: 2D, 2.5D, 2D+time –projection, cross section, stack of projections, time sequence Image quality – pixel intensity and spatial resolution – amount of noise; signal/noise ratio –spatial distortions and intensity bias

10. CAS, Srping 2002 © L. Joskowicz 10 Medical images: characteristics (2) Field of view Radiation to patient and to surgeon Functional or anatomical imaging –neurological activity, blood flow, cardiac activity What it’s best at for –bone, soft tissue, fetus, surface/deep tumors, etc Clinical use –diagnosis, surgical, navigation,

11. CAS, Srping 2002 © L. Joskowicz 11 X-ray images Measure absorption of x-ray radiation from source to set of receptors Film X-ray has very high resolution Gray value proportional to radiation energy

12. CAS, Srping 2002 © L. Joskowicz 12 X-ray Fluoroscopy

13. CAS, Srping 2002 © L. Joskowicz 13 Fluoroscopic images

14. CAS, Srping 2002 © L. Joskowicz 14 X-ray image properties Traditional, cheap, widely available Two-dimensional projections (at least two required) High resolution, low noise (more fluoroscope) –film size, 64K gray levels –fluoroscopic images: TV quality, 20cm field of view Relatively low radiation Bone and metal images very well Fluoroscopy used for intraoperative navigation

15. CAS, Srping 2002 © L. Joskowicz 15 Ultrasound imaging (US) Measure refraction properties of an ultrasound wave as it hits tissue No radiation Poor resolution, distortion, noise Low penetration properties One 2D slice or several slices (2.5D) Relatively cheap and easy to use Preoperative and intraoperative use

16. CAS, Srping 2002 © L. Joskowicz 16 Ultrasound imaging

17. CAS, Srping 2002 © L. Joskowicz 17 Computed Tomography (CT)

18. CAS, Srping 2002 © L. Joskowicz 18 Computed Tomography Images cuts d = 35mmd = 25mm d = 15mm d = 5mm

19. CAS, Srping 2002 © L. Joskowicz 19 Computed Tomography Principle angle intensity X-rays

20. CAS, Srping 2002 © L. Joskowicz 20 Computed Tomography Properties Sepcifications: –512x512 12bit gray level images; pixel size 0.5mm –slice interval 1-10mm depending on anatomy –50-200 slices per study –noise in the presence of metal (blooming) All digital, printed on X-ray film Acquisition 1sec/slice (spiral models) 15mins for image reconstruction Costs about $250-750K, each study $500

21. CAS, Srping 2002 © L. Joskowicz 21 Magnetic Resonance Imaging Similar principle and construction than CT machine, but works on magnetic properties of matter –magnetic fields of 0.1 to 4 Teslas Similar image quality characteristics as CT Excellent resolution for soft tissue Costs $1-2M, each study $1,000 Open MR: intraoperative device (only 15 to date)

22. CAS, Srping 2002 © L. Joskowicz 22 Magnetic Resonance Images

23. CAS, Srping 2002 © L. Joskowicz 23 Nuclear Medicine Imaging (NMI) Same slices principle Source of photons or positrons is injected in the body. Shortly after, radiation of metabolism is measured Poor spatial resolution Expensive machine AND installation ($4-5M) Expensive and time-consuming Provides functional info no other source does

24. CAS, Srping 2002 © L. Joskowicz 24 Nuclear medicine images

25. CAS, Srping 2002 © L. Joskowicz 25 Image Fusion: MRI and NMI MRI (anatomy) NMI (functional)

26. CAS, Srping 2002 © L. Joskowicz 26 Video images from within the body Used in laparoscopic and endoscopic surgery

27. CAS, Srping 2002 © L. Joskowicz 27 Main medical imaging modalities X-ray X-ray Fluoro US US Video CT MRI NMR Open Film Digital (2D) (2.5D) MR Pre/Intraop 2D/2.5D Resolution Radiation Anatomy Procedure Establish a comparative table of modality properties

28. CAS, Srping 2002 © L. Joskowicz 28 The imaging pipeline

29. CAS, Srping 2002 © L. Joskowicz 29 2. Medical image visualization 3D visualization of complex structures image correlation and fusion quantitative measurements and comparisons visualization of medical and CAD data Enhance diagnosis by improving the visual interpretation of medical data

30. CAS, Srping 2002 © L. Joskowicz 30 Medical image visualization

31. CAS, Srping 2002 © L. Joskowicz 31 Visualization: Technical needs image enhancing and noise reduction image interpolation: images from new viewpoints 3D visualization from 2.5D data –volume rendering: display voxels and opacity values –surface rendering: explicit reconstruction of surface 3D modeling from 2.5D data 2D and 3D segmentation 3D+T visualization (beating heart)

32. CAS, Srping 2002 © L. Joskowicz 32 Medical image visualization Much activity! Radiologists are the experts Commercial packages – 3DVIEWNIX, ANALYZE, IMIPS Main technical topics: –3D volume rendering techniques –3D image filtering and enhancement –surface construction algorithms: Marching cubes, etc. Sources: chapters 3,9, and 10 in textbook Related fields: computer graphics, image processing

33. CAS, Srping 2002 © L. Joskowicz 33 3. Segmentation and modeling Isolation of relevant anatomical structures based on pixel properties Model creation for the next computational task –real-time interaction and visualization –simulation –registration, matching, –morphing Extract clinically useful information for a given task or procedure

34. CAS, Srping 2002 © L. Joskowicz 34 Segmentation and modeling

35. CAS, Srping 2002 © L. Joskowicz 35 Segmentation and modeling: technical needs Segmentation: – landmark feature detection –isosurface construction (Marching cubes) –contour extraction, region identification Modeling: –points, anatomical landmarks, surface ridges –surfaces as polygon meshes, surface splines –model simplification methods (Alligator, Wrapper)

36. CAS, Srping 2002 © L. Joskowicz 36 Segmentation and modeling Medical images have very special needs! Commercial packages – 3DVIEWNIX, ANALYZE, IMIPS Main technical topics: –Volumetric segmentation techniques for CT, MRI –2D and 3D segmentation with deformable elements –surface and model simplification algorithms Sources: chapters 4 and 8 in textbook Related fields: image processing, computer vision

37. CAS, Srping 2002 © L. Joskowicz 37 4. Virtual and augmented reality Create a virtual model for viewing during surgery Project the model on the patient or integrate with surgeon’s view Useful for intraoperative anatomy exploration and manipulation Telesurgery systems Use images to create or enhance a surgical situation

38. CAS, Srping 2002 © L. Joskowicz 38 Virtual and augmented reality

39. CAS, Srping 2002 © L. Joskowicz 39 Virtual and augmented reality Part manipulation, visual and sensory feedback Interaction devices: goggles, gloves, etc Only a handful of systems exist Main technical topics: –a couple of the working systems; simulators –telesurgery systems Sources: chapters 14 and 15 in textbook Related fields: computer graphics

40. CAS, Srping 2002 © L. Joskowicz 40 5. Preoperative analysis and planning Task and procedure dependent Spatial and volume measurements Stress and fracture analysis Implant and tool selection and positioning Surgical approach planning: bone rearrangement, angle evaluation, radiation dose planning, etc Use images and models to assist surgeons in planning a surgery and evaluate options

41. CAS, Srping 2002 © L. Joskowicz 41 Preoperative analysis and planning

42. CAS, Srping 2002 © L. Joskowicz 42 Preoperative analysis and planning About a dozen planners exist for different procedures Main technical topics: –planning systems for orthopaedics, neurosurgery –application of engineering analysis techniques: finite- element methods, stress analysis, etc Sources: chapters 11, 25, 33, 41, 52--56in textbook Related fields: CAD, computational geometry, engineering analysis

43. CAS, Srping 2002 © L. Joskowicz 43 6. Image and robot registration Define correspondance features –point-to-point, point-to-line, surface-to-surface Establish correspondances between features Establish a similarity measure Formulate and solve dissimilarity reduction problem Related tasks: image fusion, morphing, atlas matching Establish a quantitative relation between different refererence frames

44. CAS, Srping 2002 © L. Joskowicz 44 Multimodal registration problems Great differences depending on –the type of data to be matched –the anatomy that is being imaged –the specific clinical requirements of procedures Feature selection and extraction: stereotactic frame, implanted fiducials, anatomical landmarks and surfaces, contours and surfaces in Manual vs. automatic feature selection, pairing Rigid vs. deformable registration Nearly similar vs. dissimilar images Noiseless vs. noisy images (outlier removal)

45. CAS, Srping 2002 © L. Joskowicz 45 Registration chain example 3D surface model X-rays CT Patient Tracker Instruments Infrared tracker

46. CAS, Srping 2002 © L. Joskowicz 46 Image and robot registration Rich topic of very great importance! Types of registration methods vary widely Main technical topics: rigid registration methods: three points and more –deformable registration: local and global methods –intensity-based registration Sources: chapters 5-7 in textbook, many papers Book on Medical Image Registration Related fields: vision, robotics

47. CAS, Srping 2002 © L. Joskowicz 47 7. Medical robotics devices Task and procedure dependent Accurate, steady, and repeatable 3D positioning Navigation and localization aids Cutting and milling, biopsies Key issues are: –kinematic design, trajectory planner –controller, safety provisions Semi-active and active mechanical devices for improving surgical outcome

48. CAS, Srping 2002 © L. Joskowicz 48 Medical robotics devices

49. CAS, Srping 2002 © L. Joskowicz 49 Medical robotics devices Mostly passive and semiactive devices Rich topic of very great importance! Main technical topics: –compare features and functionalities of systems –discuss and compare design considerations –devices for specific surgeries laparoscopy) Sources: chapters 16-18, 22, 29, 34, 39, 45, 47, and 48 in textbook Related fields: robotics, mechatronics

50. CAS, Srping 2002 © L. Joskowicz 50 8. Real-Time tracking devices Ideally, an accurate Global Positioning System! Current technologies offer only partial solution Based on different principles –video: follow known objects –optical: follow light-emitting diodes –magnetic: measure the variation of –acoustic: works like a radar Hardware to follow in real time the precise position and orientation of anatomy and instruments during surgery

51. CAS, Srping 2002 © L. Joskowicz 51 Optical and video tracking devices camera instrument Passive markers Instrument has infrared LEDs attached to it Active markers

52. CAS, Srping 2002 © L. Joskowicz 52 What kind of accuracy?

53. CAS, Srping 2002 © L. Joskowicz 53 9. Safety, man-machine interfaces Medical systems have very stringent safety requirements Reported cases of radiation overdose due to faulty system design Important issues in man-machine interfaces Ideas for presentations –the radiotherapy accident –chapters 12-15 and 19 in textbook

54. CAS, Srping 2002 © L. Joskowicz 54 10. Systems integration Complete systems that address specific clinical problems in domains Use available technology to develop the system The hard part: make it all work! Main technical topics: –systems in orthopaedics, neurosurgery, etc Sources: chapters in each section of Related fields: all!