Tracking Objects of Interest from CT/MR Data using Dynamic 3D Ultrasound Group 6: Michael Shetliffe Mohammad Yaqub Mohammed Alam

Outline Problem Statement Problem Statement Background & Significance Background & Significance Overall Aims Overall Aims Methods Methods Ultrasound Ultrasound MRI MRI Registration Registration Results Results Problems Problems Solutions Solutions Competition Competition Future Work Future Work Conclusion Conclusion

Problem Statement

Problem There is a critical need to update information based on changes occurring during surgery. Changes: Shift Shift Deformation Deformation Vascular movement Vascular movement

Overview of Proposal The overall objective of this proposal is to show feasibility and develop a cost-effective and efficient approach to monitor and predict deformation during surgery, allowing accurate, and real-time intra-operative information to be provided reliably to the surgeon. The central hypothesis is that deformation can be followed intra-operatively using ultrasound technology.

Background & Significance Approaches to update the pre-op data: Intra-operative MR Intra-operative MR Obtain updated (lower-resolution) MR data Obtain updated (lower-resolution) MR data Costly, significant setup time, OR compatibility considerations Costly, significant setup time, OR compatibility considerations Finite Element modeling of surrounding tissues Finite Element modeling of surrounding tissues Challenges of parameter estimation Challenges of parameter estimation Parameter variation with physiologic changes Parameter variation with physiologic changes Long computation times Long computation times Use 3D Ultrasound Use 3D Ultrasound Convenient, safe, cheap Convenient, safe, cheap

Overall Aims CT US MR Registration Evaluate the effectiveness, accuracy, and usefulness of any techniques that were used Testing the method Image Acquisition Segmentation

Image Acquisition – 3D Ultrasound

What is Ultrasound ? High-energy sound waves (ultrasound) are bounced off internal tissues or organs and make echoes. (2-13 MHz) Pros: Non invasive Non invasive Real time imaging Real time imaging Cons: Cannot image bony structures Cannot image bony structures Poor native resolution Poor native resolution Image depends on time to echo (pixel position) and echo strength (pixel intensity)

Ultrasound Images (Breast) Objects of Interest Spurious Artifacts Bench Surface Reflection Characteristic US Noise

3D Ultrasound – Background 3D Ultrasound: 3D Ultrasound: Tracks 2D ultrasound probe to build 3D volume from ultrasound “slices” Tracks 2D ultrasound probe to build 3D volume from ultrasound “slices” Can use conventional, portable ultrasound equipment to obtain 3D volumes Can use conventional, portable ultrasound equipment to obtain 3D volumes Still relies on ultrasonic backscattering from tissue structures  intrinsically noisy Still relies on ultrasonic backscattering from tissue structures  intrinsically noisy  Image Processing becomes especially important  Image Processing becomes especially important

3D Ultrasound - System 2D Ultrasound Polaris Tracking System “Stradx” Software Slices in 3D Space 2D Screen Frames Probe 3D Position Frames / sec & Probe Motion

3D Ultrasound - Output Output from Stradx: Output from Stradx:.sx file: Text-based file containing 3D positions, orientations, sizes, times, etc. of individual 2D slices.sx file: Text-based file containing 3D positions, orientations, sizes, times, etc. of individual 2D slices.sxi file: Binary file of 8-bit pixel values.sxi file: Binary file of 8-bit pixel values What we need: 3D image data in “conventional” format (e.g. dicom, Analyze) that can be read into other systems for processing 3D image data in “conventional” format (e.g. dicom, Analyze) that can be read into other systems for processing (Prior to starting this project, it was thought that this was what we would have.) (Prior to starting this project, it was thought that this was what we would have.)

3D Ultrasound - Output Examples: Examples: 1-Pass Probe Motion

3D Ultrasound - Output Examples: 2-pass had problems with “inter-pass” alignment Examples: 2-pass had problems with “inter-pass” alignment 2-Pass Probe Motion

3D Ultrasound - Post Processing

2 Additional Tools (Provided as part of the Stradx distribution): 2 Additional Tools (Provided as part of the Stradx distribution): SelectSX: (adjust for data too “dense”) SelectSX: (adjust for data too “dense”) StackSX: (create evenly spaced, uniformly aligned slices) StackSX: (create evenly spaced, uniformly aligned slices) 3D Ultrasound - Conversion

Read new.sx file as “Raw” 8-bit data into a medical imaging tool (xmedcon, osirix, 3d-slicer) Read new.sx file as “Raw” 8-bit data into a medical imaging tool (xmedcon, osirix, 3d-slicer) Region of Interest moves within resliced image between slices.

3D Ultrasound - Conversion At this point, we can treat our US data as a standard set of images, and export to other convenient formats At this point, we can treat our US data as a standard set of images, and export to other convenient formats Slice separation distance determined from.sx file position data.

3D Ultrasound – Post-Processing Basic Objectives: Basic Objectives: Reduce background noise Reduce background noise Segment (or at least highlight) object(s) of interest Segment (or at least highlight) object(s) of interestApproach: Stradx has some inbuilt segmentation/registration capabilities – not used Stradx has some inbuilt segmentation/registration capabilities – not used Used MATLAB to investigate feasible techniques that could later be integrated into a stand-alone system. Used MATLAB to investigate feasible techniques that could later be integrated into a stand-alone system.

3D Ultrasound – Post-Processing Noise Reduction Techniques: Noise Reduction Techniques: Original Image Median FilterGaussian Blur

3D Ultrasound – Post-Processing Noise Reduction Techniques: Noise Reduction Techniques: Original Image Median FilterGaussian Blur

3D Ultrasound – Post-Processing Noise Reduction Techniques: Noise Reduction Techniques: Original Image Median FilterGaussian Blur

3D Ultrasound – Summary In this portion of work we have: In this portion of work we have: Acquired updated ultrasound image data that better meets our requirements for working with objects of interest. Acquired updated ultrasound image data that better meets our requirements for working with objects of interest. Converted the image data to formats that can be easily used in a wide variety of medical imaging systems. Converted the image data to formats that can be easily used in a wide variety of medical imaging systems. Processed the ultrasound image data to enable higher quality results from subsequent registration with other modalities. Processed the ultrasound image data to enable higher quality results from subsequent registration with other modalities.

Image Acquisition - MRI / CT

What is MRI? MRI stands for Magnetic Resonance Imaging. The MR images used to image internal structures of the body, particularly the soft tissues. We used GE 4 tesla MRI machine to get some MR images for the phantom we have. MR brain image

GE MRI machine

What is CT? CT stands for Computed Tomography. CT image is a specialized form of x-ray imaging. It shows bony structures. We used CT images in our project. It did not give good results. CT brain image

MRI/CT Images (Breast) MR T-2 Images MR T-1 Images CT Images No big difference because the phantom is MR incompatibleNo internal information

Image Segmentation MRI

MRI Segmentation No quantitative phantom information. Introduce artificial shift of objects of interest. We did the movement using a manual linear interpolation method. We did the movement using a manual linear interpolation method.Segmentation Object matching Object matching Thresholding Thresholding

Example Original MRI sliceManually shifted object of interest

Example (cont.) Original segmented object of interest inside an MR image Automatically segmented object of interest inside the shifted MR image

Image Registration

Registration Registering different modalities: Original MR to shifted MR Original MR to shifted MR Original MR to CT Original MR to CT Original MR to Ultrasound Original MR to Ultrasound Manual landmark. An automated registration - mutual information. Need more human interaction.

Example Original fixed modalityMoving modality (The Shifted object of interest)

Example (cont.) A slice that contains the registered data Showing both the original & the shifted points Original object of interest shifted object of interest

Example (cont.) MR image with three objectsUltrasound image with three objects Need for deformable registration method

Problems, Proposed Solutions and Future Work

ProblemsPhantom Echo-sensitive material Echo-sensitive material Water Content Water Content MRI Compatible MRI Compatible Atlas AtlasUltrasound Quality of Images Quality of Images System Calibration System CalibrationRegistration

Proposed Solutions - Phantoms Multi modality Phantoms - $

Proposed Solutions - Ultrasound Better Ultrasound Transducer

Proposed Solutions - Registration Existing Research

Competitors Ultrasound Probes - Philips, Siemens, GE Registration Techniques – Related research available Overall Intraoperative Tracking System using Ultrasound – GE Overall Intraoperative Tracking System using Ultrasound +preop MRI - ????

Future Work Research areas Better post processing of US images Better post processing of US images Newer registration techniques Newer registration techniques Faster and effective calibration methods Faster and effective calibration methods Bring all individual modules to work as a single system Testing and evaluation techniques

Conclusion A very feasible and highly applicable research area. Cost effective when compared to I-MRI Cost effective when compared to I-MRI Relatively “real time” data Relatively “real time” data Accuracy from fusion with pre-op MR Accuracy from fusion with pre-op MR Improved surgical outcome Improved surgical outcome Works in current clinical settings Works in current clinical settings Adaptable to other surgical procedures involving: Adaptable to other surgical procedures involving: Brain Brain Breast Breast Prostate Prostate Others … Others …

Thank You!