An Open-Source Solution for Interactive Acquisition, Processing and Transfer of Interventional Ultrasound Images Jonathan Boisvert, David Gobbi, Siddharth Vikal, Robert Rohling, Gabor Fichtinger and Purang Abolmaesumi Medical Image Analysis (MedIA) Laboratory Queen’s University Kingston, Ontario, Canada

Introduction Ultrasound is a very common interventional image modality Ultrasound is a very common interventional image modality Open-interface machines makes it even more attractive to IGT developers Open-interface machines makes it even more attractive to IGT developers Often used with motion tracking devices. Often used with motion tracking devices. Problem: Problem: Acquisition, synchronization and transfer to existing IGT systems is more difficult than it should be. Acquisition, synchronization and transfer to existing IGT systems is more difficult than it should be.

Integrated navigation software such as: Integrated navigation software such as: CustusX 1, IGSonic 2, SonoNav 3 or Stradx 4. CustusX 1, IGSonic 2, SonoNav 3 or Stradx 4. Software frameworks such as: Software frameworks such as: IGSTK 5 IGSTK 5 Slicer specific software development Slicer specific software development A module for Slicer 2 initially developed by Boctor et al. 6 A module for Slicer 2 initially developed by Boctor et al. 6 Prior work

Sophisticated registration, data- fusion and visualization capabilities Sophisticated registration, data- fusion and visualization capabilities Modular architecture Modular architecture Multi-platform Multi-platform Support for the new OpenIGTLink protocol Support for the new OpenIGTLink protocol Open-source software platform: Slicer

Ultrasound Image Acquisition Ultrasound Image Acquisition 3D Tracking 3D Tracking Synchronization Synchronization Ultrasound Volumes Reconstruction Ultrasound Volumes Reconstruction Open-Source Open-Source Standardized communication with independent IGT platforms. Standardized communication with independent IGT platforms. Objectives

Architecture : SynchroGrab Provide real-time ultrasound processing capabilities. Provide real-time ultrasound processing capabilities. Delegate visualization, registration and navigation to an OpenIGTLink compatible software (Slicer). Delegate visualization, registration and navigation to an OpenIGTLink compatible software (Slicer).

Architecture : SynchroGrab Provide real-time ultrasound processing capabilities. Provide real-time ultrasound processing capabilities. Delegate visualization, registration and navigation to an OpenIGTLink compatible software (Slicer). Delegate visualization, registration and navigation to an OpenIGTLink compatible software (Slicer).

Architecture : SynchroGrab Provide real-time ultrasound processing capabilities. Provide real-time ultrasound processing capabilities. Delegate visualization, registration and navigation to an OpenIGTLink compatible software (Slicer). Delegate visualization, registration and navigation to an OpenIGTLink compatible software (Slicer).

Architecture : SynchroGrab Provide real-time ultrasound processing capabilities. Provide real-time ultrasound processing capabilities. Delegate visualization, registration and navigation to an OpenIGTLink compatible software (Slicer). Delegate visualization, registration and navigation to an OpenIGTLink compatible software (Slicer).

Architecture : SynchroGrab Provide real-time ultrasound processing capabilities. Provide real-time ultrasound processing capabilities. Delegate visualization, registration and navigation to an OpenIGTLink compatible software (Slicer). Delegate visualization, registration and navigation to an OpenIGTLink compatible software (Slicer).

SonixRP SonixRP Open interface Open interface Access to pre-scan converted images, post-scan converted images, imaging parameters, RF raw-data, etc. Access to pre-scan converted images, post-scan converted images, imaging parameters, RF raw-data, etc. New vtk class “vtkSonixVideoSource” New vtk class “vtkSonixVideoSource” Easy integration Easy integration Highly reusable Highly reusable Ultrasound image acquisition

Inherit from vtkVideoSource Inherit from vtkVideoSource Support many data types: Support many data types: BPreScan, BPost 8 bit, BPost 32 bit, RF 16 bit, MPost 8 bit, Color+ B 32 bits BPreScan, BPost 8 bit, BPost 32 bit, RF 16 bit, MPost 8 bit, Color+ B 32 bits vtkSonixVideoSource *sonixGrabber = vtkSonixVideoSource::New(); sonixGrabber->SetSonixIP(" "); sonixGrabber->SetImagingMode(BMode); sonixGrabber-> SetAcquisitionDataType(udtBPost); sonixGrabber->Record(); sonixGrabber->Stop(); sonixGrabber->Rewind(); // Process the first image here sonixGrabber->Seek(1) // Process the second image here sonixGrabber->Delete(); vtkSonixVideoSource *sonixGrabber = vtkSonixVideoSource::New(); sonixGrabber->SetSonixIP(" "); sonixGrabber->SetImagingMode(BMode); sonixGrabber-> SetAcquisitionDataType(udtBPost); sonixGrabber->Record(); sonixGrabber->Stop(); sonixGrabber->Rewind(); // Process the first image here sonixGrabber->Seek(1) // Process the second image here sonixGrabber->Delete(); Using vtkSonixVideoSource

Object oriented Object oriented Extensible Extensible Multi-threaded Multi-threaded Circular buffers for transformations synchronization Circular buffers for transformations synchronization 3D Tracking

Tracking and ultrasound imaging Tracking and ultrasound imaging not performed at the same frequency. not performed at the same frequency. Need to interpolate the poses (using spherical linear interpolation) Need to interpolate the poses (using spherical linear interpolation) Synchronization

Data preprocessing Data preprocessing Hardware latency, pose computation, ultrasound image processing, and copy to user-space memory. Hardware latency, pose computation, ultrasound image processing, and copy to user-space memory. OS related OS related Context switching, memory swapping, etc. Context switching, memory swapping, etc. Compensate for the mean lag Compensate for the mean lag Synchronization

vtkTaggedImageFilter vtkTaggedImageFilter Compute the transformation associated with an image. Compute the transformation associated with an image. Two inputs: a vtkTrackerTool and a vtkImageAlgorithm object Two inputs: a vtkTrackerTool and a vtkImageAlgorithm object vtkTaggedImageFilter *tagger = vtkTaggedImageFilter::New(); tagger->SetInput(sonixGrabber->GetOutput()); tagger->SetTrackerTool(trackerTool); tagger->SetTemporalLag(xxx); tagger->Update(); currentTransforms = tagger->GetTransform(); currentImage = tagger->GetOutput(); vtkTaggedImageFilter *tagger = vtkTaggedImageFilter::New(); tagger->SetInput(sonixGrabber->GetOutput()); tagger->SetTrackerTool(trackerTool); tagger->SetTemporalLag(xxx); tagger->Update(); currentTransforms = tagger->GetTransform(); currentImage = tagger->GetOutput(); Synchronization

Synchronized images and transformation Synchronized images and transformation Combined to produce 3D volumes Combined to produce 3D volumes Pixel-based reconstruction Pixel-based reconstruction Pixel nearest neighbors (PNN) Pixel nearest neighbors (PNN) Pixel trilinear interpolation (PTL) Pixel trilinear interpolation (PTL) 3D Reconstruction

SynchroGrab --calibration-file _ Calibration.results --save-images --use-tracker-transforms --nb-frames fps 5 Saving : usImage0.png Current rigid transform: Saving : usImage1.png Current rigid transform: Saving : usImage2.png Current rigid transform: …. Saving : usImage0.png Current rigid transform: Saving : usImage1.png Current rigid transform: Saving : usImage2.png Current rigid transform: …. Collecting images  Just start the application using the command line

SynchroGrab --calibration-file _ Calibration.results --reconstruct-volume --nb-frames fps 5 --use-tracker-transforms Reconstructing volumes   Just start the application using the command line

SynchroGrab --calibration-file _ Calibration.results --oigtl-transfert-images --nb-frames fps 5 Communication with Slicer Real-time acquisition using SynchroGrab. Visualization with Slicer 3.

Communication with Slicer

SynchroGrab SynchroGrab Acquisition and synchronization of tracking and ultrasound data Acquisition and synchronization of tracking and ultrasound data Can be used as a stand-alone command-line application. Can be used as a stand-alone command-line application. Can be used in conjunction to an OpenIGTLink compliant software. Can be used in conjunction to an OpenIGTLink compliant software. Development of temporal calibration protocols Development of temporal calibration protocols Future support for 3D probes Future support for 3D probes Conclusion & Future work

1.T. Lango, G. A. Tangen, R. Marvik, B. Ystgaard, Y. Yavuz, J. H. Kaspersen, O. V. Solberg, and T. A. N. Hernes. Navigation in laparoscopy–prototype research platform for improved image-guided surgery. Minim Invasive Ther Allied Technol, 17(1):17–33, BrainLAB (AG, Heimstetten, Germany). IGSonic Medtronic(Louisville, CO, U.S.). SonoNav Graham Treece, Richard Prager and Andrew Gee. Stradx/Stradwin Andinet Enquobahrie, Patrick Cheng, Kevin Gary, Luis Ibanez, David Gobbi, Frank Lindseth, Ziv Yaniv, Stephen Aylward, Julien Jomier, and Kevin Cleary. The image-guided surgery toolkit IGSTK: an open source C++ software toolkit. J Digit Imaging, 20 Suppl 1:21–33, E. M. Boctor, A. Viswanathan, S. Pieper, M. A. Choti, R. H. Taylor, R. Kikinis, and G. Fichtinger. CISUS: an integrated 3D ultrasound system for IGT using a modular tracking API. Proceedings of the SPIE, volume 5367, pages 247–256, May 2004 References

Natural Sciences and Engineering Research Council (NSERC), Natural Sciences and Engineering Research Council (NSERC), Canadian Institutes of Health Research (CIHR), Canadian Institutes of Health Research (CIHR), National Alliance in Medical Image Computing (NIH5U54EB ), National Alliance in Medical Image Computing (NIH5U54EB ), Fonds québécois de recherche sur la nature et les technologies. Fonds québécois de recherche sur la nature et les technologies. Acknowledgments

Questions