Ultrasound Visualization Pipeline A Survey Å. Birkeland, V. Šoltészová, D. Hönigmann, O. H. Gilja, S. Brekke, T. Ropinski and I. Viola 1University of Bergen, Norway 2n22 Research & Technology Transfer, Wiener Neustadt, Austria 3National Centre for Ultrasound in Gastroenterology, Haukeland University Hospital, Norway 4University of Münster, Germany

Definition of Visualization the use of (usually) computer graphics to reveal insight into data to a user to form a mental vision, image, or picture of (something not visible or present to the sight, or of an abstraction); to make visible to the mind or imagination [Oxford Engl. Dict., 1989] computer graphics, but not photo-realistic rendering The purpose of computing is insight, not numbers [R. Hamming, 1962] The purpose of visualization is insight, not pictures [B. Schneidermann, 1999] Åsmund Birkeland, University of Bergen

Ultrasound Imaging Modality Non-invasive Relatively inexpensive High resolution Spatially Temporally Noise Random Speckle Difficult to interpret Åsmund Birkeland, University of Bergen

Usage of Medical Ultrasound Diagnostics Intraoperative imaging Intervention Diagnostically Fetal examinations Functonal heart examination Intra-operatively biopsies Intervention drug delivery, Åsmund Birkeland, University of Bergen

Ultrasound Image Modality 2D ultrasound 3D ultrasound 3D freehand ultrasound Dedicated 3D ultrasound probe 4D ultrasound Functional imaging Doppler B-flow Strain Contrast enhanced US Åsmund Birkeland, University of Bergen

Ultrasound Visualization – The Pipeline Pre-processing: Processing ultrasound data prior to segmentation, registration or rendering Segmentation: Extracting features from ultrasound data Registration: Combining ultrasound data with other types of medical imaging modalities Rendering: Presenting ultrasound data Augmented Reality: Combining ultrasound rendering with the real world Pre-processing Registration Rendering Augmented Reality The essential parts of the visualization pipeline Registration Åsmund Birkeland, University of Bergen

Ultrasound Visualization - The Pipeline 60 papers Different classifications Åsmund Birkeland, University of Bergen

Åsmund Birkeland, University of Bergen Pre-processing Scan-conversion Reconstruction of freehand ultrasound Data Enhancement Reconstruction – interpolation between images Filtering – Noise removal prior to segmentation Data enhancement prior to registration Gee et al. Processing and Visualizing Three-Dimensional Ultrasound Data Åsmund Birkeland, University of Bergen

Åsmund Birkeland, University of Bergen Segmentation Automated assessment of ovarian follicles using a novel three-dimensional ultrasound software MagiCut – Interactive clipping for 3D ultrasound One autmated techniques in US workstations Clipping tools are common – planar – MagiCut Åsmund Birkeland, University of Bergen

Vessel Extraction from Ultrasound Ultrasound Painting of Vascular Tree Quickly extract 3D models live during examination Minimize interaction Using basic 2D B-mode ultrasound + tracking Åsmund Birkeland, University of Bergen

Vessel Extraction from Ultrasound

Åsmund Birkeland, University of Bergen Registration Combining two or more image modalities in the same reference frame Rigid Registration Non-Rigid Registration Difficult due to nature of imaging modalities Feature based Olesch et al. - Matching CT and Ultrasound data of the Liver by Landmark constrained Image Registration 2009 Image based W. Wein et al. - Automatic CT-ultrasound registration for diagnostic imaging and image-guided intervention Olesch et al. Matching CT and Ultrasound data of the Liver by Landmark constrained Image Registration Wiein et al Automatic CT-ultrasound registration for diagnostic imaging and image-guided intervention Åsmund Birkeland, University of Bergen

Åsmund Birkeland, University of Bergen Rendering Surface Rendering 3D freehand rendering 4D ultrasound Difficulties with ultrasound Paolo - Fattal and Lischinski Variational Classification for Visualization of 3D Ultrasound Data Åsmund Birkeland, University of Bergen

Åsmund Birkeland, University of Bergen Volume: Set of Slices Åsmund Birkeland, University of Bergen

Rendering – Volume Visualization 3D→2D, e.g., a density volume from CT etc. direct volume rendering (semi-transparent volume) Åsmund Birkeland, University of Bergen

Rendering – Transfer Functions Mapping data to color and opacity Non-regular data intensities for the same tissue Linear OTF Manually designed piecewice linear OTF Adaptively designed OTF Tube-cores – selection of voxels along the viewing direction Hönigmann et al. Adaptive Design of a Global Opacity Transfer Function for Direct Volume Rendering of Ultrasound Data Åsmund Birkeland, University of Bergen

Rendering – Multi-Modal Doppler MRI / CT Increase in image clutter Petersch et al. – Blood flow in its context: Combining 3D B-Mode and Color doppler Ultrasonic Data 2007 Viola et al. - Illustrated Ultrasound for Multimodal Data Interpretation of Liver Examinations Viola et al. Illustrated Ultrasound for Multimodal Data Interpretation of Liver Examinations Petersch et al. Blood flow in its context: Combining 3D B-Mode and Color doppler Ultrasonic Data Åsmund Birkeland, University of Bergen

Åsmund Birkeland, University of Bergen Rendering - Shading Ropinski et al. Interactive Volumetric Lighting Simulating Scattering and Shadowing Šoltészová et al. Multi-Directional Occlusion shading Chromatic Depth coloring Solteszova – Multi-Directional occlusion shading 2010 Ropinksi - Interactive Volumetric Lighting Simulating Scattering and Shadowing Chromatic Depth Coloring Åsmund Birkeland, University of Bergen

Ultrasound and Augmented Reality Top: early work in augmented ultrasound: Merging Virtual Objects with the Real World: Seeing Ultrasound Imagery within the Patient Shelton et al. - Ultrasound Visualization with the Sonic Flashlight Sato: Image Guidance of Breast Cancer Surgery Using 3-D Ultrasound Images and Augmented Reality Visualization Åsmund Birkeland, University of Bergen Åsmund Birkeland, University of Bergen 19

Åsmund Birkeland, University of Bergen Conclusion Story Åsmund Birkeland, University of Bergen

Åsmund Birkeland, University of Bergen Acknowledgements Illustrasound Project, VERDIKT, The Norwegian Research Council Helwig Hauser – VisGroup, University of Bergen, Norway Wolfgang Wein - White Lion Technologies, Munich, Germany Åsmund Birkeland, University of Bergen

Åsmund Birkeland, University of Bergen Questions? Thank you! Åsmund Birkeland, University of Bergen