Three-Dimension (3D) Whole-slide Histological Image Analytics Yanhui Liang Department of Biomedical Informatics Department of Computer Science The Pathology Informatics Summit 2016, May 23-26, 2016

Pathology Analytical Imaging Glass Slides Scanning Whole Slide Images Image Analysis As we all know, digital pathology images are more and more adopted in research. Provide rich information about morphological and functional characteristics of biological systems Have tremendous potential for understanding diseases and supporting diagnosis Introduction

Problems Digital Pathology Image analysis From raw pixels to 2D and 3D meaningful structures Data management and analytics Explore global patterns and spatial relationships In order to fully use these digital pathology images to help clinicians better understand diseases, we get two big problems. The first one is image analysis. From the pixels in images, we need to extract meaningful structures, both in 2D and 3D, for the following analysis. After we get these structures or objects, we perform spatial analytics to explore the spatial relationships between the micro-anatomic objects and discover the global spatial patterns as they are essential to understand the disease progression. How the disease propagates. In practice, we have these two issues for both 2D and 3D pathology imaging. Introduction

3D Vessel Branch and Nuclei 3D Pathology Imaging 3D pathology imaging enables stacking serial sections, which have significant potential to enhance the study of both normal and disease processes Compared to 2D objects, provide more accurate shapes, structures, spatial relationships Provide 3D volumetric models for 3D printing 3D reconstruction of cellular level objects is a critical step significant potential to enhance the study of both normal and disease processes from structural changes or spatial relationships WSIs from serial sections have significant potential to enhance the study of both normal and disease processes 3D reconstruction of cellular level objects is a critical step A Reconstructed 3D Vessel Branch and Nuclei Human Liver 3D WSI Volume Introduction

Challenges Explosion of derived data Image analysis Spatial analytics 10 billion pixels and 1 million objects per slice Tens of millions 3D objects in a 3D volume with hundreds of slices Image analysis Multiple types, large-scale Large variations in shapes, complex structures Spatial analytics High computational complexity Multi-dimensional (3D) Heavy duty geometric computations However, in reality, to solve the above two problems are quite challenging. The first challenge is the data size. One WSI may have 10 billion pixels and contains 1 million objects. This is real “big data” we need to handle. And the geometric computations on these 3D objects have pretty high time and space complexity The second challenge is to develop a generic image analysis framework. As we have seen in previous slides, in each pathology image, there may have several different types of micro-anatomic objects and their shapes are irregular and totally different. So it’s difficult to develop an efficient segmentation method to extract the boundaries. Besides, when we perform object matching between adjacent slides for 3D reconstruction, due to the large variations in shape and complicated structures, traditional multi-objects tracking doesn’t work well for our case. The last challenge we have is the high computational complexity in our spatial analytics of the extracted objects from pathology image. As we may extract hundrands of features for each extracted object, each object is stored as multi-dimension data. Also, when try to explore their spatial relationships or spatial patterns, the geometric computations are too expensive to be directly used in real applications. Thus we need scalable and cost-effective systems to perform the spatial analytics Introduction

Our work Quantitative analysis of whole slide images to derive spatial structures and features in 3D, and perform spatial analytics on derived 3D micro-anatomic objects such as blood vessels and nuclei A framework for 3D primary blood vessel reconstruction A system for 3D spatial analytics on 3D blood vessels and nuclei Motivated by the above problems and challenges, in our paper, we present a …… We are interested in 3D objects because compared to 2D, 3D present the actual strucctus of micro-annotomic object in biolgocial systems and keep the real spatial relationships. In this paper, we mainly focus on spatial relationship based queries, including spatial joins: two-way and multi-way joins, window based queries and point-in-polygon queries. A special case of spatial join is polygon overlay or spatial cross-matching queries. Spatial cross-matching is to identify and compare derived spatial objects belonging to different analyses or measurements. Such query could involve large number of spatial objects to compare and are highly expensive to process. Introduction

3D Primary Vessel Reconstruction 3D WSI Volume Image Registration Vessel Association Image Segmentation Vessel Interpolation 3D Vessel Rendering 3D Pathology Image Analysis

3D Vessel Reconstruction Pathology image volume A stack of 2D sequential image slides Image registration Align all slices into the same coordinate system Image segmentation Vessel Directed Fitting Energy (VDFE) within a variational level set framework Tissue slide, scan deformations IHC, CD34 as blood vessel biomarker, x y 0.25 micrometers, z 50 micrometers 54 slides Perform image registration to align all slide sections in the same coordinate system. As mentioned above, we need to register all slides to the .. Rigid and non-rigid are used in this step. As we can see, after registraion, there are some translations, rotaions and non-rigid transformations in the registerd slide. Prior information on vessel wall probability : Vessel directed fitting energe 3D Pathology Image Analysis

Vessel Segmentation Results Final Result Tissue slide, scan deformations Perform image registration to align all slide sections in the same coordinate system. As mentioned above, we need to register all slides to the .. Rigid and non-rigid are used in this step. As we can see, after registraion, there are some translations, rotaions and non-rigid transformations in the registerd slide. 3D Pathology Image Analysis

3D Vessel Reconstruction Vessel association Four association cases Two stages: local bi-slide mapping and global vessel structure association Extension Bifurcation Emergence Disappearance Tissue slide, scan deformations Perform image registration to align all slide sections in the same coordinate system. As mentioned above, we need to register all slides to the .. Rigid and non-rigid are used in this step. As we can see, after registraion, there are some translations, rotaions and non-rigid transformations in the registerd slide. Prior information on vessel wall probability : Vessel directed fitting energe 3D Pathology Image Analysis

3D Vessel Reconstruction Vessel interpolation Make transition more smooth 3D vessel rendering Extract the iso-surface of the volumes from 2D vessel objects Original slice t Original slice t+1 Interpolated slices Tissue slide, scan deformations Perform image registration to align all slide sections in the same coordinate system. As mentioned above, we need to register all slides to the .. Rigid and non-rigid are used in this step. As we can see, after registraion, there are some translations, rotaions and non-rigid transformations in the registerd slide. Prior information on vessel wall probability : Vessel directed fitting energe 3D Pathology Image Analysis

3D Vessel Rendering 1 2 3D Pathology Image Analysis

Database for Spatial Queries and Analytics To store, manage and analyze the derived 3D cellular objects such as blood vessels and nuclei For each 3D nucleus, return the distance to its nearest blood vessel Scalable, efficient and cost-effective system for 3D spatial objects analytics based on Hadoop 3D Data partitioning for parallel processing Indexing to speed up spatial access 3D query engine to support various spatial queries 3D spatial cross-matching 3D k-Nearest Neighbor (kNN) query The first query type is spatial cross-matching. It is used for 3D segmentation algorithm validation or evaluation. Given two segmentation results of 3D nuclei, we first extract their minimum bounding box (MBBs) and the whole space info. We then apply data partitioning and obtain the 3D objects from two datasets but belonging the same cuboid. To speed up spatial query, we build local R*-tree index on the second dataset, and use a filter –and – refine approach to perform spatial cross-matching. We first use the MBBs of objects in dataset1 for intersection checking. If their MBBs intersect, we then check if their geometries are intersected or not. This is the geometry refinement step. If required, spatial measurement such as the intersection volume is calculated. 3D NN query has broad applications in various domains. For 3D analytical pathology imaging, pathologists are interested in spatial queries such as “for each 3D cell, return its nearest 3D blood vessel and the distance”. In this case, we can perform 3D NN query. We propose both voronoi diagram based and R*-tree based methods for NN query, and each of them achieves high performance. 3D Spatial Analytics

3D Spatial Queries and Analytics 3D Input Data Data Storage on HDFS 3D Data Partitioning After we extract the structure of 3D micro-anatomic objects, we perform 3D spatial analytics to analyze the structural relationship between them. Here we mainly focus on 3D blood vessels and 3D nuclei. After we reconstruct blood vessel structures, we simulate its surrounding nuclei for spatial analysis. Here is the architecture of our system. Our system is built on Hadoop. Given the 3D data as input, we store them on HDFS for efficient access. As the input 3D space is huge, we perform data partitioning to partition the space into small cuboids. Then each cuboid is taken as a processing unit for spatial queries on MapReduce. Our spatial query engine supports 3D spatial join and nearest neighbor query. It has two indexing strategies: global indexing for cuboids and local indexing for objects within cuboids. Our system is scalable and cost-effective, and can support 3D spatial queries efficiently. 3D Global Spatial Indexing 3D ODSQUE 3D Spatial Index Builder 3D Spatial Query Processing 3D Boundary Object Handling 3D Spatial Query Engine 3D Cuboid Spatial Indexing Hadoop 3D Spatial Analytics

3D Summary 3D pathology image analysis 3D data management and spatial analytics The first query type is spatial cross-matching. It is used for 3D segmentation algorithm validation or evaluation. Given two segmentation results of 3D nuclei, we first extract their minimum bounding box (MBBs) and the whole space info. We then apply data partitioning and obtain the 3D objects from two datasets but belonging the same cuboid. To speed up spatial query, we build local R*-tree index on the second dataset, and use a filter –and – refine approach to perform spatial cross-matching. We first use the MBBs of objects in dataset1 for intersection checking. If their MBBs intersect, we then check if their geometries are intersected or not. This is the geometry refinement step. If required, spatial measurement such as the intersection volume is calculated. 3D NN query has broad applications in various domains. For 3D analytical pathology imaging, pathologists are interested in spatial queries such as “for each 3D cell, return its nearest 3D blood vessel and the distance”. In this case, we can perform 3D NN query. We propose both voronoi diagram based and R*-tree based methods for NN query, and each of them achieves high performance. 3D 3D Spatial Analytics

Questions?