MR images analysis of glioma based on deep learning Speaker: Zeju Li 2017.11.2

Overview Background Knowledge Our contributions Glioma and MR images Prognosis information of gliomas Technological difficulties Our contributions Brain tumor segmentation Brain tumor segmentation based on CNN with fully connected CRF Brain tumor segmentation using an adversarial network High order MR image reconstruction Reconstruction of thin-slice MR images using GAN 2 stage deep learning based system for high-order slice MRI interpolation Non-invasive prediction of prognostic information of gliomas Age groups related study based on Radiomics Deep learning based Radiomics and its usage in IDH1 prediction

Glioma and MR image 3/21 What is glioma? 1.Background Knowledge 2.Brain Tumor Segmentation 3.MR Image Reconstruction 4.Prognostic Information Prediction 3/21 What is glioma? Glioma is the most common malignant brain tumor Gliomas are rarely curable. The prognosis for patients with high-grade gliomas is generally poor Radiology MRI is the most popular way to form pictures of he anatomy physiological processes  MRI have high imaging quality and many modalities MRI T1 T2 Flair T1C DWI ADC

Prognosis information 1.Background Knowledge 2.Brain Tumor Segmentation 3.MR Image Reconstruction 4.Prognostic Information Prediction 4/21 Biomarker of glioma Biomarker proved to be of clinical significance of glioma IDH1 mutations (a kind of biomarker) is suggested as key events in the formation of brain tumors. High grade glioma with a wild-type IDH1 gene have a median overall survival of only 1 year, whereas IDH1-mutated glioblastoma patients have a median overall survival of over 2 years However, clinical dissection is required to obtain those information Age It has been noticed that age is also an important marker to predict patient outcome with high grade glioma

Technological difficulties 1.Background Knowledge 2.Brain Tumor Segmentation 3.MR Image Reconstruction 4.Prognostic Information Prediction 5/21 Difficulties of brain tumor segmentation The size, shape and location of glioma is unpredictable Difficulties of getting prognosis information Molecular information is shown to be expressed in MR images, molecular information is shown to be expressed in imaging, however the information is difficult to be determined and described

Work flow 6/21 Step 1 Brain tumor segmentation 1.Background Knowledge 2.Brain Tumor Segmentation 3.MR Image Reconstruction 4.Prognostic Information Prediction 6/21 Step 1 Brain tumor segmentation Step 2 High order MR image interpolation Step 3 Non-invasive prediction of prognostic information of gliomas Original MR images Maker prediction Feature selection Feature extraction Tumor segmentation

Work flow 7/21 Step 1 Brain tumor segmentation 1.Background Knowledge 2.Brain Tumor Segmentation 3.MR Image Reconstruction 4.Prognostic Information Prediction 7/21 Step 1 Brain tumor segmentation Step 2 High order MR image interpolation Step 3 Non-invasive prediction of prognostic information of gliomas Preparation Original MR images Maker prediction Feature selection Feature extraction Tumor segmentation

Work flow 8/21 Step 1 Brain tumor segmentation 1.Background Knowledge 2.Brain Tumor Segmentation 3.MR Image Reconstruction 4.Prognostic Information Prediction 8/21 Step 1 Brain tumor segmentation Step 2 High order MR image interpolation Step 3 Non-invasive prediction of prognostic information of gliomas Original MR images Maker prediction Feature selection Feature extraction Tumor segmentation

Brain tumor segmentation based on CNN with fully connected CRF 1.Background Knowledge 2.Brain Tumor Segmentation 3.MR Image Reconstruction 4.Prognostic Information Prediction 9/21 Methods We proposed a novel tumor segmentation methods based on CNN, which Have deeper network architecture Have two ways to join the three-dimensional information of the adjacent gliomas combined with the model of the fully connected condition random field (CRF)

Brain tumor segmentation based on CNN with fully connected CRF 1.Background Knowledge 2.Brain Tumor Segmentation 3.MR Image Reconstruction 4.Prognostic Information Prediction 10/21 Results The proposed method improves the existing best CNN method from the Dice similarity index from 0.78 to 0.85, an increase of 0.07 Zeju Li, et al., Low Grade Glioma Segmentation Based on CNN with Fully Connected CRF, Journal of Healthcare Engineering, 2017.

Brain tumor segmentation using an adversarial network 1.Background Knowledge 2.Brain Tumor Segmentation 3.MR Image Reconstruction 4.Prognostic Information Prediction 11/21 Methods We proposed a tumor segmentation methods using an adversarial network

Brain tumor segmentation using an adversarial network 1.Background Knowledge 2.Brain Tumor Segmentation 3.MR Image Reconstruction 4.Prognostic Information Prediction 12/21 Results The spatial consistency of segmentation results is enforced without increasing the complexity of CNN model Zeju Li, et al., Brain Tumor Segmentation Using an Adversarial Network, MICCAI-Brainlesion Workshop, 2017.

Reconstruction of thin-slice MR images using GAN 1.Background Knowledge 2.Brain Tumor Segmentation 3.MR Image Reconstruction 4.Prognostic Information Prediction 13/21 Methods We proposed a novel generative adversarial network based method for the reconstruction of thin-slice tomographic medical images from images with thick slices

Reconstruction of thin-slice MR images using GAN 1.Background Knowledge 2.Brain Tumor Segmentation 3.MR Image Reconstruction 4.Prognostic Information Prediction 14/21 Results Experiments results demonstrated that 3DSRGAN can provide better reconstruction results than other popular methods Zeju Li, et al., Reconstruction of Thin-Slice Medical Images Using Generative Adversarial Network, MICCAI-MLMI Workshop, 2017.

2 stage deep learning based system for high-order slice MRI interpolation 1.Background Knowledge 2.Brain Tumor Segmentation 3.MR Image Reconstruction 4.Prognostic Information Prediction 15/21 Methods 2 stage deep learning based system (contains a GAN model for structure reconstruction and RNN model for detail reconstruction) for high-order slice medical images interpolation

2 stage deep learning based system for high order MRI interpolation 1.Background Knowledge 2.Brain Tumor Segmentation 3.MR Image Reconstruction 4.Prognostic Information Prediction 16/21 Results The interpolation results could provide more realistic brain information. 2D Axial 2D Sagittal 2D Axial 2D Sagittal   2D Axial 2D Sagittal Reconstruct 3D Axial GMV 0.75L (0.89) 0.67L (0.80) 0.84L (0.99) 0.84L WMV 0.41L (0.93) 0.48L (0.92) 0.44L TBV 0.34L (0.65) 0.30L (0.73) 0.21L (0.95) 0.22L 3D Axial Reconstruct 3D Axial Reconstruct Zeju Li, et al., A Two-Stage Deep Learning Based System for Thin-Slice Medical Image Interpolation, preparing.

Age groups related study based on Radiomics 1.Background Knowledge 2.Brain Tumor Segmentation 3.MR Image Reconstruction 4.Prognostic Information Prediction 17/21 Methods An approach was developed to investigate the internal relationship between MRI features and the age-related origins of glioblastomas (a kind of high-grade glioma) based on a quantitative Radiomics method 555 high-throughput features extraction Supervised features selection according to patients’ ages based on student’s t-test A heat map was drawn using unsupervised hierarchical clustering of those selected features Compare the classification results with ground truth Tumor region segmentation in MRI images

Age groups related study based on Radiomics 1.Background Knowledge 2.Brain Tumor Segmentation 3.MR Image Reconstruction 4.Prognostic Information Prediction 18/21 Results Unsupervised clustering results of those features in the heat map also show coherence with the age difference (T test, p=0.006) Zeju Li, et al., Age Groups Related Glioblastoma Study Based on Radiomics Approach, Computer Assisted Surgery, 2017.

Deep learning based Radiomics and its usage in IDH1 prediction 1.Background Knowledge 2.Brain Tumor Segmentation 3.MR Image Reconstruction 4.Prognostic Information Prediction 19/21 Methods We proposed deep learning based Radiomics (DLR), which Uses CNN to effectively identify the characteristics of glioma, directly from the CNN network of tumor segmentation to obtain effective information The three dimensions of the tumor area were reduced to the same dimension by Fisher Vector

Deep learning based Radiomics and its usage in IDH1 prediction 1.Background Knowledge 2.Brain Tumor Segmentation 3.MR Image Reconstruction 4.Prognostic Information Prediction 20/21 Results Some of the information obtained in the feature layer is highly reflected in the molecular information of the tumor DLR is able to improve 6% prediction accuracy in IDH1 prediction of grade II gliomas compared with traditional Radiomics framework. DLR based on multi-model MR images is used to achieve 95% IDH1 prediction results Zeju Li, et al., Deep Learning based Radiomics (DLR) and its Usage in Noninvasive IDH1 Prediction for Low Grade Glioma, Scientific Reports, 2017.

Thanks!