1. MR images analysis of glioma
based on deep learning
Speaker: Zeju Li

2. Overview Background Knowledge Our contributions Glioma and MR images
Prognosis information of gliomas
Technological difficulties
Our contributions
Brain tumor segmentation
Brain tumor segmentation based on CNN with fully connected CRF
Brain tumor segmentation using an adversarial network
High order MR image reconstruction
Reconstruction of thin-slice MR images using GAN
2 stage deep learning based system for high-order slice MRI interpolation
Non-invasive prediction of prognostic information of gliomas
Age groups related study based on Radiomics
Deep learning based Radiomics and its usage in IDH1 prediction

3. Glioma and MR image 3/21 What is glioma?
1.Background Knowledge
2.Brain Tumor Segmentation
3.MR Image Reconstruction
4.Prognostic Information Prediction
3/21
What is glioma?
Glioma is the most common malignant brain tumor
Gliomas are rarely curable. The prognosis for patients with high-grade gliomas is generally poor
Radiology
MRI is the most popular way to form pictures of he anatomy physiological processes 
MRI have high imaging quality and many modalities
MRI T1 T2
Flair
T1C
DWI
ADC

4. Prognosis information
1.Background Knowledge
2.Brain Tumor Segmentation
3.MR Image Reconstruction
4.Prognostic Information Prediction
4/21
Biomarker of glioma
Biomarker proved to be of clinical significance of glioma
IDH1 mutations (a kind of biomarker) is suggested as key events in the formation of brain tumors.
High grade glioma with a wild-type IDH1 gene have a median overall survival of only 1 year, whereas IDH1-mutated glioblastoma patients have a median overall survival of over 2 years
However, clinical dissection is required to obtain those information
Age
It has been noticed that age is also an important marker to predict patient outcome with high grade glioma

5. Technological difficulties
1.Background Knowledge
2.Brain Tumor Segmentation
3.MR Image Reconstruction
4.Prognostic Information Prediction
5/21
Difficulties of brain tumor segmentation
The size, shape and location of glioma is unpredictable
Difficulties of getting prognosis information
Molecular information is shown to be expressed in MR images, molecular information is shown to be expressed in imaging, however the information is difficult to be determined and described

6. Work flow 6/21 Step 1 Brain tumor segmentation
1.Background Knowledge
2.Brain Tumor Segmentation
3.MR Image Reconstruction
4.Prognostic Information Prediction
6/21
Step 1 Brain tumor segmentation
Step 2 High order MR image interpolation
Step 3 Non-invasive prediction of prognostic information of gliomas
Original MR images
Maker prediction
Feature selection
Feature extraction
Tumor segmentation

7. Work flow 7/21 Step 1 Brain tumor segmentation
1.Background Knowledge
2.Brain Tumor Segmentation
3.MR Image Reconstruction
4.Prognostic Information Prediction
7/21
Step 1 Brain tumor segmentation
Step 2 High order MR image interpolation
Step 3 Non-invasive prediction of prognostic information of gliomas
Preparation
Original MR images
Maker prediction
Feature selection
Feature extraction
Tumor segmentation

8. Work flow 8/21 Step 1 Brain tumor segmentation
1.Background Knowledge
2.Brain Tumor Segmentation
3.MR Image Reconstruction
4.Prognostic Information Prediction
8/21
Step 1 Brain tumor segmentation
Step 2 High order MR image interpolation
Step 3 Non-invasive prediction of prognostic information of gliomas
Original MR images
Maker prediction
Feature selection
Feature extraction
Tumor segmentation

9. Brain tumor segmentation based on CNN with fully connected CRF
1.Background Knowledge
2.Brain Tumor Segmentation
3.MR Image Reconstruction
4.Prognostic Information Prediction
9/21
Methods
We proposed a novel tumor segmentation methods based on CNN, which
Have deeper network architecture
Have two ways to join the three-dimensional information of the adjacent gliomas
combined with the model of the fully connected condition random field (CRF)

10. Brain tumor segmentation based on CNN with fully connected CRF
1.Background Knowledge
2.Brain Tumor Segmentation
3.MR Image Reconstruction
4.Prognostic Information Prediction
10/21
Results
The proposed method improves the existing best CNN method from the Dice similarity index from 0.78 to 0.85, an increase of 0.07
Zeju Li, et al., Low Grade Glioma Segmentation Based on CNN with Fully Connected CRF, Journal of Healthcare Engineering, 2017.

11. Brain tumor segmentation using an adversarial network
1.Background Knowledge
2.Brain Tumor Segmentation
3.MR Image Reconstruction
4.Prognostic Information Prediction
11/21
Methods
We proposed a tumor segmentation methods using an adversarial network

12. Brain tumor segmentation using an adversarial network
1.Background Knowledge
2.Brain Tumor Segmentation
3.MR Image Reconstruction
4.Prognostic Information Prediction
12/21
Results
The spatial consistency of segmentation results is enforced without increasing the complexity of CNN model
Zeju Li, et al., Brain Tumor Segmentation Using an Adversarial Network, MICCAI-Brainlesion Workshop, 2017.

13. Reconstruction of thin-slice
MR images using GAN
1.Background Knowledge
2.Brain Tumor Segmentation
3.MR Image Reconstruction
4.Prognostic Information Prediction
13/21
Methods
We proposed a novel generative adversarial network based method for the reconstruction of thin-slice tomographic medical images from images with thick slices

14. Reconstruction of thin-slice
MR images using GAN
1.Background Knowledge
2.Brain Tumor Segmentation
3.MR Image Reconstruction
4.Prognostic Information Prediction
14/21
Results
Experiments results demonstrated that 3DSRGAN can provide better reconstruction results than other popular methods
Zeju Li, et al., Reconstruction of Thin-Slice Medical Images Using Generative Adversarial Network, MICCAI-MLMI Workshop, 2017.

15. 2 stage deep learning based system for
high-order slice MRI interpolation
1.Background Knowledge
2.Brain Tumor Segmentation
3.MR Image Reconstruction
4.Prognostic Information Prediction
15/21
Methods
2 stage deep learning based system (contains a GAN model for structure reconstruction and RNN model for detail reconstruction) for high-order slice medical images interpolation

16. 2 stage deep learning based system for high order MRI interpolation
1.Background Knowledge
2.Brain Tumor Segmentation
3.MR Image Reconstruction
4.Prognostic Information Prediction
16/21
Results
The interpolation results could provide more realistic brain information.
2D Axial
2D Sagittal
2D Axial
2D Sagittal
2D Axial
2D Sagittal
Reconstruct
3D Axial
GMV
0.75L (0.89)
0.67L (0.80)
0.84L (0.99)
0.84L
WMV
0.41L (0.93)
0.48L (0.92)
0.44L
TBV
0.34L (0.65)
0.30L (0.73)
0.21L (0.95)
0.22L
3D Axial
Reconstruct
3D Axial
Reconstruct
Zeju Li, et al., A Two-Stage Deep Learning Based System for Thin-Slice Medical Image Interpolation, preparing.

17. Age groups related study
based on Radiomics
1.Background Knowledge
2.Brain Tumor Segmentation
3.MR Image Reconstruction
4.Prognostic Information Prediction
17/21
Methods
An approach was developed to investigate the internal relationship between MRI features and the age-related origins of glioblastomas (a kind of high-grade glioma) based on a quantitative Radiomics method
555 high-throughput features extraction
Supervised features selection according to patients’ ages based on student’s t-test
A heat map was drawn using unsupervised hierarchical clustering of those selected features
Compare the classification results with ground truth
Tumor region segmentation in MRI images

18. Age groups related study
based on Radiomics
1.Background Knowledge
2.Brain Tumor Segmentation
3.MR Image Reconstruction
4.Prognostic Information Prediction
18/21
Results
Unsupervised clustering results of those features in the heat map also show coherence with the age difference (T test, p=0.006)
Zeju Li, et al., Age Groups Related Glioblastoma Study Based on Radiomics Approach, Computer Assisted Surgery, 2017.

19. Deep learning based Radiomics and its usage in IDH1 prediction
1.Background Knowledge
2.Brain Tumor Segmentation
3.MR Image Reconstruction
4.Prognostic Information Prediction
19/21
Methods
We proposed deep learning based Radiomics (DLR), which
Uses CNN to effectively identify the characteristics of glioma, directly from the CNN network of tumor segmentation to obtain effective information
The three dimensions of the tumor area were reduced to the same dimension by Fisher Vector

20. Deep learning based Radiomics and its usage in IDH1 prediction
1.Background Knowledge
2.Brain Tumor Segmentation
3.MR Image Reconstruction
4.Prognostic Information Prediction
20/21
Results
Some of the information obtained in the feature layer is highly reflected in the molecular information of the tumor
DLR is able to improve 6% prediction accuracy in IDH1 prediction of grade II gliomas compared with traditional Radiomics framework. DLR based on multi-model MR images is used to achieve 95% IDH1 prediction results
Zeju Li, et al., Deep Learning based Radiomics (DLR) and its Usage in Noninvasive IDH1 Prediction for Low Grade Glioma, Scientific Reports, 2017.

21. Thanks!