Synthesis of X-ray Projections via Deep Learning Ye Duan University of Missouri at Columbia

Outline Multiview CNN/RNN for shape understanding Multiview Image Synthesis Conclusion & Future Work

Outline Multiview CNN/RNN for shape understanding Multiview Image Synthesis Conclusion & Future Work

MV-CNN Su et al., “Multi-view convolutional neural networks for 3d shape recognition”, ICCV 2015

MV-RNN (Truc Le & Ye Duan, Computer & Graphics, 2017) Convolutional Neural Network Fully-connected Long Short-Term Memory Conditional Random Field

MV-RNN: Input K views uniformly distributed Views are ordered Phong’s shading (light source is always behind the camera) Image resolution: 128 x 128 Store 2D-3D correspondence

CNN Module Each view shaded image is passed through an identical image-based CNN Many CNNs can be used: FCN DeconvNet HED (we use this one because it is designed for 2D edge detection) Output: edge confidence maps

Recurrent Neural Network: LSTM Module Edge probability maps are orderly fed to a two-layer LSTM as sequence Peep-hole LSTM with 1024 hidden units Fully-connected layer after the second LSTM Output: edge images for all views

Side-by-side Comparison

Side-by-side Comparison

Outline Multiview CNN/RNN for shape understanding Multiview Image Synthesis Conclusion & Future Work

Synthesis of X-ray Projections via Deep Learning

Synthesis of X-ray Projections via Deep Learning

Synthesis of X-ray Projections via Deep Learning Collaborated with Dr. Ge Wang and Qingsong Yang. X-ray imaging has been widely used in medical diagnostics. The potential risk of x-ray has drawing more and more concerns. To reduce the x-ray dose, one possible method is to reduce the number of x-ray projections. We proposed a method that infers the projection image from projections at adjacent angles using deep learning.

Synthesis of X-ray Projections via Deep Learning The proposed method can be used for limited-view CT reconstruction, where only a few projection data are acquired. The network can generate the missing data from known projections and guide the few-view reconstruction. The proposed method can be used in x-ray tomosynthesis, in which only two or three projection data are acquired

Network Architecture Discriminator 32@256x256 32@128x128 64@64x64

Experiments on Laser Scan 3D Data Result Ground Trust Input

Experiments on Laser Scan 3D Data Result Ground Trust Input

Experiments on Synthetic X-Ray Result Ground Trust Input

Experiments on Synthetic X-Ray Result Ground Trust Input

Experiments on Synthetic X-Ray Result Ground Trust Input

Training details Number of models: 20. Each model contains 72 images. Image size: 256x256 Batch size: 10 Epoches: 20 Learning rate: 1.0e-4 Adam optimization

Image Quality Evaluation Datasets PSNR SSIM Laser Scan 35.14 0.973 Synthetic X-Ray 34.34 0.969

Future Work Multi-view Image Super Resolution

Future Work 3D Volume synthesis

Acknowledgements Truc Le Giang Bui

Thank you! duanye@missouri.edu