TeraTomo project: a fully 3D GPU based reconstruction code for exploiting the imaging capability of the NanoPET™/CT system M. Magdics 1 ), L. Szirmay-Kalos 1), Á. Szlavecz 1), G. Hesz 1), B. Benyó 1), Á. Cserkaszky 2), J. Lantos 2), D. Légrády 2), Sz. Czifrus 2), A. Wirth 3), B. Kári 3), G. Patay 4), D. Völgyes 4), T. Bükki 4), P. Major 4), G. Németh 4), B. Domonkos 4) 1) Department of Informatics, Budapest University of Technology and Economics, Hungary 2) Institute of Nuclear Techniques, Budapest University of Technology and Economics, Hungary 3) Department of Radiology, Semmelweis University of Budapest, Hungary 4) Mediso Ltd., Budapest, Hungary Forward projection Backprojection + TV regularization yLyL ~ yLyL sVsV xVxV xVxV xVxV Filtering (n+1) * ^ (n)(n) xVxV (n)(n) Measured LORs Reconstructed voxels Positron range Direct: Geometry Scatter + Attenuation + Detector model Detector model Direct Indirect Combination+ Stochastic iteration voxels Expected LORs The TeraTomo project is dedicated to the development of a fully 3D iterative reconstruction code for multi-modality (PET/SPECT/CT) imaging. Recently, we have employed the EM/OSEM scheme for reconstruction of PET images; we have decided to focus on the on-the-fly calculation of the system matrix elements as precisely as possible taking the following physical effects into account: 3D geometry, detector response, positron-range attenuation, and scatter in the medium. The reconstruction algorithms have been tailored to the massively parallel GPU platform (using CUDA technology), enabling to execute the code in parallel on multiple graphics cards [1]. The reconstruction algorithm employs Monte Carlo (MC) techniques for sampling the Lines Of Responses (LOR) and voxels in forward- and back- projection steps. To attack the ill-posed EM scheme, our implementation contains regularization techniques like Gaussian and median filtering, as well as Total Variation regularization that significantly increases the quality of reconstructions at a negligible additional cost. With these advancements, over resolution voxel arrays can be reconstructed in a few minutes. 1. Scattering points 2. Ray marching from detector to scattering points 3. Combination of paths  Scatter compensation of adjMC method Detector response modeling 3D adjMC reconstruction without detector response compensation Effect of Total Variation regularization TeraTomo reconstruction of the rotated Derenzo phantom, rod sizes are 1.0, 1.1, 1.2, 1.3, 1.4, and 1.5 mm. GATE simulation in NanoPET™/CT PET detector geometry was reconstructed into 144x144x128 voxels volume. The central sagittal slice of the reconstructed volume at 50 Iterations is depicted below System overview The reconstruction algorithm is an iterative maximum likelihood estimation method (EM/OSEM), which alternatively executes photon transport simulation (forward projection) and source correction (backprojection). We implemented two types of simulation approaches, both running on multiple GPUs: 1.) Using Monte Carlo particle transport simulation (MC) [2] and 2.) Using adjoint Monte Carlo approximation (adjMC) [3]. Adjoint Monte Carlo approximation (adjMC) The adjMC method employs approximate adjoint transfer in forward projection and a geometric backprojection, assigning LORs to threads in forward projection and voxels in backprojection. In order to increase the accuracy of integral quadratures, we use quasi-Monte Carlo techniques combined with stochastic iteration and filtered sampling. Forward projection of the adjMC method The sampling process estimating the scattered contribution has three steps. First, scattering points are sampled with a probability density that is proportional to the scattering cross section of the material. Then, the total annihilation and out-scattering is computed between these sample points and the detector crystals. In the final step, we just combine these results together and compute the direct component. As the number of crystal pairs is much larger than the number of crystals and scattering points, the scattering computation has just a small overhead with respect to that of the direct component. Photons may get scattered in detector crystals before they get finally absorbed. Unlike the measured object, which is different in each measurement, the detectors are fixed, so the probabilities of photon paths between detector crystals can be pre-computed, and these pre-computed data can be included in the estimator. During the pre- computation we consider a single input crystal and incident photons coming from a direction of given inclination and azymuthal angles are simulated and we compute the probabilities that this photon is absorbed in another crystal. These probabilities can be visualized as a two dimensional image, where arrival probabilities are depicted by gray levels. These images are two large to be sampled efficiently. So, during the pre-computation, we pre-generate quasi- Monte Carlo sample sets that contain just a few samples, but their cumulative distribution is as close as the simulated distribution as possible. GATE Phantom 4D detector response image Source estimation Source correction Compute expected detector response Expected detector response Compare Mathematical phantoms TeraTomo Tomography reconstruction In order to validate our system, we used both simulated and measured data. Mathematical phantoms were simulated by GATE [4]. Both simulated and measured list-mode data were binned into LOR files before reconstruction. Then, taking the simulated detector hits, we reconstructed the source distribution with our program, and compared the result with the original phantom. In case of measured data, the normalization information was passed to the TeraTomo reconstruction engine. PET equipment: NanoPET™/CT The NanoPET™/CT is an ultra-high resolution, high sensitivity pre-clinical PET-CT system using the most advanced, commercially available components, i.e. an 18 cm diameter PET-detector polygon with 12 detector modules, each consisting of 81×39 LYSO crystals (1.12×1.12×13 mm³) tightly packed and coupled to two 256-channel PS-PMTs. The imaging capability of this system can only be exploited by using a fully 3D reconstruction algorithm modeling the detector response, positron range, gamma attenuation, and scatter effects. Speed of reconstruction Reconstructing the Derenzo phantom using 144×144×128 voxels, assuming the NanoPET™/CT PET detector geometry [5] that contains 180 million LORs, the speed of reconstruction is similar to that of FORE rebinning and 2D- EM running on multiple CPUs. The TeraTomo reconstruction times on a single Nvidia Fermi GPU card are shown in the following table. When two GPU cards are enabled, running time is halved. Results of measured data reconstruction Monte Carlo particle transport (MC) In both forward projection and backprojection, MC assigns particles to GPU computing threads. The particle transport is simulated by each thread closely mimicking nature by sampling each possible interaction with the corresponding probability distributions as long as the particles are in the object (phantom). Particles initiated at a voxel would hit a detector with a given probability, otherwise miss; by the latter the computing effort spent on a particle is lost. To minimize this price we pay for the simulation accuracy, non-analog MC techniques are used such as source direction biasing, implicit capture, biased source sampling and precomputed detector response. Free path sampling is achieved by Woodcock tracking, thereby the simulation efficiency is only loosely dependent on material constituents. References [1] DOMONKOS, B., AND JAKAB, B. A Programming Model for GPU-based Parallel Computing with Scalability and Abstraction. In Spring Conference on Computer Graphics (2009), pp. 115–122. [2] WIRTH, A., CSERKASZKY, A., KÁRI, B., LÉGRÁDY, D., FEHÉR, S., CZIFRUS, S., AND DOMONKOS, B. Implementation of 3D Monte Carlo PET Reconstruction Algorithm on GPU. In IEEE Medical Imaging Conference (2009) [3] SZIRMAY-KALOS, L., TÓTH, B., MAGDICS, M., LÉGRÁDY, D., AND PENZOV, A. Gamma Photon Transport on the GPU for PET. Lecture Notes in Computer Science 5910: pp (2010) [4] GATE, see and JAN S. et. al., GATE: a simulation toolkit for PET and SPECT. Phys. Med. Biol. 49 (2004) http://opengatecollaboration.healthgrid.org/ [5] More information about NanoPET™/CT device, visit: Results of incorporating detector response modeling 3D reconstruction of the GATE simulated Derenzo phantom. The effects of 3D geometry and of detector response modeling in the reconstruction have significantly improved the image quality. Verification At the recent stage of TeraTomo development, we have already successfully implemented and carefully verified the 3D geometry based reconstruction engine including detector response modeling. Future work We are going to deal with the verification of scatter and attenuation modeling in the near future. Then we are planning to implement random coincidence modeling as well as dead time correction in order to archive a 3D quantitative reconstruction tool. Each particle history contains but a few events from positron annihilation to escape or to energy cut-off, hence computing threads hardly diverge. Overall simulation speed offers sampling up to 5  10 8 positrons/sec, roughly 2.5 magnitudes over a general purpose MC particle transport code. Advantages of faithful physics simulation are expected at media with high scattering components in exchange for slower convergence and more pronounced tendency for noise build-up. Forward projection and backprojection employ the same MC engine for calculating system matrix elements. Optimum samplings of voxels in both reconstruction phases were found to be proportional to the activity. The forward projector of the adjoint method examines LORs one-by-one and computes the expected number of hits due to annihilations in voxels intersected by the line samples of this LOR (direct contribution) and all voxels (scattered contribution). Scattering in the object and scattering in the detectors are handled independently via the interface of sample points generated on the surface of the detector crystal. The high-dimensional integral of a LOR is estimated by quasi-Monte Carlo quadrature including Poisson-disk sampling. In order to reduce the variance of this estimator, we employ both spatial and temporal filtering. Temporal filtering is called stochastic iteration, which averages the estimated LOR values with the results of previous steps. Spatial filtering may include either Gaussian or median filtering before the execution of the forward projector. Backprojection of the adjMC method The backprojector of the adjMC method has been developed with the objective of efficient GPU execution. Thus, it is also of gathering type where a thread computes the update of a voxel from all LORs intersecting it. To select these LORs, a point is sampled in the voxel, then the detector module is centrally projected onto its coincidence pair via the voxel sample. The backprojector is also responsible for TV regularization, thus it also reads the neighboring voxel values of the previous step, obtains the gradient and includes it in the iteration formula. voxel Detector 1 Detector 2 3D EM reconstruction of the F-18 mouse bone PET study, taken by NanoPET™/CT. There was no scatter, attenuation, and detector model compensation. ForwardBack 3D-MC28 sec/iteration 3D-adjMC2 sec/iteration16 sec/iteration Noise tolerance Regularization methods guarantee correct reconstruction even for low-dose measurements when the average number of hits per LOR is very low. The left image depicts a Derenzo that was reconstructed from 4 hits per LOR in average, the right image shows the reconstruction from only 0.2 hits per LOR in average. 2D reconstruction: SSRB + OSEM 3D adjMC reconstruction with detector response compensation