Registration of Pathological Images Xiao Yang1, Xu Han1, Eunbyung Park1, Stephen Aylward2, Roland Kwitt3, Marc Niethammer1 Department of Computer Science 1University of North Carolina at Chapel Hill 3University of Salzburg 3Kitware Inc.

Deformable Image Registration Registration of Pathological Images Deformable Image Registration Goal: Spatial alignment of two images (beyond affine). ? Source Image Target Image minimize Irregularity of transformation + image mismatch

Issue: Intensity/Structural changes Image Registration Registration of Pathological Images Issue: Intensity/Structural changes Traumatic brain injury Brain tumor What do we do if we need to deal with pathologies? “Similar looking regions” do not correspond. Structures only exist in one of the two images.

Simplest Solution: Cost-function masking Registration of Pathological Images Simplest Solution: Cost-function masking One solution: Cost function masking [Brett2001] = ignoring matching cost in region of source image Image: gerdleonhard.typepad.com X … let’s just not look at it!

Other Solution Approaches Other Approaches Registration of Pathological Images Other Solution Approaches Segmentation-based methods Cost function masking Joint Segmentation and Registration Geometric Metamorphosis All these methods require segmentations!

magic What if? What if there were a method to transform an image with Direct Mapping Registration of Pathological Images What if? What if there were a method to transform an image with pathology into a healthy-looking image? magic We can learn such a method from population data ...

Other Solution Approaches Quasi-Normal by Regression Registration of Pathological Images Other Solution Approaches Low-rank/sparse approach (LRS) Separate the image into low-rank part and sparse part Requires interleaved registration and LRS decomposition Our approach is similar in spirit to LRS, but directly predicts a quasi-normal image w/o registrations preserves fine image detail

Prediction via a variational autoencoder Quasi-Normal by Regression Registration of Pathological Images Prediction via a variational autoencoder Goal: Directly learn a regression model from pathological to quasi-normal We use a variational autoencoder, but other regression models are of course possible too ... ... with this model we can also estimate output uncertainties.

Denoising variational autoencoder (DVAE) Quasi-Normal by Regression Registration of Pathological Images Denoising variational autoencoder (DVAE) Pathology is considered noise: removing the noise result in the quasi-normal image Variational formulation hidden layer values are random variables (Gaussian) sampling the network allows for uncertainty estimates Autoencoder (AE)) Denoising AE Variational AE DVAE

Training the network We train the network using pathological data Network Training Registration of Pathological Images Training the network We train the network using pathological data + includes deformations of pathologies (mass effect) - unclear what the quasi-normal image should be We use two approaches Loss function masking: ignores tumor area during training Simulated lesion: we add a “QuasiLesion” layer to simulate lesions for which we know the normal appearance

Loss Function Masking For input image Network Training Registration of Pathological Images Loss Function Masking For input image replace pathology by constant intensity plus noise For output image disregard the pathological area in backpropagation Lesion segmentations are only required at training time!

Quasi-Lesion Layer (i.e., simulated lesions) Network Training Registration of Pathological Images Quasi-Lesion Layer (i.e., simulated lesions) Add simulated (quasi) lesions to normal areas and then train the network to remove those

FakeLesion Layer (i.e., simulated lesions) Network Training Registration of Pathological Images FakeLesion Layer (i.e., simulated lesions) Add simulated (quasi) lesions to normal areas and then train the network to remove those Choice of simulated lesion texture real tumor; mean or random intensity; random noise Choice did experimentally not make a significant difference

Uncertainty-Guided Registration Registration of Pathological Images Uncertainty-Guided Registration Can assess uncertainty if the reconstruction via sampling Mean = reconstruction result Variance = reconstruction uncertainty

Experimental Settings Experiments Registration of Pathological Images Experimental Settings 2D synthetic image + BRATs training dataset Network training: torch + rmsprop Uncertainty-weighted registration: modified NiftyReg Synthetic experiment: register OASIS images to BRATs (cost fcn. masking) add BRATs tumor to deformed OASIS images 500 training images, 50 for testing, 196x232 BRATs experiment: cross-validation: 4 sets of 244 training images + 30 test images Adaptive histogram equalization

Example Results (based on simulated data for illustration) Experiments Registration of Pathological Images Example Results (based on simulated data for illustration) Original Original + Tumor Improved/sharper reconstruction Uncertainty map to weight registration LRS reconstruction direct reconstruction uncertainty

Image Registration Result: Synthetic Experiments Registration of Pathological Images Image Registration Result: Synthetic Ground truth deformation LRS Our result without uncertainty Our result with uncertainty

Evaluation on BRATS 2015 Dataset Experiments Registration of Pathological Images Evaluation on BRATS 2015 Dataset 274 images in training dataset Cross-validation: 4 sets of 244 training + 30 testing images Landmarks for evaluation (avg. 10 landmarks per image) Direct reconstruction ("our model") works generally best. Uncertainty helps. Cost function masking does not perform well.

BRATs testcase Original image LRS result Our result Uncertainty map Experiments Registration of Pathological Images BRATs testcase Original image LRS result Our result Uncertainty map

BRATs testcase Original image Cost function masking LRS method Experiments Registration of Pathological Images BRATs testcase Cost function masking LRS method Original image Our method, no uncertainty Our method with uncertainty

Conclusion / Discussion Registration of Pathological Images Conclusion / Discussion Potential for 3D data: 2.5D network; 14 slices at once; slow but feasible 3D patch-based network (including location) More realistic simulated lesion appearance: likely possible with more training data Faithful reconstruction of normal areas is important excessive smoothing by LRS loses fine details important for registration Faithful reconstruction of pathological areas may not be that important if combined with an uncertainty measure may be possible to outperform cost function masking

Thanks for your attention! The End … Registration of Pathological Images Thanks Thanks for your attention! Questions?