1. Registration of Pathological Images
Xiao Yang1, Xu Han1, Eunbyung Park1, Stephen Aylward2, Roland Kwitt3, Marc Niethammer1
Department of Computer Science
1University of North Carolina at Chapel Hill
3University of Salzburg
3Kitware Inc.

2. Deformable Image Registration
Registration of Pathological Images
Deformable Image Registration
Goal: Spatial alignment of two images (beyond affine). ?
Source Image
Target Image
minimize Irregularity of transformation + image mismatch

3. Issue: Intensity/Structural changes
Image Registration
Registration of Pathological Images
Issue: Intensity/Structural changes
Traumatic brain injury
Brain tumor
What do we do if we need to deal with pathologies?
“Similar looking regions” do not correspond.
Structures only exist in one of the two images.

4. Simplest Solution: Cost-function masking
Registration of Pathological Images
Simplest Solution: Cost-function masking
One solution: Cost function masking [Brett2001]
= ignoring matching cost in region of source image
Image: gerdleonhard.typepad.com X
… let’s just not look at it!

5. Other Solution Approaches
Other Approaches
Registration of Pathological Images
Other Solution Approaches
Segmentation-based methods
Cost function masking
Joint Segmentation and Registration
Geometric Metamorphosis
All these methods require segmentations!

6. magic What if? What if there were a method to transform an image with
Direct Mapping
Registration of Pathological Images
What if?
What if there were a method to transform an image with
pathology into a healthy-looking image?
magic
We can learn such a method from population data ...

7. Other Solution Approaches
Quasi-Normal by Regression
Registration of Pathological Images
Other Solution Approaches
Low-rank/sparse approach (LRS)
Separate the image into low-rank part and sparse part
Requires interleaved registration and LRS decomposition
Our approach is similar in spirit to LRS, but
directly predicts a quasi-normal image w/o registrations
preserves fine image detail

8. Prediction via a variational autoencoder
Quasi-Normal by Regression
Registration of Pathological Images
Prediction via a variational autoencoder
Goal: Directly learn a regression model from pathological to quasi-normal
We use a variational autoencoder, but other regression
models are of course possible too ...
... with this model we can also estimate output uncertainties.

9. Denoising variational autoencoder (DVAE)
Quasi-Normal by Regression
Registration of Pathological Images
Denoising variational autoencoder (DVAE)
Pathology is considered noise:
removing the noise result in the quasi-normal image
Variational formulation
hidden layer values are random variables (Gaussian)
sampling the network allows for uncertainty estimates
Autoencoder
(AE))
Denoising AE
Variational AE
DVAE

10. Training the network We train the network using pathological data
Network Training
Registration of Pathological Images
Training the network
We train the network using pathological data
+ includes deformations of pathologies (mass effect)
- unclear what the quasi-normal image should be
We use two approaches
Loss function masking: ignores tumor area during training
Simulated lesion: we add a “QuasiLesion” layer to simulate lesions for which we know the normal appearance

11. Loss Function Masking For input image
Network Training
Registration of Pathological Images
Loss Function Masking
For input image
replace pathology by constant intensity plus noise
For output image
disregard the pathological area in backpropagation
Lesion segmentations are only required at training time!

12. Quasi-Lesion Layer (i.e., simulated lesions)
Network Training
Registration of Pathological Images
Quasi-Lesion Layer (i.e., simulated lesions)
Add simulated (quasi) lesions to normal areas
and then train the network to remove those

13. FakeLesion Layer (i.e., simulated lesions)
Network Training
Registration of Pathological Images
FakeLesion Layer (i.e., simulated lesions)
Add simulated (quasi) lesions to normal areas
and then train the network to remove those
Choice of simulated lesion texture
real tumor; mean or random intensity; random noise
Choice did experimentally not make a significant difference

14. Uncertainty-Guided Registration
Registration of Pathological Images
Uncertainty-Guided Registration
Can assess uncertainty if the reconstruction via sampling
Mean = reconstruction result
Variance = reconstruction uncertainty

15. Experimental Settings
Experiments
Registration of Pathological Images
Experimental Settings
2D synthetic image + BRATs training dataset
Network training: torch + rmsprop
Uncertainty-weighted registration: modified NiftyReg
Synthetic experiment:
register OASIS images to BRATs (cost fcn. masking)
add BRATs tumor to deformed OASIS images
500 training images, 50 for testing, 196x232
BRATs experiment:
cross-validation: 4 sets of 244 training images test images
Adaptive histogram equalization

16. Example Results (based on simulated data for illustration)
Experiments
Registration of Pathological Images
Example Results (based on simulated data for illustration)
Original
Original + Tumor
Improved/sharper
reconstruction
Uncertainty map
to weight
registration
LRS reconstruction
direct reconstruction
uncertainty

17. Image Registration Result: Synthetic
Experiments
Registration of Pathological Images
Image Registration Result: Synthetic
Ground truth deformation
LRS
Our result without uncertainty
Our result with uncertainty

18. Evaluation on BRATS 2015 Dataset
Experiments
Registration of Pathological Images
Evaluation on BRATS 2015 Dataset
274 images in training dataset
Cross-validation: 4 sets of 244 training + 30 testing images
Landmarks for evaluation (avg. 10 landmarks per image)
Direct reconstruction ("our model") works generally best.
Uncertainty helps.
Cost function masking does not perform well.

19. BRATs testcase Original image LRS result Our result Uncertainty map
Experiments
Registration of Pathological Images
BRATs testcase
Original image
LRS result
Our result
Uncertainty map

20. BRATs testcase Original image Cost function masking LRS method
Experiments
Registration of Pathological Images
BRATs testcase
Cost function masking
LRS method
Original image
Our method,
no uncertainty
Our method
with uncertainty

21. Conclusion / Discussion
Registration of Pathological Images
Conclusion / Discussion
Potential for 3D data:
2.5D network; 14 slices at once; slow but feasible
3D patch-based network (including location)
More realistic simulated lesion appearance:
likely possible with more training data
Faithful reconstruction of normal areas is important
excessive smoothing by LRS loses fine details important for registration
Faithful reconstruction of pathological areas may not be that important if combined with an uncertainty measure
may be possible to outperform cost function masking

22. Thanks for your attention!
The End …
Registration of Pathological Images
Thanks
Thanks for your attention!
Questions?