1. A Novel Image Registration Pipeline for 3- D Reconstruction from Microscopy Images Kun Huang, PhD; Ashish Sharma, PhD; Lee Cooper, MS; Kun Huang, PhD; Ashish Sharma, PhD; Lee Cooper, MS; Tony Pan, MS; Metin Gurcan, PhD; Joel Saltz, MD, PhD Department of Biomedical Informatics Ohio State University

2. Creating Geometry from Images Placenta H+E SlidesAlignment Segmentation Visualization/Surface Extraction Aperio Scanner

3. Registration Registration between different modalities (e.g, MRI and PET) Mapping of different samples to the same reference (e.g., brain mapping) 3-D reconstruction

4. An optimization problem Initialization Point feature matching Automatic vs. manual

5. Issues with automatic registration Initialization Landmark- or image- based? Linear or nonlinear? Error metric / Meaningful morphology / Domain specific knowledge Computation Structural constraints

6. Fast initialization using landmarks

7. S1S1 Fast initialization S2S2 S2’S2’ S1’S1’ S3’S3’ Matching pairs: (S 1, S 1 ’) (S 1, S 3 ’) (S 2, S 2 ’) S1S1 S2S2 S2’S2’ S1’S1’ S3’S3’ d 12 d 12 ’ d 13 ’

8. Fast initialization Maximum Clique Maximum Cyclic Structure ( S 1, S 1 ’, S 2, S 2 ’, θ, T )

9. Fast initialization Difference between two images. Difference after the automatic initialization using region features. Difference after the MMI algorithm.

10. Fast initialization

11. Registration of Large Images Using Landmarks

12. Registration of Large Images Using High- Level Features No need to globally transform the image Multi-level registration – rigid to nonrigid Parallelizable – local operations

13. Registration of Large Images Using Landmarks

14. Registration of Large Images Using High- Level Features Point feature does NOT contain global information For global transformation (e.g., rotation and translation), we need “global” features such as high-level features. For nonlinear transformation, which is local, we need “local” features such as point features. Global first, local second.

15. Stacks of microscopy images Principal component analysis (PCA) – based rigid registration

16. Stacks of microscopy images

22. 3-D reconstruction vs. registration The current metric for registration is between two images and is just for the sake of perfect “registration”. We do “registration” for the sake of 3-D reconstruction. The structural constraint should be incorporated in the “cost function” instead of just used as a post processing or validation criterion. New multiple image registration algorithm is needed!

23. 3-D reconstruction via registration Feature extraction Feature matching/ tracking Trajectory generation Trajectory smoothing and adjustment New location for the features in every image Nonlinear transformation for every image Collective adjustment of trajectories

24. 3-D reconstruction via registration Tracked trajectory Smoothed trajectory Registration moves the landmarks to the new locations.

25. 3-D reconstruction via registration

27. 3-D reconstruction via Registration

28. Summary, future work and discussion Technical issues related to automatic registration. Two step approach to achieve “good” nonlinear registration. The paradigm for 3-D reconstruction is different with pure registration. New registration pipeline is proposed and implemented.

29. Summary, future work and discussion Parallelization – especially in nonlinear transformation stage. Multiresolution / hierarchical approach.

30. Acknowledgement BMI Imaging group Collaborators Thank you !