1. TBA #23 GE Corporate R&D Niskayuna, NY lorensen@crd.ge.com

2. Unification of Vision, Geometry and Graphics Through Toolkits Bill Lorensen GE Corporate R&D Niskayuna, NY lorensen@crd.ge.com

3. What is a Toolkit? Mathematics + Algorithms + Software Edelsbrunner, 2001

4. Dual Interests

5. Marching Cubes 1984

6. Baseball Visualization 1989

7. Stream Polygons - 1991

8. Triangle Decimation - 1992

9. IEEE CG&A 1992

10. Swept Surfaces 1993 Removal Path Swept Surface

11. Virtual Endoscopy 1994

12. Creating Models from Segmented Medical Data

13. Surface and Volume Rendering

14. Hypothesis Many real world problems cannot be solved by a single discipline

15. Core Technologies for 3D Medical Image Analysis Registration –Intra-modality (MRI to MRI, CT to CT) –Inter-modality (MRI to PET) –Model to Modality (Atlas to MRI) –Metadata to Modality (Clinical data, biochip to MRI/CT) Filters –Edge preserving –Noise reduction –Non uniform intensity correction Segmentation –Edge detection –Region growing –Multi-channel Pattern Recognition – Tissue classification Visualization – Surface / volume rendering – Fusion Quantification – Area, volume, shape Change detection – Longitudinal tracking – Signal variation Information Analysis/Visualization

16. Discipline-specific Toolkits Use “best of breed” algorithms implemented by domain experts –Point matching –Voronoi diagram computation –Registration –Pose estimation –Isosurface extraction –Mathematical morphology –Skeletonization –Subdivision surfaces –Similarity measures –Surface simplification –Geometric compression

17. Discipline-specific Toolkits Examples –vtk, The Visualization Toolkit –Open Inventor, Graphics –Insight, Segmentation and Registration –CGAL, Computational Geometry –vxl, Image Understanding –Khoros, Image Processing

18. vtk, The Visualization Toolkit Open source toolkit for scientific visualization, computer graphics, and image processing C++ Class Library 250,000 Lines of Code –(~120,000 executable) 20+ developers 8 years of development 1000 user mailing list public.kitware.com/VTK

19. Insight Segmentation and Registration Toolkit

20. What is it? A common Application Programmers Interface (API). –A framework for software development –A toolkit for registration and segmentation –An Open Source resource for future research A validation model for segmentation and registration. –A framework for validation development –Assistance for algorithm designers –A seed repository for validated segmentations

21. Who’s sponsoring it? The National Science Foundation The National Institute for Dental and Craniofacial Research The National Institute of Neurological Disorders and Stroke $7.5 million, 3 year contract

22. Who’s creating it?

23. Contractor Roles GE CRD/Brigham and Womens –Architecture, algorithms, testing, validation Kitware –Architecture, user community support Insightful (formerly MathSoft)/UPenn –Statistical segmentation, mutual information registration, deformable registration, level sets –Beta test management Utah –Level sets, low level image processing UNC/Pitt –Image processing, registration, high-dimensional segmentation UPenn/Columbia –Deformable surfaces, fuzzy connectedness, hybrid methods

24. Toolkit Requirements Shall handle large datasets –Visible Human data on a 512MB PC Shall run on multiple platforms –Sun, SGI, Linux, Windows Shall provide multiple language api’s Shall support parallel processing Shall have no visualization system dependencies Shall support multi-dimensional images Shall support n-component data

25. Insight - Schedule Alpha Release, April 4, 2001. –Source code snapshot –Some non-consortium participation Limited Public Alpha Version, Aug 8, 2001. Public Beta Release, December 15, 2001. Software Developer’s Consortium Meeting –Nov. 8-9, 2001, NLM, Bethesda. www.itk.org

26. Testing Design Distributed testing –Developers and users must be able to easily contribute testing results –Pulled together in a central dashboard Separate data from presentation Cross-platform solution Strive to have the same code tested in all locations

29. Using vtk and Insight Registration of Volumetric Medical Data

30. Mutual Information Computes “mutual information” between two datasets, a reference and target –MI(X,Y) = H(X) + H(Y) – H(X,Y) Small parameter set Developed by Sandy Wells (BWH) and Paul Viola (MIT) in 1995 Defacto standard for automatic, intensity based registration

31. Insight Mutual Information Registration There is no MI open source implementation The Insight Registration and Segmentation Toolkit has an implementation GE and Brigham as Insight contractors have early access to the code Code was developed at MathSoft (now called Insightful) GE was able to “guide” development with input from Sandy Wells

32. Longitudinal MRI Study Register multiple volumetric MRI datasets of a patient taken over an extended time Create a batch processing facility to process dozens of datasets Resample the datasets

33. Approach Validate the algorithm Pick a set of parameters that can be used across all the studies For each pair of datasets –Perform registration –Output a transform View the resampled source dataset in context with the target dataset

34. Division of Labor vtk itk vtk Read data Normalize data Export data Import Data Register Report transform Read data Reslice Display MRIRegistration.cxx MultiCompare.tcl

35. The Pipeline ImageReaderImageCast ImageShiftScale ImageStatistics ImageShrink3D ImageExport ImportImage ImageToImageRigidMutualInformationGradientDescentRegistration vnl_quaternion Matrix4x4

36. Oregon Data 25 Registrations 13 Subjects Qualitative comparison One set of parameters for all studies

37. Longitudinal MRI No Registration Checkerboard Source Original image Difference Target Original image

38. Longitudinal MRI Registration Checkerboard Source Original image Difference Target Original image

39. Multi Field MRI Data Register 1.5T and 3T to 4T data Resampled 1.5T and 3T to correspond to the 4T sampling Volume rendering of the 3 datasets from the same view

40. 1.5T vs 4T MRI No Registration Checkerboard Source Original Image Difference Target Original Image

41. 1.5T vs 4T MRI Registration Checkerboard Source Original Image Difference Target Original Image

42. 3D Visualization of the same subject Scanned with different MR field Strengths 4T 3T 1.5T All Registered To 4T

43. CT Lung Longitudinal Study Register two CT exams of the same patient taken at two different times Side-by-side synchronized view for visual comparison

44. Lung CT No Registration Checkerboard Source Original Image Difference Target Original Image

45. Lung CT Registration Checkerboard Source Original Image Difference Target Original Image

46. microPet/Volume CT

47. Back to the Software

48. Why Now? Internet enables distributed software development There are some successful Open Source projects A basic set of algorithms (and sometimes mathematics) exist Light weight software engineering processes exist –Low investment to support software development –Minimally invasive

49. Software Trends Lightweight Software Engineering Processes

50. IEEE Computer October, 1999

51. Extreme Programming

52. Extreme Testing

53. Continuous Testing

55. Insight Project Management Robust code repository (cvs) Active mailing list (mailman) Automated documentation (doxygen) Stable, cross platform build environment (cmake) Weekly t-cons Stable nightly build and test (300 builds) Continuous build Stable nightly dashboard (dart) Quarterly face-to-face developer meetings Semi-annual project meetings

56. Recipe for Success Vision Openness Community Strong core team Core Architecture Funding

57. Unification of Vision, Geometry and Graphics Through Toolkits Bill Lorensen GE Corporate R&D Niskayuna, NY lorensen@crd.ge.com