1. NA-MIC National Alliance for Medical Image Computing http://na-mic.org Large-Scale Computing Frameworks for Developing Image Analysis Tools Steve Pieper, PhD Isomics, Inc. Founder and CEO of Isomics, Inc. a technology development company that works closely with multiple research institutions on projects including NIH grants and software development. Much of this work is done in collaboration with the Surgical Planning Laboratory at BWH.

2. National Alliance for Medical Image Computing http://na-mic.org 2 Acknowledgments F. Jolesz, R. Kikinis, C. Tempany, P. Black, S. Wells, CF. Westin, M. Halle, N. Hata, T. Kapur, A.Tannenbaum, M. Shenton, E. Grimson, P.Golland, W.Schroeder, J. Miller, N. Aucoin, K. Hayes, A. Yarmarkovich, C. Lisle, D. Marcus, J. Miller, R. Gollub, S. Pujol, S. Barre, W. Plesniak, B. Fischl, D. Greve and many more….

3. National Alliance for Medical Image Computing http://na-mic.org 3 Overview Software Infrastructure –NA-MIC Kit Components –3D Slicer Modules and Extensibility Population Analysis Organization and Community

4. National Alliance for Medical Image Computing http://na-mic.org Large Scale Computing Working at a High Level –Major Functional Components are Readily Available –Tested, Cross Platform, Documented… Suite of Tools and Systems for Organizing and Processing –Databases and Batch Computing Mechanisms to Translate Research to Clinical Users –Plug New Research into a Usable Platform A Talented and Productive Community –Multidisciplinary –National and International 4

5. National Alliance for Medical Image Computing http://na-mic.org 5 NA-MIC Kit The NA-MIC Kit is a Collection of Software and Methodologies for Medical Image Computing Most components of the NA-MIC kit predate NA- MIC (National Alliance for Medical Image Computing)

6. National Alliance for Medical Image Computing http://na-mic.org 6 NA-MIC Kit Packaging of: –Applications –Algorithms (toolkits) –Methodologies

7. National Alliance for Medical Image Computing http://na-mic.org 7 NA-MIC Kit Components End User Application –3D Slicer Image Analysis, Visualization, and GUI libraries –ITK, VTK, KWWidgets, Teem Archive and Distributed Processing Tools –XNAT, Batchmake, BIRN GRID tools Software Engineering Tools –CMake, CDash, CTest, CPack http://www.na-mic.org/Wiki/index.php/SoftwareInventory Provided by Pieper, Kikinis

8. National Alliance for Medical Image Computing http://na-mic.org 8 Visualization Toolkit - vtk Open source toolkit for scientific visualization, computer graphics, and image processing

9. National Alliance for Medical Image Computing http://na-mic.org 9 With support from several IC’s

10. National Alliance for Medical Image Computing http://na-mic.org 10 KWWidgets Usage –Cross-platform GUI with open source license –Object Oriented C++ API (very VTK-like) Active Development –New widgets –Work flow support –File/directory browser Future –Tracing (for testing) –Registration inspection widget –Camera control widget –Interface to ITK (spatial objects) –Cleanup kwwidgets.org –cvs, dashboard, bug tracker, wiki… –BSD license

11. National Alliance for Medical Image Computing http://na-mic.org 11 NA-MIC Software Process Source CDash Testing Dashboard Version Control Svn, CVS with web access Developers review results Developers check-in code CMake, CTest Multiplatform Software Compilation, Testing

12. National Alliance for Medical Image Computing http://na-mic.org 12 Development Methodology 6 months 4 months 2 months ReleasePatchNightlyContinuous Release X.Y Release X.Y.1 Release X.Y.2 Release X.Y.3 Extreme lifecycle Private Sandbox NA-MIC Sandbox Slicer ITK Dashboard CMake CTest CPack Testing

13. National Alliance for Medical Image Computing http://na-mic.org 13 NA-MIC Software Process Emphasizes Cross Platform Testing Provides informal –Requirements specification –Design documents Which are captured via collaboration tools –Tcon, email, wiki, software documentation Traceability –Implicit in logs and collaboration

14. National Alliance for Medical Image Computing http://na-mic.org 14 What is 3D Slicer? A platform for image analysis and visualization Current Releases 2.7 and 3.2 –2.x most features and documentation –3.x focus of current activity A freely-downloadable program –Source code and executables available for Windows, Linux, and Mac OS X Slicer is a research platform: –NOT an FDA approved medical device –NOT finished – some parts will work better than others

15. National Alliance for Medical Image Computing http://na-mic.org Slicer 3.2 May 2008 Kitware, Inc. GE Research Isomics, Inc. BWH UCLA UCSD U Iowa MIT GaTech UNC U Utah MGH UCI Harvard WashU… NAC NCIGT NA-MIC mBIRN fBIRN… http://slicer.org

16. National Alliance for Medical Image Computing http://na-mic.org 16 3D Slicer Numbers Numbers May 2007: Subversion Commits: 3,407 Lines of Code*: 371,428 Bugs & Features: 154 Submitted 63 Closed Active Developers with svn access † : 33 Numbers July 2008: Subversion Commits: 7303 Lines of Code*: 448,175 Developers: 42 Image provided by A. Golby, F. Talos, P. Black *: find. -iname \*.h -o -iname \*.cxx -o -iname \*.tcl -o -iname \*.java -o -name \*.py | grep -v svn | xargs wc †: svn log | grep "^r" | cut -d " " -f 3 | sort | uniq | wc

17. National Alliance for Medical Image Computing http://na-mic.org 17 Slicer3 “Observer MVC” Pattern MRML (Model) –For Scene Description and Application State –MRML Nodes are Persistent and Undoable –Scene and Nodes are Observable –XML Serialization, Undo/Redo Logic Encapsulate VTK and ITK Pipelines (Controller) –Observe MRML to Configure Pipelines –Help Create/Manage Nodes –No UI Components (no Widgets, Actors, Mappers, Renderers or RenderWindows) GUI (View) –Observe and Edit MRML –Interact with User and Display Hardware Logic MRML Nodes GUI WidgetsRenderers Edit Observe Edit “Observe” means generic event mechanisms are used to pass information. “Edit” means code can directly call methods. Example: GUI can call methods in Logic classes, but Logic cannot call GUI methods. MRML cannot call Logic or GUI methods. There can be many observers for any event.

18. National Alliance for Medical Image Computing http://na-mic.org 18 Scene Description Provided by D. Gering

19. National Alliance for Medical Image Computing http://na-mic.org 19 Image/Scene Management XML-Based MRML File Stores Scene Description –Volumes (Images, Label Maps) –Models –Hierarchical Affine Transforms –Scene Data (Cameras, Colors, Fiducials, etc). Manipulated in World Coordinates based on Patient RAS Provided by S. Pieper

20. National Alliance for Medical Image Computing http://na-mic.org 20 Alignment of all pre-operative datasets to the intra-operative images achieved during the neurosurgery. Provided by Archip, Warfield Neurosurgery Example Archip N, Clatz O, Whalen S, Kacher D, Fedorov A, Kot A, Chrisochoides N, Jolesz F, Golby A, Black PM, Warfield SK. Non-rigid alignment of pre- operative MRI, fMRI, and DT-MRI with intra-operative MRI for enhanced visualization and navigation in image- guided neurosurgery. Neuroimage. 2007 Apr 1;35(2):609-24

21. National Alliance for Medical Image Computing http://na-mic.org Modules 21

22. National Alliance for Medical Image Computing http://na-mic.org Slicer3 Command Line Modules Common architecture for interactive and batch processing Slices Courtesy Jim Miller

23. National Alliance for Medical Image Computing http://na-mic.org Each module has … … an entry in the module menu … a panel of user interface controls

24. National Alliance for Medical Image Computing http://na-mic.org Slicer Modules Interactive Modules –Interact Closely with GUI –Volume Rendering, Models, Editor… Command Line Modules –“Batch Mode” (Can be run from command line or script) –Registration, Filtering, DTI Tractography… Both Interactive and Command Line Modules are Dynamically Loaded –They can be Built and Distributed Independent of Slicer Today we will look in detail only at Command Line Modules

25. National Alliance for Medical Image Computing http://na-mic.org 25 EM Interactive Module MRMLLogicGUI Global Parameters Hierarchical parameters Image data Segmentation output Manage MRML nodes API for access to parameters Manage hierarchy Segmentation algorithm Window to parameter set Wizard Interaction with images Provided by B. Davis

26. National Alliance for Medical Image Computing http://na-mic.org 26 Command Line Module MRMLLogicCLI Global Parameters Hierarchical parameters Image data Segmentation output Manage MRML nodes API for access to parameters Manage hierarchy Segmentation algorithm XML Parameter File File I/O Batchable Provided by B. Davis

27. National Alliance for Medical Image Computing http://na-mic.org Existing Command Line Modules Demonstration –Execution Model Tour –Your new hello world module Converters –Create a DICOM Series –Dicom DWI loader –Dicom to Nrrd –GE Dicom to NRRD Converter –Orient Images Filtering –Calculate Volume Statistics –CheckerBoard Filter –Extract Skeleton –Histogram Matching –Otsu Threshold –Resample Volume/ Resample Volume 2 –Voting Binary Hole Filling –Zero Crossing Based Edge Detection Filter –Filtering.Arithmetic: Add Images Subtract Images

28. National Alliance for Medical Image Computing http://na-mic.org Existing Command Line Modules Filtering.Denoising –Curvature Anisotropic Diffusion –Gaussian Blur –Gradient Anisotropic Diffusion Median Filter Segmentation –EMSegment Simple –FreesurferSurfaceSectionExtraction –Otsu Threshold Segmentation –Simple region growing Filtering.Morphology –Grayscale Fill Hole –Grayscale Grind Peak Model Generation –Grayscale Model Maker –Label Map Smoothing –Model Maker –Multiple models example –Probe Volume With Model (Paint) Meshing.VoxelMesh –Voxel Meshing

29. National Alliance for Medical Image Computing http://na-mic.org Registration Command Line Modules Registration –Affine registration –Deformable BSpline registration –(Utah) Deformable BSpline registration –Linear registration –RealignVolume

30. National Alliance for Medical Image Computing http://na-mic.org Diffusion Command Line Modules Diffusion Tensor –Resample DTI Volume Diffusion Tensor Estimation Diffusion Tensor Scalar Measurements Simple DWI IO Test Rician LMMSE Image Filter Tractography.Editor: ROISelect Tractography.Seeding Stochastic Tractography –Generate Connectivity Map –Stochastic ROI Tract Filter –Stochastic Stochastic Tractography Filter

31. National Alliance for Medical Image Computing http://na-mic.org Example Modules http://www.nitrc.org/projects/slicer3examples/ http://www.na-mic.org/Wiki/index.php/Slicer3.2:Training

32. National Alliance for Medical Image Computing http://na-mic.org Parameters | | | | | | | | | | | [type="scalar|label|tensor|diffusion-weighted|vector|model"] | [type="fiberbundle|model"] | [multiple="true|false"] [coordinateSystem="lps|ras|ijk"] | [multiple="true|false"] [coordinateSystem="lps|ras|ijk"] GUI Automatically Generated from XML C++ Argument Parsing Code Automatically Generated from XML

33. National Alliance for Medical Image Computing http://na-mic.org Parameter Example 33 HelloWorld Example … Input/Output Input/output parameters helloSlicerInputVolume Input Volume input 0 None Input volume … Discrete Gaussian Parameters Parameters of the Discrete Gaussian Filter variance --variance Variance ( width of the filter kernel) Variance 0.5

34. National Alliance for Medical Image Computing http://na-mic.org C++ Module 34 #include #include "HelloSlicerCLP.h" #include "itkImage.h" #include "itkImageFileReader.h" #include "itkImageFileWriter.h" #include "itkDiscreteGaussianImageFilter.h" int main(int argc, char * argv []) { PARSE_ARGS; typedef itk::Image ImageType; typedef itk::ImageFileReader ReaderType; typedef itk::ImageFileWriter WriterType; ReaderType::Pointer reader = ReaderType::New(); WriterType::Pointer writer = WriterType::New(); reader->SetFileName( helloSlicerInputVolume.c_str() ); writer->SetFileName (helloSlicerOutputVolume.c_str()); typedef itk::DiscreteGaussianImageFilter FilterType; FilterType::Pointer filter = FilterType::New(); try { filter->SetInput(reader->GetOutput()); filter->SetVariance(variance); writer->SetInput(filter->GetOutput()); writer->Update(); } catch (itk::ExceptionObject &excep) { std::cerr << argv[0] << ": exception caught !" << std::endl; return EXIT_FAILURE; } return EXIT_SUCCESS; }

35. National Alliance for Medical Image Computing http://na-mic.org Python Module XML = """ Filtering.Denoising... def toXML(): return XML; def Execute ( inputVolume, outputVolume, conductance=1.0, timeStep=0.0625, iterations=1 ): print "Executing Python Demo Application!" Slicer = __import__ ( "Slicer" ); slicer = Slicer.Slicer() in = slicer.MRMLScene.GetNodeByID ( inputVolume ); out = slicer.MRMLScene.GetNodeByID ( outputVolume ); filter = slicer.vtkITKGradientAnisotropicDiffusionImageFilter.New() filter.SetConductanceParameter ( conductance ) filter.SetTimeStep ( timeStep ) filter.SetNumberOfIterations ( iterations ) filter.SetInput ( in.GetImageData() ) filter.Update() out.SetAndObserveImageData(filter.GetOutput()) return

36. National Alliance for Medical Image Computing http://na-mic.org Behind the scenes Tasks queued for processing thread Three types of modules: –executable, –shared object, and –python modules Scalar images sent via files for executables and memory* for shared objects and python Vector images, tensor images, geometry, tables, transforms sent via files Scalars, file names, directories, fiducials, regions sent via command line * VTK-based modules using scalar images are only supported as executable (command line) modules.

37. National Alliance for Medical Image Computing http://na-mic.org Population Statistics 37

38. National Alliance for Medical Image Computing http://na-mic.org BIRN Integration BIRN acquisition protocols distortion correction tools local databases workflows analysis tools population statistics...... PubMed IBVD BrainInfo Visualization & Interpretation (.xcat) (.xar)

39. National Alliance for Medical Image Computing http://na-mic.org mBIRN Informatics BIRN Data Repository (BDR) eXtensible Neuroimaging Archive Tool (XNAT) http://www.xnat.org XML-Based Clinical Experiment Data Exchange Schema (XCEDE) http://www.xcede.org Stores Images, Demographics, Clinical Data, Analysis Results 3D Slicer Interoperability (currently read-only, read/write prototype exists)

40. National Alliance for Medical Image Computing http://na-mic.org Query Atlas Module BIRN Scientific Interpretation Tool –Compatible with fMRI and Morphometry Datasets in BIRN- Standard Formats Provides a Link Between Images and Text Databases –Uses Anatomic Labeling + Ontologies –Links to Definitions, Publications, Quantifications –E.g. Wikipedia, PubMed, IBVD, BrainInfo

41. National Alliance for Medical Image Computing http://na-mic.org Query Atlas Features Input:.xcat or.qdec from.xar Hardware Accelerated Interactive 3D Annotation Ontology Engine –FreeSurfer, UMLS, BIRNLex, NeuroNames, IBVD –Interactive Browser Direct Browser Launch to Search Sites using Selected Terms

42. National Alliance for Medical Image Computing http://na-mic.org BIRN Population Statistics Query, Design, Estimate, Contrast of Population Statistics – XNAT to Select Subjects – FreeSurfer QDEC Runs on Server –.QDEC file in.XAR Web Download – Slicer3 Interactive Visualization – Integrated with Query Atlas

43. National Alliance for Medical Image Computing http://na-mic.org 43 Community

44. National Alliance for Medical Image Computing http://na-mic.org 44 Overall Goals Why Medical Research? –To Help Patients!

45. National Alliance for Medical Image Computing http://na-mic.org 45 NA-MIC Roots: NIH Roadmap Initiative http://grants1.nih.gov/grants/guide/rfa-files/RFA- RM-04-003.html –…will create a networked national effort to build the computational infrastructure for biomedical computing in the nation… –The establishment of the NIH NCBC was called for in the Biomedical Information Science and Technology Initiative report in 1999

46. National Alliance for Medical Image Computing http://na-mic.org 46 NA-MIC Governance Software Sharing: http://grants1.nih.gov/grants/guide/rfa-files/RFA-RM-04-003.html …software should be freely available … …permit the commercialization of enhanced or customized versions … …include the ability of researchers outside the center and its collaborating projects to modify the source code and to share modifications …

47. National Alliance for Medical Image Computing http://na-mic.org 47 NA-MIC Kit Governance Free, BSD style Open-Source NA-MIC Provides “Reference Standard Distribution” –Maintain Official Releases –Avoid Code Forking, Avoid Non-Free Code Creeping into Core No “Knowing” Addition of Patented Techniques into NA-MIC Kit Technology Base for Suited for both Research and Commercialization –Okay to build GPL or Closed SW on top of NA-MIC Kit –But no GPL or Closed SW in NA-MIC Kit Emerging community process for contributions http://insight-journal.org Support Required for Contributions (no “shoot and forget”) Copyright to Substantive Works Remains with Creators –Available under NA-MIC Approved Licenses for Distribution with Kit

48. National Alliance for Medical Image Computing http://na-mic.org 48 FOSS – A Public Highway… Open-source is like a Public Road System –Provides Infrastructure for a Variety of Uses –Driveways can Lead to Anything: a Public Park a Private Facility FOSS= Free Open Source Software Provided by Pieper, Kikinis

49. National Alliance for Medical Image Computing http://na-mic.org 49 NA-MIC Policy Industry GPL NA-MIC Kit Research Only

50. National Alliance for Medical Image Computing http://na-mic.org 50 NA-MIC: A Network of Peers Leadership: –BWH: Ron Kikinis, (Overall PI) Core 1 Algorithms –Utah: Ross Whitaker (Core 1 PI) –MIT: Eric Grimson –UNC: Guido Gerig –MGH: Bruce Fischl, Dave Kennedy –GaTech: Allen Tannenbaum Core 2 Engineering –Kitware: Will Schroeder (Core 2 PI) –GE: Jim Miller –Isomics: Steve Pieper –UCSD: Mark Ellisman –UCLA: Art Toga –WashU: Dan Marcus Core 3 DBP 2004-2007 –BWH: Martha Shenton –Dartmouth: Andy Saykin –UCI: Steve Potkin –UofT: Jim Kennedy DBP 2007 –UNC: H. Cody –BWH: M. Kubicki –Mind Institute: J. Bockolt –Queens University: G. Fichtinger Core 4 Service –Kitware: Will Schroeder Core 5 Training –MGH: Randy Gollub Core 6 Dissemination –Isomics: Steve Pieper, Tina Kapur Core 7 Management –BWH: S. Manandhar Provided by Pieper, Kikinis

51. National Alliance for Medical Image Computing http://na-mic.org 51 Communication Daily – e-mail, dashboards, wikis Weekly – telephone conferences Periodic – architecture reviews, workshops Semi-annual– Programmer/Project week Yearly – All Hands Meeting

52. National Alliance for Medical Image Computing http://na-mic.org 52 All Hands Meeting

53. National Alliance for Medical Image Computing http://na-mic.org 53 Programmer’s Week

54. National Alliance for Medical Image Computing http://na-mic.org NA-MIC Project Week June 23-27, 2008 –120 Participants –Dozens of Projects –Hands On Software Development

55. National Alliance for Medical Image Computing http://na-mic.org 55 Dissemination and Training National and International Events –MIT, MGH, UNC, EPFL, NIH, UNM, UCSD… All Materials on Wiki Project Weeks –Full Week Each Summer –½ Week at Winter AHM Workshops –MICCAI 2005, 2006, 2007, 2008.. –OHBM, RSNA, Munich, NCI…

56. National Alliance for Medical Image Computing http://na-mic.org Training Courses June In Munich –2 Packed Days, 40 Participants –Cover Spectrum of Use and Development Data Visualization Registration Tracked Surgical Instruments Developing Custom Modules Upcoming –NIH/NCI, Robarts Inst., Stanford, MICCAI, RSNA, WashU > 1,000 Participants Since 2004

57. National Alliance for Medical Image Computing http://na-mic.org NAMIC Training Portfolio http://www.na-mic.org/Wiki/index.php/Slicer:Workshops:User_Training_101 Google for “Slicer 101” Most training material is for Slicer2.x with increasing amounts of Slicer3 content

58. National Alliance for Medical Image Computing http://na-mic.org 58 It’s all great, but… A few caveats to keep in mind It’s a big, distributed group –Good spin: “Wisdom of the Crowds” –Bad spin: “Designed by Committee” It’s all research –Constantly evolving –Never enough documentation –Plurality of approaches The net result is overwhelmingly positive.

59. National Alliance for Medical Image Computing http://na-mic.org 59 Getting Involved Winter Project Week 2009 –January, Salt Lake City, UT Summer Project Week 2009 –June at MIT Stata Center, Cambridge MA Collaborations PAR-05-063 –Automated FE Mesh Development N. Grosland PI, U Iowa + Isomics, Inc. –Measuring Alcohol and Stress Interaction with Structural and Perfusion MRI J. Daunais PI, Wake Forest University and Virginia Polytechnic Institute + K. Pohl BWH –An Integrated System for Image-Guided Radiofrequency Ablation of Liver Tumors K. Cleary PI, Georgetown + N. Hata BWH –http://www.na-mic.org/Wiki/index.php/Collaborator:Resources

60. National Alliance for Medical Image Computing http://na-mic.org Summary End-to-End Platform for Translational Image Research Open Source and Extensible Standard Methodologies Facilitate Sharing and Communication 60

61. National Alliance for Medical Image Computing http://na-mic.org 61 More Information SPL website: http://www.spl.harvard.edu NA-MIC wiki: http://wiki.na-mic.org 3D Slicer http://www.slicer.org