Stephen R. Aylward, Director of Medical Imaging, Kitware

CTK Workshop Heidelberg, June 29/30, 2009 Overview Open-source toolkits  Medical image processing (ITK)  Image-guided surgery (IGSTK)  Microscopy (FARSIGHT)  Image display (VTK)  Information visualization (VTK)  Scientific computing (VTK)  Computer vision (VidTK) Open-API  Databases (MIDAS)  Open-source applications  Medical images (VolView)  Scientific data (ParaView) NEW!  Open-source application building  ITK, VTK, Qt (Maverick)

CTK Workshop Heidelberg, June 29/30, 2009 Overview Open-source toolkits  Medical image processing (ITK)  Image-guided surgery (IGSTK)  Microscopy (FARSIGHT)  Image display (VTK)  Information visualization (VTK)  Scientific computing (VTK)  Computer vision (VidTK) Open-API  Databases (MIDAS)  Open-source applications  Medical images (VolView)  Scientific data (ParaView) NEW!  Open-source application building  ITK, VTK, Qt (Maverick)

CTK Workshop Heidelberg, June 29/30, 2009 IGSTK Features  PI: Kevin Clary, Georgetown (NIH)  Toolkit for application builders  Building blocks for C++ programmers  Trackers, display layouts, scenes, …  Scene graph based on ITK’s SpatialObjects  Objects are tied to trackers  Displays show objects in the scenes  State machine for application control  All events, not just GUI Tracker coordinates, type of data loaded, tracker unplugged  Patient safety is a top priority  State machine can be visualized and validated Confirm application never enters an invalid state  Some cost added for application developers.

CTK Workshop Heidelberg, June 29/30, 2009 FARSIGHT Features  PI: Badri Roysam, RPI (NIH)  Toolkit for biologists  Python scripting  3D/4D Image analysis and visualization  Scene graph based on ITK’s SpatialObjects  Arrangement of cells wrt vasculature  Scene spans time and space  Flexibility is a top priority  New methods for processing and display Created by biologists (scripts) and computer scientists (new ITK/VTK methods)

CTK Workshop Heidelberg, June 29/30, 2009 Information Visualization using VTK (OverView/ParaView) Figure 3 - The OverView application displaying a large graph linking miRNA sequences with the proteins they affect. Figure 4 - The OverView informatics application is shown retrieving PubMed articles from a search string ("HIV"), clustering them, and displaying the articles in geospatial, graph, landscape, and tabular views.

CTK Workshop Heidelberg, June 29/30, 2009 Maverick  PI: Kitware (AFRL)  Toolkit for application development  Qt modules for C++ programmers  “Qt designer” to define applications Signals / slots (GUI-driven)  Scene graph  Memory (mavScene – SpatialObjects / MRML)  Disk (MRML)  Display (mavSceneViewers)  Extensible  Slicer Execution Model  GUI (User) is the priority  Workflow of interdependent modules (maximize user-interaction re-use)  Consistent look-and-feel (GUI design standard)

CTK Workshop Heidelberg, June 29/30, 2009 Roadmap (dreams / milestones)  Maverick is released (weeks)  FARSIGHT and IGSTK momentum continues  Slicer converted to Qt and supports multiple workflows  …CTK evolves and is integrated into above systems

CTK Workshop Heidelberg, June 29/30, 2009 What's most important for open-source? NO DUPLICATION! - Specialization for an audience Community support  License  Transparent  Value added  User support  Funding beyond delivery Pick ONE audience  Clinical / biomedical research  1. Extensible application  2. Custom workflow applications Application builder  Algorithms research  Scripting  Pharma  Integration into existing workflows

CTK Workshop Heidelberg, June 29/30, 2009 What's most important for open-source medical imaging applications? Medical imaging  We are processing organs, tumors, vessels; not images.  Portable data on disk Crosses applications Patient record  Data representation in memory ITK Spatial Objects (scene graph) IsInside ValueAt Transform  GUI scene Present using intuitive names, hierarchy, etc. Why don’t commercial CAD companies fear open-source CAD efforts  Integration / turn-key  GUI  Patents  Clinical testing  Outreach  Cross-institution distribution  These are out targets

CTK Workshop Heidelberg, June 29/30, 2009 Dreams  Imagine…  Custom application from my institute walks me through a workflow for planning a brain tumor biopsy procedure  Save scene  Load scene into Slicer to merge with new DTI data  Save scene  Load scene into surgical suite running IGSTK (robotics)  Biopsy pre-chosen sites  Send biopsy to lab  Lab processes using a new research module loaded into their FARSIGHT-based application  Multiple biomarkers (multiple scales: MRI, DTI, cell, proteomics) are compared with a database of cases to predict best course of treatment.  Radiation oncology program loads scene and biopsy results...

CTK Workshop Heidelberg, June 29/30, 2009 Reality  Legacy applications, formats, data, expertise, …  Legacy funding…  Good news: different audiences  Opportunity for sharing  Solution: CTK  Scene definition (Disk and Memory)  IO Library for scene  Scene converters for other types of scenes/data  API and classes for methods that will run in Slicer, MITK, OpenMAF, Maverick, XIP,...

CTK Workshop Heidelberg, June 29/30, 2009 CTK Implementation  Scenes (Disk, Memory, Display)  IGSTK / ITK / MITK (SpatialObjects)  Slicer (MRML)  XIP (OpenInventor)  OpenSG  XNAT  Modular methods  Slicer Execution Model Implement as ITK/VTK command-line programs Automatically come available as shared libraries Passes events to parent program Missing: interaction with parent GUI (mouse clicks) Can be done, but no standard set  GUI Modules?