© Fraunhofer MEVIS , Heidelberg Collaboratory for Image Processing Frank Heckel, PhD Software Support for Oncological Therapy Response Assessment

© Fraunhofer MEVIS 1 / 52 FRAUNHOFER MEVIS Bremen Additional employees in Berlin, Leipzig, Heidelberg & Nijmegen Lübeck

© Fraunhofer MEVIS 2 / 52 Largest organization for applied research in Europe Areas of research: life science, communication, mobility, security, energy, environment 66 institutes, employees 2.0 billion EUR research budget, >70% from industry and public agencies Fraunhofer-Gesellschaft

© Fraunhofer MEVIS 3 / institutes in Germany Institutes Branches, Working Groups, Application Centers Bremerhaven Fraunhofer-Gesellschaft

© Fraunhofer MEVIS 4 / 52 Fraunhofer MEVIS Non-profit Commercial (~100 employees) (~150 employees) 51% Institute for Medical Image Computing Bremen (since 01/2009) Project Group Image Registration Lübeck (since 04/2010) MeVis BreastCare GmbH & Co. KG Bremen (since 10/2001) MeVis Medical Solutions AG Bremen (since 08/2007)

© Fraunhofer MEVIS 5 / 52 Image acquisition and reconstruction Image computing, analysis and visualization Modelling and simulation Application, workflow and usability engineering Computer assistance for image-based, personalized diagnosis and therapy Solutions for clinical problems

© Fraunhofer MEVIS 6 / 52 Competences Diagnosis Clinical Workflow Early Detection Diagnostic Planning Procedure Monitoring Therapy Organs Liver Lung Breast Brain Heart/Vessels Bones/Joints Methods MeVisLab Validation Navigation Risk analysis Visualization Quantification Segmentation Registration Modeling/ Simulation Imaging/ Modality

© Fraunhofer MEVIS 7 / 52 Organization Chart Institute Directors Prof. Kikinis, Prof. Hahn Extended Committee Steering Committee plus representatives for: Software/IT, QA, Employees, Equal Rights, WTR, PR Steering Committee Prof. Kikinis, Prof. Hahn, T. Forstmann, Prof. Preußer, Prof. Günther, Prof. Modersitzki, Dr. Heldmann, Dr. Olesch, Dr. Papenberg, Dr. Kraß, Dr. Lang, Dr. Prause Administration T. Forstmann Advisory BoardEmployee Representatives

© Fraunhofer MEVIS 8 / 52 Organization of Work Team-oriented Open-minded Self-organized Flexible Adaptive

© Fraunhofer MEVIS 9 / 52 Certification Certificate for quality assurance Introduction and application of a quality management system in compliance with EN ISO 9001 & EN ISO (medical devices) Since 2005 in Bremen Since 2012 in Lübeck Scope: Research and development for computer assistance of medical diagnosis and therapy Development and production of software for medical products

© Fraunhofer MEVIS 10 / 52 University of Bremen Mathematics (H.-O. Peitgen, until Sep 2012) Medical Image Computing (R. Kikinis, since Jan 2014) MR Imaging & Physics (M. Günther) Jacobs University Bremen Analysis & Visualization (H. Hahn) Modeling & Simulation (T. Preußer) University of Lübeck Mathematics & MEVIS Project Group (B. Fischer †, J. Modersitzki) University of Nijmegen Computer-Aided Detection & Diagnosis (N. Karssemeijer, B. van Ginneken) Links to Universities

© Fraunhofer MEVIS 11 / 52 INNOVATION CENTER COMPUTER ASSISTED SURGERY (ICCAS)

© Fraunhofer MEVIS 12 / 52 Innovation Center Computer Assisted Surgery (ICCAS) Part of medical faculty Universität Leipzig Clinical disciplines: ENT-surgery, Heart surgery, Neurosurgery

© Fraunhofer MEVIS 13 / 52 ICCAS Research Areas STDDPMMAI MAI – Model-based automation and integration, DPM – Digital patient model, STD - Standardization

© Fraunhofer MEVIS 14 / 52 Research Area: Model-based Automation and Integration Navigation data Model visualisations System monitoring Tracked ultrasound probe Augmented Reality for microscopes Ultrasound imaging Information and communication technology in the OR Head: Prof. Thomas Neumuth

© Fraunhofer MEVIS 15 / 52 Research Area: Model-based Automation and Integration patientsurgeon Surgical Workflow HMI ImagingNavigation

© Fraunhofer MEVIS 16 / 52 Research Area: Model-based Automation and Integration Integration into therapeutic process Ressource monitoring Process monitoring Workflow management Data consolidation and integration

© Fraunhofer MEVIS 17 / 52 Research Area: Digital Patient Models Head: Dr. Kerstin Denecke

© Fraunhofer MEVIS 18 / 52 Research Area: Standardization Head: Prof. Heinz Lemke

© Fraunhofer MEVIS 19 / 52 Research Area: Image-guided Interventions Head (and Insitute Director): Prof. Andreas Melzer

© Fraunhofer MEVIS 20 / 52 ONCOLOGICAL THERAPY RESPONSE ASSESSMENT

© Fraunhofer MEVIS 21 / 52 Overview Background Semi-Automatic Segmentation Segmentation Editing Partial Volume Correction The Ground Truth Problem Workflow Aspects

© Fraunhofer MEVIS 22 / 52 Background Cause for 13% of all deaths worldwide Every 2 nd man gets cancer  every 4 th dies Treatment examples: Surgery Radiotherapy Radiofrequency ablation and … Chemotherapy Lung nodules, metastases, enlarged lymph nodes Systemic treatment Severe side effects Different agents Cancer and Chemotherapy

© Fraunhofer MEVIS 23 / 52 Background CT-Based Follow-Up Examination Baseline Find tumorsFind tumors Identify target lesionsIdentify target lesions Measure target lesionsMeasure target lesions ReportingReporting 1st Follow-Up Find target lesionsFind target lesions Measure responseMeasure response Look for new lesionsLook for new lesions ReportingReporting Additional Follow-Ups 3-6 months

© Fraunhofer MEVIS 24 / 52 Background Change in tumor size is an important criterion RECIST 1 1.1: Sum of maximum diameters of target lesions  Relative change Volume is a more accurate measure Many tumors grow/shrink irregularly in 3D Requires appropriate segmentation Progress/response not defined Not used in clinical routine Oncological Therapy Response Monitoring 1 RECIST: Response Evaluation Criteria In Solid Tumors Complete Response Partial Response Stable Disease Progressive Disease Disappearance< -30%-30% – 20%> +20%

© Fraunhofer MEVIS 25 / 52 Background Diameter vs. Volume complete response no longer visible partial response < -30% > + 73% stable disease Small change < -66% progressive disease > +20% Classification Diameter Volume

© Fraunhofer MEVIS 26 / 52 Background Simulated example: Measured 2% change Reality: 26% change (roughly double volume!) Robustness of Diameter Measurement

© Fraunhofer MEVIS 27 / 52 The Segmentation Problem Ultimate Goal: Automatic segmentation for a wide range of objects Reproducible results with no effort for the user Solutions for specific purposes Might fail (low contrast, noise, biological variability) Unsolved or insufficient for many real-world problems Alternatives: Manual segmentation Semi-automatic or interactive tools (Semi-)automatic algorithm followed by manual correction Drawback: Variability due to different inputs or judgment

© Fraunhofer MEVIS 28 / 52 Semi-Automatic Segmentation Familiar user Interaction: draw the (maximum) diameter Core method: “Smart Opening” 1 Region Growing Erosion Dilation Refinement Specific variation for lung nodules, liver metastases and lymph nodes 2 For lymph nodes a spiral-scanning solution has been developed as well 3 1 Kuhnigk et al., IEEE TMI, 25(4), Moltz et al., IEEE Journal of Selected Topics in Signal Processing, 3(1), Wang et al., SPIE Medical Imaging, 2012

© Fraunhofer MEVIS 29 / 52 Semi-Automatic Segmentation Examples for Challenging Lung Nodules

© Fraunhofer MEVIS 30 / 52 Semi-Automatic Segmentation Positive examples: Negative examples: Examples for Challenging Liver Metastases

© Fraunhofer MEVIS 31 / 52 Semi-Automatic Segmentation Smart Opening (top) vs. Spiral Scanning (bottom) Examples for Challenging Lymph Nodes

© Fraunhofer MEVIS 32 / 52 Semi-Automatic Segmentation Lung: LIDC-Data (674 cases (solid nodules), 4 reference segmentations) Liver: MDS-Data (371 cases, 1 reference segmentation) Evaluation Volume overlapHausdorff distanceComputation time Lung68,3%2,46 mm0,41 s Liver62,6%4,20 mm0,75 s

© Fraunhofer MEVIS 33 / 52 Semi-Automatic Segmentation Clinical Evaluation: Amount of Lesions that have not been manually corrected Lung Liver Evaluation

© Fraunhofer MEVIS 34 / 52 Semi-Automatic Segmentation Clinical Evaluation: Amount of Lesions that have not been manually corrected Lymph nodes Evaluation

© Fraunhofer MEVIS 35 / 52 Segmentation Editing Most existing methods are low-level and unintuitive in 3D High-level correction has not received much attention in research

© Fraunhofer MEVIS 36 / 52 Segmentation Editing Sketch-Based Editing in 2D add remove add + remove replace

© Fraunhofer MEVIS 37 / 52 Segmentation Editing Image-based method (→ shortest path) Image-independent method (→ RBF-based 3D object reconstruction) 3D Extrapolation Heckel et al., Computer Graphics Forum, 32(8), 2013

© Fraunhofer MEVIS 38 / 52 Segmentation Editing 131 representative tumor segmentations in CT (lung nodules, liver metastases, lymph nodes) 5 radiologists with different level of experience Editing rating score: Qualitative Evaluation Heckel et al., SPIE Journal of Medical Imaging, 1(3), 2014

© Fraunhofer MEVIS 39 / 52 Segmentation Editing Quantitative Evaluation

© Fraunhofer MEVIS 40 / 52 Segmentation Editing Problem: High effort and bad reproducibility of user studies Idea: Replace user by a simulation Benefits: Objective and reproducible validation Objective comparison Improved regression testing Better parameter tuning Simulation-Based Evaluation

© Fraunhofer MEVIS 41 / 52 Segmentation Editing Step 1: Find most probably corrected 3D error Step 2: Select slice and view where the error is most probably corrected Step 3: Generate user-input for sketching Step 4: Apply editing algorithm Simulation-Based Evaluation Heckel et al., Scandinavian Conferences on Image Analysis, 2013

© Fraunhofer MEVIS 42 / 52 Segmentation Editing Simulation-Based Evaluation

© Fraunhofer MEVIS 43 / 52 Partial Volume Correction Smoothing effect caused by limited spatial resolution (of CT) Ill-defined border between tumor and healthy tissue, making segmentation an ill-defined problem Could cause significant differences in size measurements The Partial Volume Effect 28.4 ml (-27.5%) 39.2 ml 56.8 ml (+44.9%)

© Fraunhofer MEVIS 44 / 52 Partial Volume Correction Method Heckel et al., IEEE TMI, 33(2), 2014

© Fraunhofer MEVIS 45 / 52 Partial Volume Correction Software Phantom Results

© Fraunhofer MEVIS 46 / 52 Partial Volume Correction Hardware Phantom Results

© Fraunhofer MEVIS 47 / 52 Partial Volume Correction Multi-Reader Data Results

© Fraunhofer MEVIS 48 / 52 The Ground Truth Problem Expert segmentations differ significantly Variability depends on several aspects (lesion size, contrast, partial volume effects, interpretation, …) We need to consider n>1 reference segmentations Who are experts? Only clinicians? There is no „Ground Truth“! Jan Moltz, PhD Thesis, Jacobs University Bremen, 2013

© Fraunhofer MEVIS 49 / 52 The Ground Truth Problem What is a „good“ segmentation result?

© Fraunhofer MEVIS 50 / 52 Workflow Aspects CAD Lesion Matching Visualization Reporting Schwier et al., IJCARS, 6(6), 2011 Schwier et al., CARS 2009Jan Moltz et al., ISBI, 2009

© Fraunhofer MEVIS 51 / 52 Workflow Aspects Prototyping

© Fraunhofer MEVIS 52 / 52 Thank you!

© Fraunhofer MEVIS 53 / 52