System Challenges in Image Analysis for Radiation Therapy Stephen M. Pizer Kenan Professor Medical Image Display & Analysis Group University of North Carolina Co-authors: Edward L. Chaney, Julian G. Rosenman Credits to many others in UNC MIDAG Stephen M. Pizer Kenan Professor Medical Image Display & Analysis Group University of North Carolina Co-authors: Edward L. Chaney, Julian G. Rosenman Credits to many others in UNC MIDAG

Objective: Segmentation in Radiation Treatment Planning & Delivery ä ä Target problems, in everyday RT ä ä Planning radiotherapy ä ä Segmentation of normal organs to be spared ä ä New patients ä ä Kidney, liver, head and neck ä ä Segmentation of regions implied by segmented organs: lymph levels ä ä Adaptive radiotherapy, incl. IGRT ä ä Segmentation of organs to be spared and target organ ä ä Day to day changes within a patient ä ä Male pelvic organs: bladder, prostate, rectum ä ä Target problems, in everyday RT ä ä Planning radiotherapy ä ä Segmentation of normal organs to be spared ä ä New patients ä ä Kidney, liver, head and neck ä ä Segmentation of regions implied by segmented organs: lymph levels ä ä Adaptive radiotherapy, incl. IGRT ä ä Segmentation of organs to be spared and target organ ä ä Day to day changes within a patient ä ä Male pelvic organs: bladder, prostate, rectum bladder, protsate, rectum

Goal: Segmentation for Radiation Treatment Planning & Delivery ä ä Our objective: A whole new level of segmentation ability ä ä On tough images ä ä As good as humans in most cases ä ä The principle ä ä Make use of probability distribution of geometric variation: p(m) ä ä Make use of probability distribution of geometry-relative intensity patterns p(I | m) ä ä Posterior optimization ä ä arg max m p(m | I) = arg max m [log p(I | m) + log p(m) ] ä ä Our objective: A whole new level of segmentation ability ä ä On tough images ä ä As good as humans in most cases ä ä The principle ä ä Make use of probability distribution of geometric variation: p(m) ä ä Make use of probability distribution of geometry-relative intensity patterns p(I | m) ä ä Posterior optimization ä ä arg max m p(m | I) = arg max m [log p(I | m) + log p(m) ] Bladder CTs Prostate

To achieve segmentation via Obtaining Training Data To achieve segmentation via p(m) and p(I | m): Obtaining Training Data ä ä Fitting m to training binary images ä ä arg min m f(m | binary I) = arg min m [image match penalty + geometric penalty] ä ä Tight fit of geometric model to binary is critical ä ä Extracting object-relative intensity patterns from corresponding CT images ä ä The tight fit of geometric model to binary makes regional intensity patterns more informative ä ä Fitting m to training binary images ä ä arg min m f(m | binary I) = arg min m [image match penalty + geometric penalty] ä ä Tight fit of geometric model to binary is critical ä ä Extracting object-relative intensity patterns from corresponding CT images ä ä The tight fit of geometric model to binary makes regional intensity patterns more informative

Representing Representing m and I | m ä ä Principle: representation should support PCA ä ä Representing object geometry m ä ä M-rep: sheet of medial atoms ä ä Captures local twisting, bending, magnification of interior ä ä Unfamiliar to physicians ä ä Representing image pattern relative to geometry ä ä I | m = = I relative to m = RIQF(interior), RIQF(exterior) ä ä RIQF: regional intensity quantile function ä ä Unfamiliar to physicians ä ä Principle: representation should support PCA ä ä Representing object geometry m ä ä M-rep: sheet of medial atoms ä ä Captures local twisting, bending, magnification of interior ä ä Unfamiliar to physicians ä ä Representing image pattern relative to geometry ä ä I | m = = I relative to m = RIQF(interior), RIQF(exterior) ä ä RIQF: regional intensity quantile function ä ä Unfamiliar to physicians Prostate

Segmentation Program ä ä Initialize pose of mean according to image landmarks ä Conjugate gradient optimization of over coefficients of 9 principal geodesics of ä Conjugate gradient optimization of log p(I | m) + log p(m) over coefficients of 9 principal geodesics of p(m) ä ä Objects are thereby restricted to credible shapes ä ä Initialize pose of mean according to image landmarks ä Conjugate gradient optimization of over coefficients of 9 principal geodesics of ä Conjugate gradient optimization of log p(I | m) + log p(m) over coefficients of 9 principal geodesics of p(m) ä ä Objects are thereby restricted to credible shapes

System Challenges ä ä Challenges of doing the research within a clinical setting ä ä Challenges of getting the research results evaluated in a clinical context ä ä Challenges of clinical adoption of the research results ä ä Challenges of doing the research within a clinical setting ä ä Challenges of getting the research results evaluated in a clinical context ä ä Challenges of clinical adoption of the research results

System Challenges in the Research ä Acquisition of image data of adequate quality ä Meet HIPAA regulations: anonymization ä Homogeneous, high resolution data sets ä Full volume of interest ä As artifact free as possible, or with typical artifacts ä Training cases from target population (with cancer) and from patients with normal anatomy ä Conversion of images and segmentations images and segmentations in RT planning and delivery system into research system ä Acquisition of image data of adequate quality ä Meet HIPAA regulations: anonymization ä Homogeneous, high resolution data sets ä Full volume of interest ä As artifact free as possible, or with typical artifacts ä Training cases from target population (with cancer) and from patients with normal anatomy ä Conversion of images and segmentations images and segmentations in RT planning and delivery system into research system

ä ä High quality manual segmentations in RT planning and delivery system ä ä Consistency across training cases ä ä In adaptive RT: consistency with MD’s planning day ä ä Planning of RT: Include multi-expert variation ä ä Improved manual segmentation tools were developed ä ä High quality manual segmentations in RT planning and delivery system ä ä Consistency across training cases ä ä In adaptive RT: consistency with MD’s planning day ä ä Planning of RT: Include multi-expert variation ä ä Improved manual segmentation tools were developed System Challenges in the Research, cont.

ä ä Need research-tolerant and interested physicians on the team ä ä Need physician input all along, without too heavily disappointing them with early results ä ä New approach was expected to, and did, take a decade to develop ä ä Obtaining a large number of careful, manual segmentations ä ä Need research-tolerant and interested physicians on the team ä ä Need physician input all along, without too heavily disappointing them with early results ä ä New approach was expected to, and did, take a decade to develop ä ä Obtaining a large number of careful, manual segmentations

Evaluation experiments ä ä Retrospective ä ä Our data ä ä Other hospital’s data ä ä Prospective ä ä Within clinical practice, but not interfering with it ä ä “Jeopardy” that they might use the good results clinically during the test ä ä Complete their own segmentations first ä ä Retrospective ä ä Our data ä ä Other hospital’s data ä ä Prospective ä ä Within clinical practice, but not interfering with it ä ä “Jeopardy” that they might use the good results clinically during the test ä ä Complete their own segmentations first

System Challenges in the Evaluation ä ä Access to software that is used clinically ä ä Adding objects with geometry, not just image slice contours (in one orientation), to RT system and its philosophy ä ä Providing segmentations with clinically useful measures of tolerance ä ä Hiding the image analysis details from the clinical user, while allowing access to them by the image analysis researcher ä ä Software’s robustness, reproducibility, user independence, speed (also for clinical use) ä ä Access to software that is used clinically ä ä Adding objects with geometry, not just image slice contours (in one orientation), to RT system and its philosophy ä ä Providing segmentations with clinically useful measures of tolerance ä ä Hiding the image analysis details from the clinical user, while allowing access to them by the image analysis researcher ä ä Software’s robustness, reproducibility, user independence, speed (also for clinical use)

System Challenges in the Evaluation, cont. ä ä Need consensus performance standards ä ä No gold standard with real clinical material ä ä Comparisons of computer vs.human differences against inter-human or intra-human differences ä ä Means of generating synthetic but realistic cases with known truth ä ä Community-wide test case collections ä ä Need consensus performance standards ä ä No gold standard with real clinical material ä ä Comparisons of computer vs.human differences against inter-human or intra-human differences ä ä Means of generating synthetic but realistic cases with known truth ä ä Community-wide test case collections Case index distance Cf. human-to-human prostate agreement: 1.9mm average surface distance

System Challenges of the Clinical Context ä ä Adding indications of non-credibility to segmentations; they may believe too readily ä ä Adding editing capability to computer generated segmentations ä ä Software continuing to change, during clinical tests and clinical use ä ä Adding indications of non-credibility to segmentations; they may believe too readily ä ä Adding editing capability to computer generated segmentations ä ä Software continuing to change, during clinical tests and clinical use

ConclusionConclusion ä ä Research on IGT methods has not only technical and clinical challenges but also significant software system challenges