Extracting Branching Object Geometry via Cores Doctoral Dissertation Defense Yoni Fridman August 17, 2004 Advisor: Stephen Pizer Doctoral Dissertation Defense Yoni Fridman August 17, 2004 Advisor: Stephen Pizer

OutlineOutline ä Motivation, Background, and Thesis Statement ä Cores in 3D ä Handling Branching Objects with Cores ä Ends of Cores ä Evaluation and Results ä Scientific Contributions and Future Work ä Motivation, Background, and Thesis Statement ä Cores in 3D ä Handling Branching Objects with Cores ä Ends of Cores ä Evaluation and Results ä Scientific Contributions and Future Work

Motivation: Endovascular Embolization ä Driving problem: Endovascular embolization of a cerebral aneurysm Endovascular embolization Courtesy Toronto Brain Vascular Malformation Study Group Aneurysm in a DSA projection image

Motivation: Endovascular Embolization ä Difficulty: How to guide catheter to aneurysm ä In 2D, projection overlap makes geometry of vasculature ambiguous ä In 3D, information lost when viewing 1 slice at a time ä Difficulty: How to guide catheter to aneurysm ä In 2D, projection overlap makes geometry of vasculature ambiguous ä In 3D, information lost when viewing 1 slice at a time Axial slice of head MRA data Aneurysm in a DSA projection image

GoalGoal ä Automatically extract representations of anatomic objects from medical images 3D vessel tree representation Axial projection image of head MRA data

Motivation: Radiation Treatment Planning ä Driving problem: 3D radiation treatment planning Tumor in axial slice of abdominal CT data Courtesy Shands Health Care

GoalGoal ä Automatically extract representations of anatomic objects from medical images 3D kidney representation Tumor in axial slice of abdominal CT data Courtesy Shands Health Care

Blum’s medial axis ä The medial axis is a formulation that describes objects by focusing on their middles ä Can be thought of as a skeleton or backbone ä Pioneered by Blum (Blum 1967, Blum & Nagle 1978) for biological structures ä The medial axis is a formulation that describes objects by focusing on their middles ä Can be thought of as a skeleton or backbone ä Pioneered by Blum (Blum 1967, Blum & Nagle 1978) for biological structures An object and its medial axis

Overview of Cores (in 2D) ä A core is a medial axis of an object at scale (i.e., in a blurred image) ä Why at scale? ä To reduce image noise ä So small indentations and protrusions on the object boundary are not reflected in the core ä A core is a medial axis of an object at scale (i.e., in a blurred image) ä Why at scale? ä To reduce image noise ä So small indentations and protrusions on the object boundary are not reflected in the core A synthetic object and its core

Overview of Cores (in 2D) ä A core is a medial axis of an object at scale (i.e., in a blurred image) ä Each location on the core stores orientation and radius information ä A core is a medial axis of an object at scale (i.e., in a blurred image) ä Each location on the core stores orientation and radius information A synthetic object and its core

Cores as Object Representations ä An object’s core provides a discrete representation, at scale, of the object ä This can be seen by taking the union of disks centered along the core, with the given radii ä This representation is computed automatically ä An object’s core provides a discrete representation, at scale, of the object ä This can be seen by taking the union of disks centered along the core, with the given radii ä This representation is computed automatically Recreating an object from its core

Types of Cores ä Two mathematically distinct types of cores have been studied: 1. Maximum convexity cores (Morse 1994, Eberly 1996, Damon 1998, Miller 1998, Damon 1999, Keller 1999) 2. Optimum parameter cores (Furst 1999, Aylward & Bullitt 2002) ä This dissertation deals with optimum parameter cores; details later ä Two mathematically distinct types of cores have been studied: 1. Maximum convexity cores (Morse 1994, Eberly 1996, Damon 1998, Miller 1998, Damon 1999, Keller 1999) 2. Optimum parameter cores (Furst 1999, Aylward & Bullitt 2002) ä This dissertation deals with optimum parameter cores; details later

Thesis Statement Optimum parameter cores with branch- handling and end-detection provide an effective means for extracting the branching geometry of tubular structures from 3D medical images and for extracting the branching geometry of general structures from relatively low noise 3D medical images.

OutlineOutline ä Motivation, Background, and Thesis Statement ä Cores in 3D ä Handling Branching Objects with Cores ä Ends of Cores ä Evaluation and Results ä Scientific Contributions and Future Work ä Motivation, Background, and Thesis Statement ä Cores in 3D ä Handling Branching Objects with Cores ä Ends of Cores ä Evaluation and Results ä Scientific Contributions and Future Work

Core Computation ä Optimize the derivative of Gaussians’ fit to the image by varying location, radius, and orientation ä Take a step forward and iterate ä Optimize the derivative of Gaussians’ fit to the image by varying location, radius, and orientation ä Take a step forward and iterate ä Initialize a medial atom and place a derivative of a Gaussian at the tips of two spokes Computing the core of a synthetic object

ä Cores of 3D objects are generically 2D (sheets). Objects represented by 2D cores are called “slabs” ä Special case: Cores of tubes are 1D (curves) ä Cores of 3D objects are generically 2D (sheets). Objects represented by 2D cores are called “slabs” ä Special case: Cores of tubes are 1D (curves) Cores in 3D – Slabs and Tubes

What is a Medial Atom?  A medial atom m = (x, r, F,  ) is an oriented position with two spokes. In 3D ä (Morse 1994, Fritsch et al. 1995, Pizer et al. 1998, Furst 1999, Pizer et al. 2003)  A medial atom m = (x, r, F,  ) is an oriented position with two spokes. In 3D ä (Morse 1994, Fritsch et al. 1995, Pizer et al. 1998, Furst 1999, Pizer et al. 2003) ä x is its location in 3-space ä r is its radius, or the length of two spokes, p and s ä F is a frame that defines its orientation ä b is the bisector of the spokes   is its object angle x r  p s b Medial atom geometry

My Medial Atoms  I constrain  to  /2, so m = (x, r, F) ä This improves the resistance of core computation  I constrain  to  /2, so m = (x, r, F) ä This improves the resistance of core computation to image noise ä It is also less natural and affects core computation in other ways ä I quantify these effects in the dissertation and show that the constraint is beneficial overall x r  p s b My medial atom geometry

What is the Medialness of a Medial Atom m? ä Medialness M(m) is a scalar function that measures the fit of a medial atom to image data ä A kernel K(m) is created from m by placing two directional derivatives of volumetric Gaussians, one at each spoke tip, with the derivatives taken in the spoke directions ä Medialness M(m) is a scalar function that measures the fit of a medial atom to image data ä A kernel K(m) is created from m by placing two directional derivatives of volumetric Gaussians, one at each spoke tip, with the derivatives taken in the spoke directions x p s ä M(m) is then computed by integrating image intensities as weighted by K(m)

Maximizing Medialness ä Given an approximate atom m = (x, r, F), find x, r, and F that maximize M(m)  Let x = (s, t, u) and F = (az, alt,  ) ä u is in the spoke direction and (s, t) span the normal plane  az and alt are the azimuth and altitude of b,  is spin about b ä Maximize with respect to position and then parameters: ä This defines the optimum parameter cores I use ä Given an approximate atom m = (x, r, F), find x, r, and F that maximize M(m)  Let x = (s, t, u) and F = (az, alt,  ) ä u is in the spoke direction and (s, t) span the normal plane  az and alt are the azimuth and altitude of b,  is spin about b ä Maximize with respect to position and then parameters: ä This defines the optimum parameter cores I use

Core-FollowingCore-Following ä Now we need to follow a 2D sheet – can’t simply step forward ä Rather, march along a grid ä Now we need to follow a 2D sheet – can’t simply step forward ä Rather, march along a grid Following the 2D core of a slab in 3D 2D core of a kidney

Core-FollowingCore-Following ä I add two features to core-following that improve its resistance to noise: 1. When optimizing medialness, I penalize significant changes in radius and/or orientation between neighboring atoms 2. I compute the core at a coarse sampling (taking large steps between atoms) and then refine the sampling ä I add two features to core-following that improve its resistance to noise: 1. When optimizing medialness, I penalize significant changes in radius and/or orientation between neighboring atoms 2. I compute the core at a coarse sampling (taking large steps between atoms) and then refine the sampling Refining a core

Cores of Tubes ä Cores of tubes are computed using medial atoms with a set of (8) concentric spokes ä The resulting core is a curve ä Problem: Euclidean optimization must cover 2 dimensions, not 1 ä Cores of tubes are computed using medial atoms with a set of (8) concentric spokes ä The resulting core is a curve ä Problem: Euclidean optimization must cover 2 dimensions, not 1 Computing the core of a tube ä Solution: Pick any 2 directions that span normal plane to core, e.g., 2 orthogonal spokes ä These cores are more noise- resistant than cores of slabs ä Solution: Pick any 2 directions that span normal plane to core, e.g., 2 orthogonal spokes ä These cores are more noise- resistant than cores of slabs

OutlineOutline ä Motivation, Background, and Thesis Statement ä Cores in 3D ä Handling Branching Objects with Cores ä Ends of Cores ä Evaluation and Results ä Scientific Contributions and Future Work ä Motivation, Background, and Thesis Statement ä Cores in 3D ä Handling Branching Objects with Cores ä Ends of Cores ä Evaluation and Results ä Scientific Contributions and Future Work

Problem: Cores Don’t Branch ä Problem: Cores don’t branch, so what happens when a core reaches an object bifurcation? Cores of separate branches (in cross-section).  Core-following (in cross-section)

Solution: Jump to New Cores ä Solution: Stop following the core when it reaches an object bifurcation and jump to the new core(s) ä One for a slab, two for a tube ä Solution: Stop following the core when it reaches an object bifurcation and jump to the new core(s) ä One for a slab, two for a tube Cores of separate branches (in cross-section).  Jumping to cores of new branches

Corner Detection ä Apply an affine-invariant corner detector to the image: I uu I v 2 ä v is the gradient direction and u is the first eigenvector of the Hessian in the normal plane to v ä Apply an affine-invariant corner detector to the image: I uu I v 2 ä v is the gradient direction and u is the first eigenvector of the Hessian in the normal plane to v

Corner Detection ä Apply an affine-invariant corner detector to the image: I uu I v 2 ä v is the gradient direction and u is the first eigenvector of the Hessian in the normal plane to v ä Apply an affine-invariant corner detector to the image: I uu I v 2 ä v is the gradient direction and u is the first eigenvector of the Hessian in the normal plane to v A projection imageApplication of the corner detector Maxima of cornerness

Corner Detection ä A spoke tip located at a maximum of cornerness indicates a potential object branch Core-following A projection imageApplication of the corner detector Maxima of cornerness

Rejection of False Positive Branches ä Maxima of cornerness can also be false positives ä For example, bends in the object ä Maxima of cornerness can also be false positives ä For example, bends in the object Maxima of cornerness; red indicates false positive

Rejection of False Positive Branches ä I reject false positives using combinations of heuristics, e.g., non-increasing radius along the core Maxima of cornerness; red indicates false positive True branchFalse positive branch

Re-seeding Cores Along New Branches ä Predict the locations and parameters of the new branches using geometric information from the computed core ä Prediction need not be accurate ä Predict the locations and parameters of the new branches using geometric information from the computed core ä Prediction need not be accurate Re-seeding cores of new branches

OutlineOutline ä Motivation, Background, and Thesis Statement ä Cores in 3D ä Handling Branching Objects with Cores ä Ends of Cores ä Evaluation and Results ä Scientific Contributions and Future Work ä Motivation, Background, and Thesis Statement ä Cores in 3D ä Handling Branching Objects with Cores ä Ends of Cores ä Evaluation and Results ä Scientific Contributions and Future Work

Types of Object Ends ä I consider two types of object ends: 1. Explicit ends, where an object is capped 2. Implicit ends, where an object narrows until it is indiscernible in the image ä I consider two types of object ends: 1. Explicit ends, where an object is capped 2. Implicit ends, where an object narrows until it is indiscernible in the image DNA with explicit ends Blood vessel with an implicit end

Core Termination ä Case 1: Stop following a core at an object end ä Case 2: Stop following a core when there is not enough evidence that you’re on an object ä Case 1: Stop following a core at an object end ä Case 2: Stop following a core when there is not enough evidence that you’re on an object DNA with explicit ends Blood vessel with an implicit end

Core Termination ä Use local statistics on medialness values to determine when we’ve lost track of the object ä At each step of core-following, create a set S of randomly positioned and oriented medial atoms (yellow) in a region surrounding the current atom, m 0 (blue) ä Compute the medialness value of each atom in S ä m 0 is valid iff ä Use local statistics on medialness values to determine when we’ve lost track of the object ä At each step of core-following, create a set S of randomly positioned and oriented medial atoms (yellow) in a region surrounding the current atom, m 0 (blue) ä Compute the medialness value of each atom in S ä m 0 is valid iff

OutlineOutline ä Motivation, Background, and Thesis Statement ä Cores in 3D ä Handling Branching Objects with Cores ä Ends of Cores ä Evaluation and Results ä Scientific Contributions and Future Work ä Motivation, Background, and Thesis Statement ä Cores in 3D ä Handling Branching Objects with Cores ä Ends of Cores ä Evaluation and Results ä Scientific Contributions and Future Work

Demonstration: Core of a Kidney Core of a kidney Surface implied by the core

Demonstration: Core of a Kidney and Renal Artery Cores of a kidney and adjoining renal artery Surfaces implied by the cores

Demonstration: Cores of Blood Vessels Axial, sagittal, and coronal MIP views of 3D head MRA data Surfaces implied by cores of blood vessels

Analysis of the Effects of Object Geometry on Cores ä Using synthetic images (with known truth), I analyze the effects of object geometry on cores ä For tubes: Width, bending, branching angle, branch size, rate of tapering at ends ä For slabs: Rate of narrowing, bending, branching angle ä This analysis provides: ä Validation of my methods ä The ability to predict performance on new objects and images ä A better understanding of the behavior of cores ä Using synthetic images (with known truth), I analyze the effects of object geometry on cores ä For tubes: Width, bending, branching angle, branch size, rate of tapering at ends ä For slabs: Rate of narrowing, bending, branching angle ä This analysis provides: ä Validation of my methods ä The ability to predict performance on new objects and images ä A better understanding of the behavior of cores

Tube Width ä Results: ä Success of tubular core-following is related to object width and inversely related to image noise ä In MR, tubular core-following is reliable down to tubes of diameter 1 voxel ä Results: ä Success of tubular core-following is related to object width and inversely related to image noise ä In MR, tubular core-following is reliable down to tubes of diameter 1 voxel

Tube Bending ä Results: ä Curvature and torsion are not a limiting factor in tubular core-following ä Cores of sharply curved tubes can be followed by decreasing step size ä Results: ä Curvature and torsion are not a limiting factor in tubular core-following ä Cores of sharply curved tubes can be followed by decreasing step size

Tube Branching Angle ä Results: ä Tubular branch-handling is reliable in low-noise images regardless of branching angle ä In ultrasound, failure rates were ~20% but all failures were false negatives; there were no false positives ä Results: ä Tubular branch-handling is reliable in low-noise images regardless of branching angle ä In ultrasound, failure rates were ~20% but all failures were false negatives; there were no false positives

Tube Branch Size ä Results: ä Tubular branch-handling deteriorates when one branch is significantly narrower than the other, particularly in noisier images ä Results: ä Tubular branch-handling deteriorates when one branch is significantly narrower than the other, particularly in noisier images

Tube Rate of End Tapering ä Results: ä Core-following does not continue past tube ends ä Core-following stops near tube ends in low-noise images, stops earlier with increased noise ä Results: ä Core-following does not continue past tube ends ä Core-following stops near tube ends in low-noise images, stops earlier with increased noise Core-termination errors

Slab Rate of Narrowing ä Results: ä Success of slab-like core-following is inversely related to both narrowing angle and image noise ä Medialness is proportional to the cosine of the narrowing angle ä Results are the same whether the object narrows or widens in the direction of core-following ä Results: ä Success of slab-like core-following is inversely related to both narrowing angle and image noise ä Medialness is proportional to the cosine of the narrowing angle ä Results are the same whether the object narrows or widens in the direction of core-following

Slab Bending ä Results: ä Sharply bent slabs can be followed better by decreasing step size (like tubes) ä Failures occur occasionally (4% failure rate) due to intra-object interference (unlike tubes) ä Results: ä Sharply bent slabs can be followed better by decreasing step size (like tubes) ä Failures occur occasionally (4% failure rate) due to intra-object interference (unlike tubes)

Slab Branching Angle ä Results: ä Success of slab-like branch-handling is related to branching angle and inversely related to image noise ä All failures were false negatives (like tubes) ä Results: ä Success of slab-like branch-handling is related to branching angle and inversely related to image noise ä All failures were false negatives (like tubes)

OutlineOutline ä Motivation, Background, and Thesis Statement ä Cores in 3D ä Handling Branching Objects with Cores ä Ends of Cores ä Evaluation and Results ä Scientific Contributions and Future Work ä Motivation, Background, and Thesis Statement ä Cores in 3D ä Handling Branching Objects with Cores ä Ends of Cores ä Evaluation and Results ä Scientific Contributions and Future Work

Scientific Contributions ä A robust implementation of 1D cores in 3D ä A system for handling branches using 1D cores in 3D ä A method for detecting ends of cores ä An implementation of 2D cores with branch- handling in 3D ä An analysis of the effects of object geometry on the performance of cores ä A robust implementation of 1D cores in 3D ä A system for handling branches using 1D cores in 3D ä A method for detecting ends of cores ä An implementation of 2D cores with branch- handling in 3D ä An analysis of the effects of object geometry on the performance of cores

Future Work ä Handle loops, not just tree topology, with tubular cores ä Handle slab-like children branching from tubular parents ä Improve the resistance of the slab-like method to noise ä Integrate cores with a model-based method like m-reps to use prior information but still find unmodeled branches ä Improve running time ä Handle loops, not just tree topology, with tubular cores ä Handle slab-like children branching from tubular parents ä Improve the resistance of the slab-like method to noise ä Integrate cores with a model-based method like m-reps to use prior information but still find unmodeled branches ä Improve running time

AcknowledgementsAcknowledgements ä Advisor: Steve Pizer ä Committee: Stephen Aylward, Jim Damon, Guido Gerig, Jack Snoeyink ä MIDAG, Delphi Bull, and others in the department ä Funding: Office of Naval Research, National Institutes of Health ä Russell Taylor and NSRG, Bob Goldstein and his biology lab ä My wife, Adelia ä Advisor: Steve Pizer ä Committee: Stephen Aylward, Jim Damon, Guido Gerig, Jack Snoeyink ä MIDAG, Delphi Bull, and others in the department ä Funding: Office of Naval Research, National Institutes of Health ä Russell Taylor and NSRG, Bob Goldstein and his biology lab ä My wife, Adelia

Extracting Branching Object Geometry via Cores Doctoral Dissertation Defense Yoni Fridman August 17, 2004 Advisor: Stephen Pizer Doctoral Dissertation Defense Yoni Fridman August 17, 2004 Advisor: Stephen Pizer

Effects of   How does  affect core computation and medial atom geometry? ä Cores may end sooner at branches ä Cores shift slightly ä Response is weaker because spokes aren’t orthogonal to object boundaries  How does  affect core computation and medial atom geometry? ä Cores may end sooner at branches ä Cores shift slightly ä Response is weaker because spokes aren’t orthogonal to object boundaries

Why 3D Cores Are Better Than 2D Cores ä Using a larger number of Gaussian spokes integrates over a larger area, reducing the effects of noise A slice of a synthetic 3D image of a bent tube, with the core computed successfully despite extreme noise ä Constraining the spokes to circular cross-sections further reduces the effects of noise Computing the core of a tube

Penalties on Medialness ä How exactly do I penalize medialness in slabs? ä Separately penalize changes in radius, orientation ä No penalty for a small change ä Beyond that, penalize linearly with changes ä How exactly do I penalize medialness in slabs? ä Separately penalize changes in radius, orientation ä No penalty for a small change ä Beyond that, penalize linearly with changes

Re-seeding Cores

Tubular Core-Following