Medial Object Shape Representations for Image Analysis & Object Synthesis Stephen M. Pizer Kenan Professor Medical Image Display & Analysis Group University of North Carolina, USA Credits: Many on MIDAG, especially Daniel Fritsch, Andrew Thall, George Stetten, Paul Yushkevich Stephen M. Pizer Kenan Professor Medical Image Display & Analysis Group University of North Carolina, USA Credits: Many on MIDAG, especially Daniel Fritsch, Andrew Thall, George Stetten, Paul Yushkevich

Medial Object Shape Representations for Image Analysis & Object Synthesis

What shape representation is for ä Analysis from images ä Extract the kidney-shaped object ä Register based on the pelvic bone shapes ä Extract shape information w/o model ä Synthesis ä Design the object ä Deform the object, with physical realism  Shape science  Shape and biology  Shape-based diagnosis ä Analysis from images ä Extract the kidney-shaped object ä Register based on the pelvic bone shapes ä Extract shape information w/o model ä Synthesis ä Design the object ä Deform the object, with physical realism  Shape science  Shape and biology  Shape-based diagnosis

What shape representation is for ä Analysis from images ä Extract the kidney-shaped object ä Register based on the pelvic bone shapes ä Analysis from images ä Extract the kidney-shaped object ä Register based on the pelvic bone shapes

What shape representation is for ä Synthesis ä Design the object ä Deform the object, with physical realism ä Synthesis ä Design the object ä Deform the object, with physical realism

What shape representation is for  Shape science  Shape and biology  Shape-based diagnosis  Shape science  Shape and biology  Shape-based diagnosis Brain structures (Gerig)

Shape Sciences ä Geometry ä The spatial layout: via primitives ä Landmarks ä Boundary places and orientations ä Medial places, figural sizes and orientations ä Space itself ä Statistics ä The average shape ä Modes of variation from the average ä Computer Graphics ä Image Analysis ä Geometry ä The spatial layout: via primitives ä Landmarks ä Boundary places and orientations ä Medial places, figural sizes and orientations ä Space itself ä Statistics ä The average shape ä Modes of variation from the average ä Computer Graphics ä Image Analysis

Options for Primitives ä Space: x i for grid elements ä Landmarks: x i described by local geometry ä Boundary: (x i,normal i ) spaced along boundary ä Figural: nets of diatoms sampling figures ä Space: x i for grid elements ä Landmarks: x i described by local geometry ä Boundary: (x i,normal i ) spaced along boundary ä Figural: nets of diatoms sampling figures

Primitives for shape representation: Landmarks  Sets of points of special geometry

Primitives for shape representation: Boundaries  Boundary points with normals

Object Representation by M-Reps

Each M-figure Represented by Net of Medial Primitives

Each M-figure Represented by Net of Medial Primitives

Figural Models ä Figures: successive medial involution ä Main figure ä Protrusions ä Indentations ä Separate figures ä Hierarchy of figures ä Relative position ä Relative width ä Relative orientation ä Figures: successive medial involution ä Main figure ä Protrusions ä Indentations ä Separate figures ä Hierarchy of figures ä Relative position ä Relative width ä Relative orientation

Primitives’ Desired Properties ä Geometry ä Intuitive: simple, global + local ä Efficiently deformable ä Easily extracted or created ä Spatial tolerance inherent ä Statistics ä Unimodality: normally distributed ä Via geometrical, tolerance-sensitive metric ä Geometry ä Intuitive: simple, global + local ä Efficiently deformable ä Easily extracted or created ä Spatial tolerance inherent ä Statistics ä Unimodality: normally distributed ä Via geometrical, tolerance-sensitive metric

Figural Models with Boundary Deviations ä Hypothesis ä At a global level, a figural model is the most intuitive ä At a local level, boundary deviations are most intuitive ä Hypothesis ä At a global level, a figural model is the most intuitive ä At a local level, boundary deviations are most intuitive

Union and Difference of M-figures

Medial Primitives  x, (b,n) frame, r,  (object angle) ä Imply boundary segments with tolerance ä Similarity transform equivariant ä Zoom invariance implies width-proportionality of ä tolerance of implied boundary ä boundary curvature distribution ä spacing along net ä interrogation aperture for image  x, (b,n) frame, r,  (object angle) ä Imply boundary segments with tolerance ä Similarity transform equivariant ä Zoom invariance implies width-proportionality of ä tolerance of implied boundary ä boundary curvature distribution ä spacing along net ä interrogation aperture for image n

3D kidney model extracted from CT Four figure model of the kidneys Red represents indentation figures Four figure model of the kidneys Red represents indentation figures

Need for Special End Primitives ä Represent ä non-blobby objects ä angulated edges, corners, creases ä still allow rounded edges, corners, creases ä allow bent edges ä But ä Avoid infinitely fine medial sampling ä Maintain tangency, symmetry principles ä Represent ä non-blobby objects ä angulated edges, corners, creases ä still allow rounded edges, corners, creases ä allow bent edges ä But ä Avoid infinitely fine medial sampling ä Maintain tangency, symmetry principles

End Primitives Rounded end primitive in cross-section Corner primitive in cross-section

Displacements from Figurally Implied Boundary Boundary implied by figural modelBoundary after displacements

Coarse-to-fine representation   For each of three levels   Figural hierarchy   For each figure, net chain, successively smaller tolerance   For each net tile, boundary displacement chain   For each of three levels   Figural hierarchy   For each figure, net chain, successively smaller tolerance   For each net tile, boundary displacement chain

Multiscale Medial Model   From larger scale medial net   Coarsely sampled   Smooother figurally implied boundary   Larger tolerance   Interpolate smaller scale medial net   Finer sampled   More detail in figurally implied boundary   Smaller tolerance   Represent medial displacements   From larger scale medial net   Coarsely sampled   Smooother figurally implied boundary   Larger tolerance   Interpolate smaller scale medial net   Finer sampled   More detail in figurally implied boundary   Smaller tolerance   Represent medial displacements

Multiscale Medial Model   From larger scale medial net, interpolate smaller scale medial net and represent medial displacements   From larger scale medial net, interpolate smaller scale medial net and represent medial displacements b.

Multiscale Medial/Boundary Model   From medial net   Coarsely sampled, smoother implied boundary   Larger tolerance   Represent boundary displacements along implied normals   Finer sampled, more detail in boundary   Smaller tolerance   From medial net   Coarsely sampled, smoother implied boundary   Larger tolerance   Represent boundary displacements along implied normals   Finer sampled, more detail in boundary   Smaller tolerance

Shape Rep’n in Image Analysis ä Segmentation ä Extract an object from image ä Registration ä Find geometric transformation that brings two images into alignment ä 3D/3D ä 3D/2D ä Shape Measurement ä Find how probable a shape is ä Segmentation ä Extract an object from image ä Registration ä Find geometric transformation that brings two images into alignment ä 3D/3D ä 3D/2D ä Shape Measurement ä Find how probable a shape is

Shape Repres’n in Image Analysis ä Segmentation ä Find the most probable deformed mean model, given the image ä Probability involves ä Probability of the deformed model (prior) ä Probability of the image, given the deformed model (likelihood) ä Segmentation ä Find the most probable deformed mean model, given the image ä Probability involves ä Probability of the deformed model (prior) ä Probability of the image, given the deformed model (likelihood)

Probability of a deformed model ä From training set ä via principal components analysis, coarse-to-fine ä -C * Geometric difference from typical shape ä From training set ä via principal components analysis, coarse-to-fine ä -C * Geometric difference from typical shape

Medialness: medial strength of a medial primitive in an image ä Probability of image | deformed model ä Sum of boundariness values ä at implied boundary positions ä in implied normal directions ä with apertures proportional to tolerance ä Boundariness value ä Intensity profile distance from mean (at scale) ä statistical, based on training set ä Intensity differences ä via Gaussian derivatives ä Probability of image | deformed model ä Sum of boundariness values ä at implied boundary positions ä in implied normal directions ä with apertures proportional to tolerance ä Boundariness value ä Intensity profile distance from mean (at scale) ä statistical, based on training set ä Intensity differences ä via Gaussian derivatives

Figurally implied boundaries and rendering, via 4-figure model

3D DSL Model Deformation Initial Position of Model in Target Image

3D DSL Model Deformation Figural Deformation Iteration 3

3D DSL Model Deformation with interfigural penalties Initial position After optimization

Shape Repres’n in Image Analysis ä Registration ä Find the most probable deformation, given the image ä Registration ä Find the most probable deformation, given the image

Shape Rep’n in Image Analysis ä Prior-free medial shape analysis ä Cores: height ridges of medialness (Pizer, Fritsch, Morse, Furst) ä Statistical analysis of medial diatoms (Stetten) ä Prior-free medial shape analysis ä Cores: height ridges of medialness (Pizer, Fritsch, Morse, Furst) ä Statistical analysis of medial diatoms (Stetten)

Shape Rep’n in Image Analysis ä Cores: height ridges of medialness

M I U N C

Shape Rep’n in Image Analysis ä Statistical analysis of medial diatoms

sphere slabcylinder

sphere slabcylinder

sphere slabcylinder

sphere slabcylinder

sphere slabcylinder

Shape Rep’n in CAD/CAM Shape Rep’n in CAD/CAM ä Stock figural models ä Deformation tools: large scale ä Coarse-to-fine specification ä Figural connection tools ä Direct rendering, according to display needs ä Stock figural models ä Deformation tools: large scale ä Coarse-to-fine specification ä Figural connection tools ä Direct rendering, according to display needs

Deformation in CAD/CAM Deformation in CAD/CAM

Shape Rep’n in CAD/CAM Shape Rep’n in CAD/CAM ä Design models for image analysis

Medial Object Shape Representations for Image Analysis & Object Synthesis ä Figural models, at successive levels of tolerance ä Boundary displacements ä Work in progress ä Segmentation and registration tools ä Statistical analysis of object populations ä CAD tools, incl. direct rendering ä Connection relative critical manifolds ä … ä Figural models, at successive levels of tolerance ä Boundary displacements ä Work in progress ä Segmentation and registration tools ä Statistical analysis of object populations ä CAD tools, incl. direct rendering ä Connection relative critical manifolds ä …

Application: Image guided planning & delivery of radiotherapy ä Planning in 3D ä Extracting normal anatomy ä Extracting tumor ä Planning beam poses ä Patient placement ä Verification of plan via portal image ä Planning in 3D ä Extracting normal anatomy ä Extracting tumor ä Planning beam poses ä Patient placement ä Verification of plan via portal image

Finding Treatment Pose from Portal Radiograph and Planning DRR

Medial Net Shape Models Medial nets, positions onlyMedial net

Integrated Medialness vs. Pose Offset

Representing Boundary Displacements ä Along figurally implied boundary normals ä Coarse-to-fine ä Captures along-boundary covariance ä Useful for rendering ä Along figurally implied boundary normals ä Coarse-to-fine ä Captures along-boundary covariance ä Useful for rendering

Summing Medialness on Medial Net via Medial Weighting Function

CT Slice of Kidneys in Abdomen

Object Shape Brain structures (Gerig)

Geometric aspects : Transformations  Euclidean:  translation and rotation  Similarity:  translation, rotation, zoom  Affine  Euclidean:  translation and rotation  Similarity:  translation, rotation, zoom  Affine