The hierarchy of multi-scale deep structure Some applications The relation between biological vision and computer vision The hierarchy of multi-scale deep structure Some applications Prof. Bart ter Haar Romeny

Edge focusing toppoints Multi-scale signature function of this image line graph theory  scale 2D deep hierarchical structure MR slice hartcoronair

Structures exist at their own scale: Original  = e0 px  = e1 px  = e2 px  = e3 px Noise edges

The graph of the sign-change of the first derivative of a signal as a function of scale is denoted the scale-space signature of the signal. Zero-crossings of the second order derivative = max of first order derivative, as a function of scale

The notion of longevity can be viewed of a measure of importance for singularities [Witkin83]. The semantical notions of prominence and conspicuity now get a clear meaning in scale-space theory. In a scale-space we see the emergence of the hierarchy of structures. Positive and negative edges come together and annihilate in singularity points.

Example: Lysosome segmentation in noisy 2-photon microscopy 3D images of macrophages.

From Wikipedia: A lysosome is a membrane-bound organelle found in many animal cells and most plant cells. They are spherical vesicles that contain hydrolytic enzymes that can break down many kinds of biomolecules. A lysosome has a specific composition, of both its membrane proteins, and its luminal proteins. The lumen's pH (4.5–5.0) is optimal for the enzymes involved in hydrolysis, analogous to the activity of the stomach. 

Marching-cubes isophote surface of the macrophage. Preprocessing: - Blur with  = 3 px - Detect N strongest maxima

We interpolate with cubic splines interpolation 35 radial tracks in 35 3D orientations

The profiles are extremely noisy: Observation: visually we can reasonably point the steepest edgepoints.

Edge focusing over all profiles. Choose a start level based on the task, i.e. find a single edge.

Detected 3D points per maximum. We need a 3D shape fit function.

The 3D points are least square fit with 3D spherical harmonics:

Resulting detection:

An efficient way to detect maxima and saddlepoints is found in the theory of vector field analysis (Stoke’s theorem)

Topological winding numbers  is the wedge product (outer product for functionals)

In 2D: the surrounding of the point P is a closed path around P. The winding number  of a point P is defined as the number of times the image gradient vector rotates over 2 when we walk over a closed path around P. maximum:  = 1 minumum:  = 1 regular point:  = 0 saddle point:  = -1 monkey saddle:  = -2

Winding number = +1  extremum Winding number = -1  saddle The notion of scale appears in the size of the path.

Catastrophe theory Winding number = - 4  monkey saddle Generalised saddle point (5th order): (x+i y)5 The winding numbers add within a closed contour, e.g. A saddle point (-1) and an extremum (+1) cancel, i.e. annihilate. Catastrophe theory

Extrema Saddles

Decrease of structure over scale scales with the dimensionality. The number of extrema and saddlepoints decrease as e-N over scale

Application: Computer-Assisted Human Follicle Analysis for Fertility Prospects with 3D Ultrasound Fertility Prospects In most developed countries a postponement of childbearing is seen. E.g. in the Netherlands: Average age of bearing first child is 30 years. ter Haar Romeny, B. M., Titulaer, B., Kalitzin, S., Scheffer, G., Broekmans, F., Staal, J., & te Velde, E. (1999, June). Computer assisted human follicle analysis for fertility prospects with 3D ultrasound. In Biennial International Conference on Information Processing in Medical Imaging (pp. 56-69). Springer, Berlin, Heidelberg.

The number of follicles decreases during lifetime

Approx. 15000 couples visit fertility clinics annually As female fecundicity decreases with advancing age, an increasing number of couples is faced with unexpected difficulties in conceiving. Approx. 15000 couples visit fertility clinics annually In 70% of these cases age-related fecundicity decline may play a role A further increase is expected Menopausal age

Resting 0.03 mm initiation of growth > 120 days? A follicle’s life Resting 0.03 mm initiation of growth > 120 days? Early growing 0.03 - 0.1 mm Preantral 0.1 - 0.2 mm basal growth ~ 65 days Antral 0.2 - 2 mm Selectable 2 - 5 mm rescued by FSH window ~ 5 days Selected 5 - 10 mm Dominance 10 - 20 mm maturation ~ 15 days Ovulation

3D Ultrasound is a safe, cheap and versatile appropriate modality Kretz Medicor 530D

Trans-vaginal probe Two 3D acquisition strategies: 1. Position tracker on regular probe 2. Sweep of 2D array in transducer Regular sampling from irregularly space slices Trans-vaginal probe

Manual counting is very cumbersome  Automated follicle assessment 2-5 mm hypodense structures structured noise vessels may look like follicles ovary boundary sometimes missing

i P Detection of a singularity (i.e. a minimum) From theory of vector fields several important theorems (Stokes, Gauss) exist that relate something happening in a volume with just its surface. We can detect singularities by measurements around the singularity. i  i P 1-D: difference of signs of the gradient  i  zero crossing or extremum The surrounding of the point P are just 2 points left and right of P  1D sphere.

In subscript notation: where ij is the antisymmetric tensor.  

Example of a result: 1 cm Dataset 2563, radius Stokes’ sphere 1 pixel, blurring scale 3 pixels

generation of 200 - 500 rays in a homogeneous orientation distribution Detection of follicle boundaries: generation of 200 - 500 rays in a homogeneous orientation distribution determine most pronounced edge along ray by winding number focusing fit spherical harmonics to get an analytical description of the shape calculate volume and statistics on shape Scale  Scale  US intensity Distance along ray Distance along ray Distance along ray

3D scatterplot of detected endpoints 3D visualisation of fitted spherical harmonics function

anatominical coupes high resolution MR 3D ultrasound Validation with 2 bovine ovaria anatominical coupes high resolution MR 3D ultrasound Follicle # x center y z distance to neighbor (pixels) volume from spherical harmonics (mm 3 ) volume from MRI from anatomy v00 99 51 35 25.4 259.7 250.0 262.1 93 28 44 45.0 28.4 27.0 29.8 113 55 74 41.6 56.2 54.9 59.3 v01 33 41 66 25.3 242.3 47 22 75 44.5 34.0 64 49 38.9 54.7 v44 72 84 239.7 69 70 45.2 32 82 40.1

Conclusions: 3D ultrasound is a feasible modality for follicle-based fertilitiy state estimation automated CAD is feasible, more clinical validation needed winding numbers are robust (scaled) singularity detectors a robust class of topological properties emerges

Multi-scale watershed segmentation Watershed are the boundaries of merging water basins, when the image landscape is immersed by punching the minima. At larger scale the boundaries get blurred, rounded and dislocated.

Regions of different scales can be linked by calculating the largest overlap with the region in the scales just above.

The method is often combined with nonlinear diffusion schemes E. Dam, ITU

Nabla Vision is an interactive 3D watershed segmentation tool developed by the University of Copenhagen. Sculpture the 3D shape by successively clicking precalculated finer scale watershed details.