1. The hierarchy of multi-scale deep structure Some applications
The relation between biological vision and computer vision
The hierarchy of multi-scale deep structure Some applications
Prof. Bart ter Haar Romeny

2. Edge focusing toppoints Multi-scale signature function of this image
line
graph theory 
scale
2D deep
hierarchical
structure
MR slice hartcoronair

3. Structures exist at their own scale:
Original  = e0 px  = e1 px  = e2 px  = e3 px
Noise edges

4. The graph of the sign-change of the
first derivative of a signal as a function of scale is denoted the scale-space signature of the signal.
Zero-crossings of the second order derivative = max of first order derivative, as a function of scale

5. The notion of longevity can be viewed of a measure of importance for singularities [Witkin83].
The semantical notions of prominence and conspicuity now get a clear meaning in scale-space theory.
In a scale-space we see the emergence of the hierarchy of structures. Positive and negative edges come together and annihilate in singularity points.

6. Example:
Lysosome
segmentation
in noisy 2-photon microscopy
3D images of
macrophages.

7. From Wikipedia:
A lysosome is a membrane-bound organelle found in many animal cells and most plant cells.
They are spherical vesicles that contain hydrolytic enzymes that can break down many kinds of biomolecules.
A lysosome has a specific composition, of both its membrane proteins, and its luminal proteins.
The lumen's pH (4.5–5.0) is optimal for the enzymes involved in hydrolysis, analogous to the activity of the stomach. 

8. Marching-cubes isophote surface of
the macrophage.
Preprocessing:
- Blur with  = 3 px
- Detect N strongest maxima

9. We interpolate with cubic splines interpolation
35 radial tracks in 35 3D
orientations

10. The profiles are extremely noisy:
Observation: visually we can reasonably point the steepest edgepoints.

11. Edge focusing
over all profiles.
Choose a start level based on the task, i.e. find a single edge.

12. Detected 3D points per maximum.
We need a 3D shape fit function.

13. The 3D points are least square fit with 3D spherical harmonics:

14. Resulting detection:

15. An efficient way to detect maxima and saddlepoints is found in the theory of vector field analysis (Stoke’s theorem)

16. Topological winding numbers
 is the wedge product (outer product for functionals)

17. In 2D: the surrounding of the point P is a closed path around P.
The winding number  of a point P is defined as the number of times the image
gradient vector rotates over 2 when we walk over a closed path around P.
maximum:  = 1
minumum:  = 1
regular point:  = 0
saddle point:  = -1
monkey saddle:  = -2

18. Winding number = +1
 extremum
Winding number = -1
 saddle
The notion of scale appears in the size of the path.

19. Catastrophe theory Winding number = - 4  monkey saddle
Generalised saddle point (5th order): (x+i y)5
The winding numbers add within a closed contour, e.g.
A saddle point (-1) and an extremum (+1) cancel, i.e. annihilate.
Catastrophe theory

20. Extrema
Saddles

21. Decrease of structure over scale scales with the dimensionality.
The number of extrema and saddlepoints decrease as e-N over scale

22. Application:
Computer-Assisted Human Follicle Analysis for Fertility Prospects with 3D Ultrasound
Fertility Prospects
In most developed countries a postponement of childbearing is seen.
E.g. in the Netherlands: Average age of bearing first child is 30 years.
ter Haar Romeny, B. M., Titulaer, B., Kalitzin, S., Scheffer, G., Broekmans, F., Staal, J., & te Velde, E. (1999, June). Computer assisted human follicle analysis for fertility prospects with 3D ultrasound. In Biennial International Conference on Information Processing in Medical Imaging (pp ). Springer, Berlin, Heidelberg.

23. The number of follicles decreases during lifetime

24. Approx. 15000 couples visit fertility clinics annually
As female fecundicity decreases with advancing age, an increasing number of couples is faced with unexpected difficulties in conceiving.
Approx couples visit fertility clinics annually
In 70% of these cases age-related fecundicity decline
may play a role
A further increase is expected
Menopausal age

25. Resting 0.03 mm initiation of growth > 120 days?
A follicle’s life
Resting mm initiation of growth
> 120 days?
Early growing mm
Preantral mm basal growth
~ 65 days
Antral mm
Selectable mm rescued by FSH window
~ 5 days
Selected mm
Dominance mm maturation
~ 15 days
Ovulation

26. 3D Ultrasound is a safe, cheap and versatile appropriate modality
Kretz Medicor 530D

27. Trans-vaginal probe Two 3D acquisition strategies:
1. Position tracker on regular probe
2. Sweep of 2D array in transducer
Regular sampling from irregularly space slices
Trans-vaginal probe

29. Manual counting is very cumbersome
 Automated follicle assessment
2-5 mm hypodense structures
structured noise
vessels may look like follicles
ovary boundary sometimes missing

30. i P Detection of a singularity (i.e. a minimum)
From theory of vector fields several important theorems (Stokes, Gauss) exist that relate something happening in a volume with just its surface.
We can detect singularities by measurements around the singularity. i
 i P
1-D: difference of signs of the gradient  i  zero crossing or extremum
The surrounding of the point P are just 2 points
left and right of P  1D sphere.

31. In subscript notation:
where ij is the antisymmetric tensor.

32. Example of a result:
1 cm
Dataset 2563, radius Stokes’ sphere 1 pixel, blurring scale 3 pixels

33. generation of 200 - 500 rays in a homogeneous orientation distribution
Detection of follicle boundaries:
generation of rays in a homogeneous orientation distribution
determine most pronounced edge along ray by winding number focusing
fit spherical harmonics to get an analytical description of the shape
calculate volume and statistics on shape
Scale 
Scale 
US intensity
Distance along ray
Distance along ray
Distance along ray

34. 3D scatterplot of
detected endpoints
3D visualisation of fitted
spherical harmonics function

35. anatominical coupes high resolution MR 3D ultrasound
Validation with 2 bovine ovaria
anatominical coupes
high resolution MR
3D ultrasound
Follicle # x
center y z
distance to
neighbor
(pixels)
volume from
spherical harmonics
(mm 3 )
volume
from MRI
from
anatomy
v00 99 51 35
25.4
259.7
250.0
262.1 93 28 44
45.0
28.4
27.0
29.8
113 55 74
41.6
56.2
54.9
59.3
v01 33 41 66
25.3
242.3 47 22 75
44.5
34.0 64 49
38.9
54.7
v44 72 84
239.7 69 70
45.2 32 82
40.1

36. Conclusions:
3D ultrasound is a feasible modality for follicle-based fertilitiy state estimation
automated CAD is feasible, more clinical validation needed
winding numbers are robust (scaled) singularity detectors
a robust class of topological properties emerges

37. Multi-scale watershed segmentation
Watershed are the boundaries of merging water basins, when the
image landscape is immersed by punching the minima.
At larger scale the boundaries get blurred, rounded and dislocated.

38. Regions of different scales can be linked by calculating the largest overlap with the region in the scales just above.

39. The method is often combined with nonlinear diffusion schemes
E. Dam, ITU

40. Nabla Vision is an interactive 3D watershed segmentation tool
developed by the University of Copenhagen.
Sculpture the 3D shape by successively clicking precalculated
finer scale watershed details.