1. Purely evidence-based multi-scale cardiac tracking using optic flow Hans van Assen 1, Luc Florack 1, Avan Suinesiaputra 2, Jos Westenberg 2, Bart ter Haar Romeny 1 1 Biomedical Image Analysis, Biomedical Engineering, Technical University Eindhoven, Eindhoven, Netherlands 2 Div. Image Processing, Dept. Radiology, Leiden University Medical Center, Leiden, Netherlands

2. Contents  Introduction  Tissue Function  Tagged MRI  Proposed Method  Classical Optic Flow Constraint Equation  Multi-scale Optic Flow Constraint Equation  Application to Image Tuples  Results  Conclusion

3. Introduction  Cardiac pathologies can alter LV contraction patterns, e.g.:  valvular aorta stenosis  myocardial infarction  hyperobstructive cardiomyopathy  Aim: extract and analyse local cardiac tissue function from MR image sequences

4. Global tissue function  Global tissue function often measured based on contours using CINE data  Wall thickness  Wall thickening  Stroke volume  Ejection fraction  Wall dynamics

5. Local tissue function  Local tissue function can be derived from CINE data using  Nonrigid registration 1  Deformable models 2  Drawback: local function is derived from global observations + interpolation 1 Chandrashekara et al, LNCS 3504, 2005 2 Bistoquet et al, IEEE TMI 26(9), 2007 G. Hautvast, TU/e-BME - Philips MS

6. Local function from tagged MRI  Zerhouni 3 introduced tagging in 1988  Axel 4 introduced SPAtial Modulation of Magnetisation (SPAMM) in 1989 3 Zerhouni et al, Radiology 169, 1988 4 Axel et al, Radiology 172, 1989

7. Tagged MRI  Tagging pattern inherent in the tissue  moves along with tissue  Enables local motion analysis

8. Motion Analysis from Tagging  Sparse analysis followed by interpolation and regularisation:  Finite Element Models 5,6  “Virtual tags” 7  Constraints:  Motion field smoothness  Tissue incompressibility 5 Young, Med Image Anal 3(4), 1999 6 Haber et al, LNCS 2208, 2002 7 Axel et al, LNCS 3504, 2005

9. Motion Analysis from Tagging  Dense analysis by  Optic Flow 8,9  HARmonic Phase (HARP) 10  Multiscale Optic Flow 11,12  Nonrigid registration 13,14 8 Prince & McVeigh, IEEE TMI 11(2), 1992 9 Gupta & Prince, 14 th Int. Conf. IPMI, 1995 10 Osman et al, MRM 42, 1999 11 ter Haar Romeny, Front-End Vision & Multi-Scale Analysis, Springer 2004 12 Suinesiaputra et al, LNCS 2878, 2003 13 Sanchez-Ortiz et al, LNCS 3749, 2005 14 Chandrashekara et al, LNCS 3504, 2005

10. Proposed method  Novelties  Multi-scale OFCE 14,* on two time-synchronous sequences with perpendicular tags  No aperture problem  Multi-scale paradigm  Automatic scale selection, per pixel  Sine HARP angle images 14 Florack et al, IJCV 27(3), 1998 * OFCE = optic flow constraint equation

11. Classical OFCE (Horn & Schunck 15 )  Assumption ( L is intensity, t is time)  Andwhen  Using 0-order Taylor expansion (subtract L ) (notice: 1 equation and 2 unknowns, u and v) 15 Horn and Schunck, Artif. Intell. 17, 1981

12. The isophote landscape of an image changes drastically when we change our aperture size. This happens when we move away or towards the scene with the same camera. Left: observation of an image with  = 1 pix, isophotes L=50 are indicated. Right: same observation at a distance twice as far away. The isophotes L=50 have now changed.

13. Scalar images: intensity is kept constant with the divergence Density images: intensity ‘dilutes’ with the divergence Two types of images need to be considered:

15. Multi-scale optic flow constraint equation: For scalar images: For density images: The velocity field is unknown, and this is what we want to recover from the data. We like to retrieve the velocity and its derivatives with respect to x, y, z and t. We insert this unknown velocity field as a truncated Taylor series, truncated at first order.

16. ‘Spurious resolution’: artefact of the wrong aperture What is the best aperture? Aliasing, partial volume effect

17. Regularization is the technique to make data behave well when an operator is applied to them. A small variation of the input data should lead to small change in the output data. Differentiation is a notorious function with 'bad behaviour'. Some functions that can not be differentiated.

18. The formal mathematical method to solve the problems of ill-posed differentiation was given by Laurent Schwartz: A regular tempered distribution associated with an image is defined by the action of a smooth test function on the image. The derivative is:

19. Fields Medal 1950 for his work on the theory of distributions. Schwartz has received a long list of prizes, medals and honours in addition to the Fields Medal. He received prizes from the Paris Academy of Sciences in 1955, 1964 and 1972. In 1972 he was elected a member of the Academy. He has been awarded honorary doctorates from many universities including Humboldt (1960), Brussels (1962), Lund (1981), Tel-Aviv (1981), Montreal (1985) and Athens (1993). Laurent Schwartz (1915 - 2002)

20. Multi-scale differential operators

21. Multi-scale OFCE  Florack, Niessen and Nielsen came up with two new notions:  the optic flow constraint equation is of an observed physical system:  Gaussian differential operators  the velocity can be different in every pixel, so: the derivative with respect to the (unknown) velocity field must be zero.  The derivative with respect to a velocity field is called a Lie-derivative.  The Lie derivative of a function F with respect to a vectorfield must be zero: For scalar images: For density images:

22. The multi-scale OFCE For scalar images: For scalar images the observed (convolved with the aperture function) the optic flow constraint equation (OFCE) is written as: from which we get by partial integration:, or

23. The multi-scale OFCE For density images: For density images the observed (convolved with the aperture function) the optic flow constraint equation (OFCE) is written as: from which we get by partial integration:, or

24. Approximation of the velocity field The velocity field {u,v} is unknown. We will approximate it, put it in the equation and solve for it. We can approximate it to zero'th order: and to first order: We have 8 unknowns, and need equations to solve them (per pixel):

25. Four equations are given by: The remaining 4 have to come from external information.

26. The normal constraint in 2D is expressed as where L x and L y are constant. So we get four more equations:

27. This gives 8 equations with 8 unknowns to be solved in every pixel: But: Case: density images. Note the third order derivatives and scales σ. Suinesiaputra et al. 2005

28. Limitation 1  Assumption does not hold, due to spin-lattice relaxation (T 1 )

29. Harmonic Phase technique Spatial domain SPAMM images Fourier domain filter first harmonic peak Spatial domain Sine of HARP angle images

30. Limitation 2  Twice as many unknowns as equations (aperture problem)

31. Limitation 2  Proposed “normal flow constraint” 11 is erroneous 11 Suinesiaputra et al, LNCS 2878, 2003 so we may not add the second second set of 4 equations...

32. Normal flow Flow direction is colour-coded. Should be one color during horizontal movement and one color during vertical movement. Ball moving first horizontally and then vertically. Gradient on the ball in radial directions.

33. 1 st order multiscale OFCE

34. Application to image tuples Image 1 Image 2 Images 1 & 2 with

35. Application to image tuples

36.  Solve for every pixel in every frame  in multiple scale space  automatically select proper scale: how far is the coefficient matrix off from being singular?

37. Automatic scale selection  Stability of matrix C can be calculated with  (Squared) Frobenius norm:  Condition number: is an eigen value of C C is a m x n matrix

38. Scale selection: The condition number of the coefficient matrix exhibits an optimum over scale in many pixels, given the local density of texture.

39. Artificially created test image sequence for validation purposes Scale selection map (Frobenius norm)

40. Automatic scale selection  Using condition number Spatial scale in vertical tagging image Spatial scale in horizontal tagging image Temporal scale

41. Results SPAMMSine of HARPMasked sine of HARP

42. Tuple Optic Flow vs Normal FlowTuple Optic Flow

43. Tuple Optic Flow vs PC-MRI

44. Conclusion  We developed a tracking method that  Works at pixel resolution  Yields displacements and their differential structure (important for strain, strain rates, tissue acceleration)  Is straightforwardly extensible to 3D  Uses multi-scale paradigm with automatic scale-selection  Does not need constraints regularization

45. Future work  Evaluate quantitatively on large data set  Compute strain, strain rate, tissue acceleration  Extend to 3D using true 3D data (currently implemented)  Classify strain patterns and tissue function  Accelerate computation

46. Acknowledgements  The Netherlands Organisation for Scientific Research (NWO) is greatfully acknowledged for financial support of Luc Florack, PhD (VICI award)  BSIK is greatfully acknowledged for financial support of Hans van Assen, PhD BioMedical Image Analysis Group, TU Eindhoven