1. PDE-based Methods for Image and Shape Processing Applications Alexander Belyaev School of Engineering & Physical Sciences Heriot-Watt University, Edinburgh Institute of Sensors, Signals & Systems

2. Very active research area + dozens of books and thousands of research papers  Joachim Weickert, Anisotropic Diffusion in Image Processing  Tony Chan & Jianhong Shen, Image Processing and Analysis: Variational, PDE, Wavelet, and Stochastic Methods.  Guillermo Sapiro, Geometric Partial Diffrential Equations and Image Analysis.  Gilles Aubert & Pierre Kornprobst, Mathematical Problems in Image Processing.

3. Equations An algebraic equation An ordinary differential equation A partial differential equation Usually it is not possible to solve partial differential equations (PDEs) analytically and they are solved numerically.

4. I. Partial Differential Equations (PDEs) PDEs are equations involving partial derivatives of an unknown function. For example, the so-called heat or diffusion equation is given by Describes temperature distribution in a material or concentration of particles in a medium or a random walk.

5. Fourier transform, Gaussian smoothing, and linear diffusion It explains why boosts high frequencies Fourier transform w.r.t. x and y A very simple ordinary differential equation. Can be easily solved analytically

6. I. Linear diffusion (heat/diffusion equation)  Proof: apply Fourier transform w.r.t. x, solve the resulting ordinary differential equation, apply inverse Fourier transform to the solution.  Thus linear diffusion is equivalent to Gaussian smoothing (convolution with Gaussian). This leads to a simple way to solve the heat equation on a plane (in space).  In practice the heat equation is usually solved numerically by using finite difference approximations or finite element methods.

7. I. A Brief History of PDE Methods in IP 1955

8. I. Kovasznay & Joseph: inverting diffusion for image sharpening purposes

9. I. Image enhancement/deblurring Unsharp masking (a popular image enhancement technique) Iterated unsharp masking

10. I. Dennis Gabor on image enhancement Dennis Gabor (Nobel prize in physics for inventing holography, 1971): “Information theory and electron microscopy”, 1965

11. Simple image sharpening Original 111 111 111 Convolution with mask Blurred

12. I. Simple image sharpening Original 111 1-81 111 000 010 000 - Boosting high frequencies Sharpened

13. I. Image enhancement with stabilized inverse diffusion Can be used for deblurring Gaussian blur is ill posed (unstable). So a regularization is needed A. Belyaev, ”Implicit image differentiation and filtering with applications to image sharpening.” SIAM Journal on Imaging Sciences, 6(1):660–679, 2013.

14. I. Stabilized inverse diffusion 2

15. I. Stabilized inverse diffusion 3

16. I. Very recent use of heat (diffusion) equation

17. PDE: Hopf-Cole transformation eikonal equation Hopf-Cole transformation

18. I. PDE: Hopf-Cole transformation rhs = ones(N,1); u = -sqrt(t)*log(1-(t*D+eye(N))\rhs); Laplacian

19. I. PDE: Hopf-Cole transformation

20. I. Applications: Dynamic distance-based shape features for gait recognition T. P.Whytock, A. Belyaev, and N. M. Robertson, ”Dynamic distance-based shape features for gait recognition.” Journal of Mathematical Imaging and Vision. 2014.

21. I. Dynamic distance-based shape features for gait recognition

23. II. Perona-Malik Diffusion

24. II. Perona-Malik diffusion with Matlab P. Perona, T. Shiota, and J. Malik, “Anistropic Diffusuion.” Geometry-Driven Diffusion in Computer Vision, 1994.

25. II. Repeated averaging and nonlinear diffusion Gray-scale image Iterative local averaging : Gaussian smoothing edge-enhancing averaging

26. II. Repeated averaging and nonlinear diffusion Gray-scale image Iterative local edge-enhancing averaging : Perona-Malik diffusion :  Efficient numerical schemes  Possibilities for various generalizations and improvements

27. II. Perona-Malik diffusion and its extensions nonlinear diffusion can be used for enhancing small-scale details

28. II. Nonlinear diffusion for mesh processing 2D ImageTriangle mesh

29. II. Nonlinear diffusion for surface denoising Smoothing normals Updating vertex positions 0 Y. Ohtake, A. Belyaev, and I. A. Bogaevski, “Mesh Regularization and Adaptive Smoothing.” Computer-Aided Design, Vol. 33, No. 11, 2001, pp. 789–800.

30. II. Perona-Malik nonlinear diffusion for surface denoising Nonlinear diffusion of mesh normals Gaussian like smoothing Adding noise

31. II. Perona-Malik nonlinear diffusion for surface denoising Nonlinear diffusion of mesh normals Conventional mesh smoothing

32. II. Perona-Malik nonlinear diffusion for surface denoising

33. II. Image compression with nonlinear diffusion “. I. Galić, J. Weickert, M. Welk, M. Bruhn, A. Belyaev, H.-P. Seidel: “Image compression with anisotropic diffusion”. Journal of Mathematical Imaging and Vision. 31(2-3): 255-269, 2008.

34. III. Intro to Variational Image Processing: gradient

35. III. Intro to Variational Image Processing: max / min

36. III. Membrane energy Minimizing E(u) by gradient descent: So we have to stop this gradient descent flow at some t=T

37. III. Membrane Energy Minimizing E λ (u) by gradient descent:

38. III. Variational Approach to Image Smoothing Links to robust statistics

39. III. Variational Approach to Image Smoothing Resembles least-square fitting Given image I(x,y), we approximate it by u(x,y) data fitting termsmoothing term Energy (functional) We have to learn how to differentiate E(u) w.r.t. u(x) λ controls the amount of smoothing we add to I(x,y)

40. III. Edge-preserving image smoothing

41. III. Total Variation Energy

42. III. The Rudin-Osher-Fatemi (ROF) model

44. B.Goldlücke, Foundations of Variational Image Analysis, Lecture Notes, 2011 III. The Rudin-Osher-Fatemi (ROF) model

46. III. TV image processing models

47. III. Gradient descent minimization Curvature flow Linear diffusion

48. III. The Rudin-Osher-Fatemi (ROF) model Diffusion (heat) Total variation Original signal

49. III. TV image inpainting B.Goldlücke, Foundations of Variational Image Analysis, Lecture Notes, 2011 Original image I(x,y) Removed region R Inpainted result

50. IV. Image Deblurring  Image restoration is to restore a degraded image back to the original image  Linear image degradation model blur additive noise

51. IV. A variational approach to image deblurring Wiener filtering

52. IV. Variational image deblurring

53. IV. TV deblurring

54.  non-blind deblurring  blind deblurring IV. TV deblurring

55. V. Snakes: Active Contour Models

56. V. Geodesic active contours

57. Riemannian metric (conformal, for the sake of simplicity)

58. V. Geodesic active contours

59. V. Geodesics in heat  Possibly this approach can be used for a very efficient implementation of geodesic active contours.

60. VI. B.K.P.Horn: Shape from Shading Berthold Klaus Paul Horn, Robot Vision. The MIT Press. 1986.

61. VI. B.K.P.Horn: Shape from Shading

62. VII. Mumford-Shah Approach

63. VII. Blake-Zisserman = Mumford-Shah

64. VII. Chan-Vese active contours without contours

65. The end. Thank you!