N ational T aiwan U niversity T aipei, T aiwan (R.O.C.) Feb Institute of Communication Engineering Digital Image and Signal Processing Laboratory (DISP) MD-531 Depth Estimation and Focus Recovery ( 景深估計與聚焦重建 ) Speaker: Yu-Che Lin ( 林于哲 ) Adviser : Prof. Jian-Jiun Ding ( 丁建均 教授 )

Institute of Communication Engineering N ational T aiwan U niversity T aipei, T aiwan (R.O.C.) Feb Digital Image and Signal Processing Laboratory (DISP) MD Outlines Motivations Overview on previous works Structure of camera lens / Geometric optics Introduction to Fourier optics Blurring function / Equal-focal assumption Binocular / Stereo vision Vergence Monocular Depth from focus  Estimator of degree on focus / Sum of Laplacian  Interpolation Depth from defocus  Arbitrary changing camera parameters with large variation  Trace amount on changing camera parameters

Institute of Communication Engineering N ational T aiwan U niversity T aipei, T aiwan (R.O.C.) Feb Digital Image and Signal Processing Laboratory (DISP) MD Linear canonical transform (LCT) upon the optical system Linear canonical transform (LCT) Approximation on the optical system by LCTs Focus recovery: common method, alternative method The common method Alternative method: one point focus recovery Simulation on simple pattern Conclusions References

Institute of Communication Engineering N ational T aiwan U niversity T aipei, T aiwan (R.O.C.) Feb Digital Image and Signal Processing Laboratory (DISP) MD Motivations Depth is an important information for robot and the 3D reconstruction. Image depth recovery is a long-term subject for other applications such as robot vision and the restorations. Most of depth recovery methods based on simply camera focus and defocus. Focus recovery can help users to understand more details for the original defocus images.

N ational T aiwan U niversity T aipei, T aiwan (R.O.C.) Feb Institute of Communication Engineering Digital Image and Signal Processing Laboratory (DISP) MD-531 Overview on previous works

Institute of Communication Engineering N ational T aiwan U niversity T aipei, T aiwan (R.O.C.) Feb Digital Image and Signal Processing Laboratory (DISP) MD Structure of camera lens (1) Physical lens. Structure of camera lens against the aberration ( 像差 ) Two of aberrations : Chromatic ( 色像 ), Spherical aberration. Higher wavelength, lower refractive index cited on : cited on : of_lenses convex concave cited on : of_lenses

Institute of Communication Engineering N ational T aiwan U niversity T aipei, T aiwan (R.O.C.) Feb Digital Image and Signal Processing Laboratory (DISP) MD Common solutions for aberrations: “ Asperical ” lens and complementary lenses (Groups). One shot always has multiple lenses. Geometric on imaging. F : focal length u: object dist. v: imaging dist. D: lens diameter R: blurring radius s: dis. between lens and screen (CCD) cited on : htm Complementary convex and concave lenses Structure of camera lens (2)

Institute of Communication Engineering N ational T aiwan U niversity T aipei, T aiwan (R.O.C.) Feb Digital Image and Signal Processing Laboratory (DISP) MD Structure of camera lens (3) Combination by lenses of the real camera. The effective focal length : Due to the above effective value, we can now just ignore the complicated combinations. F L1L1 L2L2

Institute of Communication Engineering N ational T aiwan U niversity T aipei, T aiwan (R.O.C.) Feb Digital Image and Signal Processing Laboratory (DISP) MD Introduction to Fourier optics (1) Aperture effect. When the wave incident through an aperture, the observed field is the combination: 1.The unperturbed incident wave by geometric optics. 2.A diffractive wave originating from the rim of the aperture. Diffraction. Fresnel principle ( near-field diffraction ) Fraunhofer principle ( far-field diffraction ) strict on field distance,

Institute of Communication Engineering N ational T aiwan U niversity T aipei, T aiwan (R.O.C.) Feb Digital Image and Signal Processing Laboratory (DISP) MD Introduction to Fourier optics (2) The Huygens-Fresnel transform. Considering a square wave :

Institute of Communication Engineering N ational T aiwan U niversity T aipei, T aiwan (R.O.C.) Feb Digital Image and Signal Processing Laboratory (DISP) MD Introduction to Fourier optics (3) The field intensity through the circular aperture ( ex: camera aperture ) by unit amplitude plane wave under Fraunhofer diffraction theory is actually a sinc function. The structure inside the camera shot should more like a near-field condition, so the intensity pattern acts more like a Gaussian function. y1y1 y0y0 x1x1 x0x0 r 01 P0(x0,y0)P0(x0,y0) P1(x1,y1)P1(x1,y1)

Institute of Communication Engineering N ational T aiwan U niversity T aipei, T aiwan (R.O.C.) Feb Digital Image and Signal Processing Laboratory (DISP) MD Blurring function / Equal-focal assumption (1) screen F D/2 F u s v Biconvex 2R : R<0 Blurring radius: R<0 Blurring radius: R>0 F D/2 F u s 2R : R>0 screen v

Institute of Communication Engineering N ational T aiwan U niversity T aipei, T aiwan (R.O.C.) Feb Digital Image and Signal Processing Laboratory (DISP) MD Blurring function / Equal-focal assumption (2) Due to geometric optics, the intensity inside the blur circle should be constant. Considering of aberration and diffraction and so on, we easily assume a blurring function: : diffusion parameter Diffusion parameter is related to blur radius: Derived from triangularity in geometric optics For easy computation, we always assume that foreground has equal-diffusion, background has equal-diffusion and so on However, this equal-focal assumption will be a problem K: calibrated by each specific camera

N ational T aiwan U niversity T aipei, T aiwan (R.O.C.) Feb Institute of Communication Engineering Digital Image and Signal Processing Laboratory (DISP) MD-531 Binocular / Stereo vision

Institute of Communication Engineering N ational T aiwan U niversity T aipei, T aiwan (R.O.C.) Feb Digital Image and Signal Processing Laboratory (DISP) MD Vergence Vergence movement : is some kind of slow eye movement that two eyes move in different directions. Disadvantage : Correspondence problem ( trouble ).

N ational T aiwan U niversity T aipei, T aiwan (R.O.C.) Feb Institute of Communication Engineering Digital Image and Signal Processing Laboratory (DISP) MD-531 Depth from focus

Institute of Communication Engineering N ational T aiwan U niversity T aipei, T aiwan (R.O.C.) Feb Digital Image and Signal Processing Laboratory (DISP) MD Estimator of degree on focus / Sum of Laplacian Actively taking pictures at different observer distance or object distance. Estimator of degree on focus. we need an operator to abstract how “ focused ” the region is  Since the blur model is a low pass filter, the estimator can be a Laplacian  Such operator point to a measurement on a single pixel influence, a sum of Laplacian operator is needed:

Institute of Communication Engineering N ational T aiwan U niversity T aipei, T aiwan (R.O.C.) Feb Digital Image and Signal Processing Laboratory (DISP) MD Interpolation (1) We use Gaussian interpolation to form a set of approximations. We have d p that is the camera displacement performing perfect focused :, dkdk [SML] displacement NPNP Focus measure NkNk N k-1 d k- 1 dpdp Measured curve Ideal condition d k+1 N k+1

Institute of Communication Engineering N ational T aiwan U niversity T aipei, T aiwan (R.O.C.) Feb Digital Image and Signal Processing Laboratory (DISP) MD Interpolation (2) The depth solution d p from above Gaussian :

N ational T aiwan U niversity T aipei, T aiwan (R.O.C.) Feb Institute of Communication Engineering Digital Image and Signal Processing Laboratory (DISP) MD-531 Depth from defocus

Institute of Communication Engineering N ational T aiwan U niversity T aipei, T aiwan (R.O.C.) Feb Digital Image and Signal Processing Laboratory (DISP) MD Arbitrary changing camera parameters with large variation Two diffusion parameters are considering. Intuitively, we can get two pictures by changing one of camera parameters and solve the triangularity problem. On the spatial domain or frequency domain. i : equal focal subimage Replace to solve u

Institute of Communication Engineering N ational T aiwan U niversity T aipei, T aiwan (R.O.C.) Feb Digital Image and Signal Processing Laboratory (DISP) MD Trace amount on changing camera parameters (1) More accurate by changing camera parameters with trace amount. We use the power spectral density : Utilizing the fact that differential on the Gaussian function is still a Gaussian.

Institute of Communication Engineering N ational T aiwan U niversity T aipei, T aiwan (R.O.C.) Feb Digital Image and Signal Processing Laboratory (DISP) MD Trace amount on changing camera parameters (2) We have no idea on diffusion parameters, but we can replace it by camera parameters by differential factor.

N ational T aiwan U niversity T aipei, T aiwan (R.O.C.) Feb Institute of Communication Engineering Digital Image and Signal Processing Laboratory (DISP) MD-531 Linear canonical transform (LCT) upon the optical system

Institute of Communication Engineering N ational T aiwan U niversity T aipei, T aiwan (R.O.C.) Feb Digital Image and Signal Processing Laboratory (DISP) MD Linear canonical transform (LCT) : The LCT gives a scalable kernel to describe wave propagation such as the fractional Fourier transform and the Fresnel transform and etc. Definition (normalize as four parameters).  Linear canonical transform (LCT) (1)

Institute of Communication Engineering N ational T aiwan U niversity T aipei, T aiwan (R.O.C.) Feb Digital Image and Signal Processing Laboratory (DISP) MD Linear canonical transform (LCT) (2) Through a brief derivation, we can see that a combination of the LCTs is still a LCT. The reason for the parameters mapping is for its convenience coordinates transformation on the time-frequency distribution. Some important properties connected by the LCT : Scaling, phase delay (chirp multiplication), modulation, chirp convolution and the fractional Fourier transform.

Institute of Communication Engineering N ational T aiwan U niversity T aipei, T aiwan (R.O.C.) Feb Digital Image and Signal Processing Laboratory (DISP) MD Linear canonical transform (LCT) (3) Sinc value in the frequency domain of a rectangle signal in the time domain. Its parameters of LCT (A,B,C,D) = (0,1,-1,0) : the Fourier transform.

Institute of Communication Engineering N ational T aiwan U niversity T aipei, T aiwan (R.O.C.) Feb Digital Image and Signal Processing Laboratory (DISP) MD Approximation on the optical system by LCTs We now consider a simple and common optical system. The effective LCT parameters : UoUo UlUl U l’ UiUi z s Phase delay (chirp multiplication) Free space diffraction (chirp convolution)

N ational T aiwan U niversity T aipei, T aiwan (R.O.C.) Feb Institute of Communication Engineering Digital Image and Signal Processing Laboratory (DISP) MD-531 Focus recovery: common method, alternative method

Institute of Communication Engineering N ational T aiwan U niversity T aipei, T aiwan (R.O.C.) Feb Digital Image and Signal Processing Laboratory (DISP) MD The common method (1) The most common focus recovery method : Based on the assumption that a simply constructed image scene has two layers, foreground and background. Two input images, one focus on foreground ( f1 ) and the other focus on background ( f2 ). Using adjustable values R1 and R2 to generate images. Where ( i = 1, 2, a, b ) indicates point spread functions, note that a and b are adjusted parameters -- Design filters K1 and K2 !! g1 R1 K1 K2 g2 f R2

Institute of Communication Engineering N ational T aiwan U niversity T aipei, T aiwan (R.O.C.) Feb Digital Image and Signal Processing Laboratory (DISP) MD The common method (2) The matrix form. Considering the existence of the inverse matrix (singular or nonsingular).

Institute of Communication Engineering N ational T aiwan U niversity T aipei, T aiwan (R.O.C.) Feb Digital Image and Signal Processing Laboratory (DISP) MD The common method (3) Filters result.

Institute of Communication Engineering N ational T aiwan U niversity T aipei, T aiwan (R.O.C.) Feb Digital Image and Signal Processing Laboratory (DISP) MD Alternative method : one point focus recovery (1) Depth of field (DOF). The ideal case (larger aperture). Larger Aperture 。。。 thin lens sensor Position of object F These blurred areas are too large for the HVS and result in two blurring areas.

Institute of Communication Engineering N ational T aiwan U niversity T aipei, T aiwan (R.O.C.) Feb Digital Image and Signal Processing Laboratory (DISP) MD Alternative method : one point focus recovery (2) The effective focused interval (smaller aperture). Smaller Aperture sensor thin lens 。 F 。。 These blurred areas are too small for the HVS and results in an effective focused plane. Effective “depth of field” interval Position of object

Institute of Communication Engineering N ational T aiwan U niversity T aipei, T aiwan (R.O.C.) Feb Digital Image and Signal Processing Laboratory (DISP) MD Alternative method : one point focus recovery (3) Approximation by LCTs. Paraxial approximation (phase delay).  The original Fresnel transform :

Institute of Communication Engineering N ational T aiwan U niversity T aipei, T aiwan (R.O.C.) Feb Digital Image and Signal Processing Laboratory (DISP) MD Alternative method : one point focus recovery (4) Flow chart for the alternative method : one point focus recovery. SML measurement Defocused image pair Full focused image Maximum value searching Depth measurement of a point Small aperture construction Linear canonical transform based on constructed optical system focal point Using the specific depth to retrieve imaging distance

Institute of Communication Engineering N ational T aiwan U niversity T aipei, T aiwan (R.O.C.) Feb Digital Image and Signal Processing Laboratory (DISP) MD Simulation on simple pattern Considering the Gaussian point light source. For simplicity, we assumes the parameters :  (a) – the input Gaussian pattern.  (b) – LCT for s = 27 mm.  (c) – LCT for s = 30 mm.  (d) – Inverse LCT for s = 27 mm. ab d c

Institute of Communication Engineering N ational T aiwan U niversity T aipei, T aiwan (R.O.C.) Feb Digital Image and Signal Processing Laboratory (DISP) MD Conclusions Most of the literature discussed on the depth or the depth recovery fall in the equal focal problems (DFD, DFF) or the correspondent problems (stereo vision). Relying on the LCTs by the paraxial approximation system can avoid such problems. Using LCTs is more like a deblurring procedure. Such action can keep the original realities of the images from disturbance.

Institute of Communication Engineering N ational T aiwan U niversity T aipei, T aiwan (R.O.C.) Feb Digital Image and Signal Processing Laboratory (DISP) MD References Y. Xiong and S. A. Shafer, “Depth from focusing and defocusing,” IEEE Conference on Computer Vision and Pattern Recognition, pp , M. Subbarao, “Parallel depth recovery by changing camera parameters,” Second International Conference on Computer Vision, 1988, pp , K. S. Pradeep and A. N. Rajagopalan, “Improving shape from focus using defocus information,” 18th International Conference on Pattern Recognition, 2006, vol. 1, p.p , Sept M. Asif and A. S. Malik, T. S. Choi “3D shape recovery from image defocus using wavelet analysis,” IEEE International Conference on Image Processing, 2005, vol. 1, pp , Sept K. Nayar and Y. Nakagawa, “Shape from focus,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 16, issue 8, pp , Aug, 1994.

Institute of Communication Engineering N ational T aiwan U niversity T aipei, T aiwan (R.O.C.) Feb Digital Image and Signal Processing Laboratory (DISP) MD Y. Y. Schechner and N. Kiryati, “ Depth from defocus vs. stereo: how different really are they?, ” in ICPR 1998, vol. 2, pp , Aug M. Haldun Ozaktas, Zeev Zalevsky and M. Alper Kutay, “ The fractional Fourier transform with applications in optics and signal processing, ” JOHN WILEY & SONS, LTD, New York, A. Kubota and K. Aizawa, “ Inverse filters for reconstruction of arbitrarily focused images from two differently focused images, ” IEEE Conferences on Image Processing 2000, vol.1, pp , Sept A. P. Pentland, “ A new sense for depth of field ”, IEEE Transaction on Pattern Analysis and Machine Intelligence, vol. 9, no. 4, pp , M. Hansen and G. Sommer, “ Active depth estimation with gaze and vergence control using gabor filters, ”, Proceedings of the 13th International Conference on Pattern Recognition 1996, vol. 1, pp , Aug