2. Multi-Focus Range Sensor using Coded Aperture Takashi MATSUYAMA Kyoto Univ. Shinsaku HIURA Osaka Univ.

3. Depth from Defocus zDepth estimation using the quantity of blurring zPassive, and no physical motion - suit for real-time measurement - small and light-weight equipment zStable depth estimation using relative defocus analysis z  small and optimal sensor is necessary Depth from Focusing Search focused position by moving lens – not suitable for real-time measurement Depth from Defocus distance is estimated from the amount of blurring – no physical motion, suit for real-time measurement Single image analysis Planar photograph is indistinguishable from real object Multiple image analysis stable analysis against varied texture

4. Multi-Focus Camera zConvert color CCD camera zEach CCD is moved 1mm toward Optical Axis zNeutral density by re- coating the prism surface zSmall and light as same as usual color CCD camera

5. Telecentric Optics zApreture is set at front focal plane zImage size/intensity are equal among each image plane. Only blurring varies zFirst Applying to DFD: Nayar Usual optics Telecentric optics

6. Problems of past Depth from Defocus research zHigh frequency information is lost by Blur(=LPF) y  unstable range estimation y  too sensitive to the texture or environment y  high-quality noiseless image is necessary Ex. Nayar averages over 256 images to eliminate noise zIf the “blur” effect is changed to High-pass or Band-pass filter, it is possible to stabilize range estimation y  Structured aperture (coded aperture)

7. Multi-focus camera with a Coded Aperture zBlurring kernel is the scaled shape of the aperture zMagnification ratio is varied with object distance

8. Multi-focus camera with a coded aperture

9. Process of blurring Blur kernel a(x,y) convolution Input image i 1 (x,y) Input image i 2 (x,y) Input image i 3 (x,y) Dist. u Focus v Image s(x,y) K 1 magn. K 2 magn K 3 magn. Mathematical model of blurring Ｗ ｍ ： image plane

10. Range estimaton using FFT zFourier transform of blurring process zElimination of original image information zEval. Func. of range estimation p, q ： spatial freq. ， v ： focus position Original image info. is eliminated using division Minimum residual is searched by varying v (focus position). First term is calculated from two input images, and second is from blurring Model.

11. Process flow Windowing&FFT Eliminate scene texture info. Minimize residual  range value

12. Restoration of blur-less image zInverse filter zHigh-quality image can be restored, because yusing multipule images yRich information is remained using coded aperture v ： focused position W m ： weight calculated from v

13. Aperture design(1) zSpatial frequency response must be structured for easy and stable analysis zHigh freqency information must be preserved Spacial frequency Gain

14. Aperture design(2) zUsual circular aperture is not optimal. This type is suit for beautiful defocused photograph. Blurring is not observed. Monotonic, and low gain when blurred. Feasible, but more peaks are desired.

15. Simple example:2 holes aperture zFourier transform of blurring kernel is cos() zPeriod of cos() is varied with object distance. Blurring kernel （ diameter of hole is ignored ） 1-D Fourier transform of blurring kernel

16. Robustness of range estimation This “valley shape” shows the ability of robust depth estimation zResidual of evaluation function with varied range

17. Experiment: input images First CCD Center CCD Last CCD

18. Restored blur-less image

19. Reconstructed object shape Blur-free image(partial) 3-D object shape

20. Range analysis using convolution Blurring kernel is convolved optically Same convolution is applied to the opposite image, and we acquire the same image. （ becase of commutative law of convolution) Depth is estimated by searching the position that gives same images Usual circular aperture can not be used, because twice blurring gives almost flat images. Coded aperture enabled such simple principle.

21. Experiment Measured scene Input image Range image

22. Asymmetric aperture design zError range estimation is suppressed using asymmetric aperture because phase part of spacial frequency is changed. Asymmetric aperture Convolution kernel is changed at the focus plane (phase part of spacial frequency is changed)

23. Aperture symmetry and robustness of range estimation Input image

24. Motion sequence measurement using input image recording z3 images are recorded as RGB image on optical video disc recorder zImage is deteriorated by Y-C translation and cross-talk between RGB channel.

25. Result: finger motion Input imagesRange images

26. conclusions zSmall/light multi-focus camera is developed zCoded aperture is applied to depth from defocus zStable range estimation is achieved yRange estimation/image restoration by FFT yRange estimation by convolution zRecorded image can be used for motion analysis because range estimation is robust enough zReal-time range measurement is possible using image processing hardware. Simple method is easily ported to parallel hardware.

27. Real-time calculation using image processing hardware zSimple convolution method can easily be ported on image processing hardware zMassive-parallel hardware, IMAP-Vision is used for experiment zSpec: 10G instruction/sec by 256PE zCalculation of 25frames/sec can be achieved. （ However, this board does not have RGB separate capture interface; experiment is calculation only)