IITB-Monash Research Academy An Indian-Australian Research Partnership IIT Bombay Projection Defocus Correction using Adaptive Kernel Sampling and Geometric Correction in Dual-planar Environments Shamsuddin Ladha, Sharat Chandran, Kate Smith-Miles

IITB-Monash Research Academy Outline Goal Features and Contributions Solution Framework Experiments and Results

IITB-Monash Research Academy Goal Create visually appealing projection on dual-planar surfaces Artifacts to correct Defocus blur Geometric distortion

IITB-Monash Research Academy Why? R. Raskar, G. Welch, M. Cutts, A. Lake, L. Stesin, and H. Fuchs. The office of the future: A unified approach to image-based modeling and spatially immersive displays. ACM SIGGRAPH: , Humans crave for large displays Real estate is seldom available The ubiquitous office cubicle is dual-planar! Can we replace the desktop monitor(s) by a projector?

IITB-Monash Research Academy Features and Contributions Non-parametric blur estimation No assumption of exemplar (in-focus) region No explicit depth computation Sparse and adaptive sampling for defocus kernel computation Correction for both luminance and chrominance channels Geometry correction

IITB-Monash Research Academy Solution Framework P1 Q1 Projector P(x) Lambertian Dual-planar Display Surface Q2 Camera I op z In-focus point Not in-focus points f(x; z) – Defocus kernel Γ – Ambient light radiance I op = α f(x;z)P(x) + Γ L. Zhang and S. Nayar. Projection defocus analysis for scene capture and image display. ACM Transactions on Graphics, 25(3): , 2006.

IITB-Monash Research Academy Solution Framework P1 Q1 Projector P(x) Q2 Camera (I op ) z In-focus point Not in-focus points

IITB-Monash Research Academy Solution Framework P’ = arg min{d( α fP + Γ; I ip ) P(x) є [0, 255]} Project P(x) = I ip then I op ≠ I ip Project P’ = (αf ) -1 * (I op - Γ) then I op = I ip Unknowns: α f and Γ

IITB-Monash Research Academy Kernel Estimation Incorrect kernel estimates near the edge Can we do sparse and geometry aware sampling? Project point patterns and measure Point Spread Function (PSF) L. Zhang and S. Nayar. Projection defocus analysis for scene capture and image display. ACM Transactions on Graphics, 25(3): , PSFs as columns of F Defocus kernel

IITB-Monash Research Academy Sparse Sampling With increasing distance blur radius can be linearly approximated Sample few points and interpolate S. Chaudhuri and A. Rajagopalan. Depth from defocus: a real aperture imaging approach. Springer, 1999.

IITB-Monash Research Academy Adaptive Sparse Sampling Measure PSF at few points on each plane Measure PSF at the edge (adaptive) Compute geometry in image and projector spaces!

IITB-Monash Research Academy Geometry in Image Space K. Paidimarri and S. Chandran. Computer Vision, Graphics and Image Processing, pages Lecture Notes in Computer Science, Springer Project structured light Detect bend points Fit a line Detect the edge between two planar surfaces in image space

IITB-Monash Research Academy Geometry in Projector Space Find corresponding points in image and projector space for each planar surface Compute forward and inverse homography matrices Find the edge in projector space Sparse adaptive sampling can be done!

IITB-Monash Research Academy Geometric Correction Display scene surface Corrected input Corrected image (camera) Inv(H L ) Inv(H R ) HLHL HRHR Original input

IITB-Monash Research Academy Summary We have seen Compute defocus kernels and measure Γ Perform geometry correction Next Improve color fidelity of the corrected image

IITB-Monash Research Academy Better Distance Metric L. Zhang and S. Nayar. Projection defocus analysis for scene capture and image display. ACM Transactions on Graphics, 25(3): , P’ = arg min{d( α fP + Γ; I ip ), P(x) є [0, 255]} Sum of squared pixel difference R, G, B channels treated independently Humans differently sensitive to luminance and chrominance Do not treat a pixel in isolation – consider its neighborhood Neighborhood based distance metric in YCbCr space!

IITB-Monash Research Academy (absolute luminance error) (absolute chrominance error) (spatial luminance discontinuities error) (spatial chrominance discontinuities error) non-negative weighting parameters Better Distance Metric Y. Sheng, T. Yapo, and B. Cutler. Global illumination compensation for spatially augmented reality. In Computer Graphics Forum: ,2010 ‒ ‒ ‒

IITB-Monash Research Academy Experiments One time offline processing Run-time correction Homography matrices Defocus kernel matrix Measurement of Γ Pre-warp and stitch image Optimal image using steepest descent

IITB-Monash Research Academy Results

IITB-Monash Research Academy Results

IITB-Monash Research Academy Results

IITB-Monash Research Academy Results (Quantitative) Edge Pixels Input Image Corrected Input Image Output Image Corrected Output Image Cat Plant Dog Leaf

IITB-Monash Research Academy Summary A better method Projection defocus correction Geometric distortion correction Future work Indirect illumination compensation Composite framework for all corrections

IITB-Monash Research Academy References 1.K. Paidimarri and S. Chandran. Computer Vision, Graphics and Image Processing, pages Lecture Notes in Computer Science, Springer L. Zhang and S. Nayar. Projection defocus analysis for scene capture and image display. ACM Transactions on Graphics, 25(3): , Y. Sheng, T. Yapo, and B. Cutler. Global illumination compensation for spatially augmented reality. In Computer Graphics Forum: , R. Raskar, G. Welch, M. Cutts, A. Lake, L. Stesin, and H. Fuchs. The office of the future: A unified approach to image-based modeling and spatially immersive displays. ACM SIGGRAPH, 32(Annual Conference Series): , S. Chaudhuri and A. Rajagopalan. Depth from defocus: a real aperture imaging approach. Springer, 1999.

IITB-Monash Research Academy Thank You