S TRUCTURED L Prasanna RangarajanDr. Marc P Christensen Vikrant R BhaktaDr. Panos Papamichalis

Page  2 Organization Imaging under “Structured Light” –what is “Structured Light” ? –estimating depth using “Structured Light” –Optical Super-Resolution : using Structured Light to overcome the lowpass nature of an imaging system –how is Optical Super-Resolution different from Digital Super-Resolution ? –what is wrong with state-of-the-art in Optical Super-Resolution ? Macroscopic OSR using Structured Light ( Uncalibrated ) Depth estimation using Structured Light ( Uncalibrated ) OSR + Depth estimation in a single setup ( Experimental Results )

Page  3 Structured Light and its applications What is Structured Light ?..... periodic light patterns Why is it useful ? –Traditionally, used to recover depth maps & surface topology –Recently, used in microscopes to resolve spatial detail that cannot be resolved by the microscope

Page  4 Closer look at Depth from Structured Light Phase Measuring Profilometry Principle –project a sinusoidal (periodic) pattern onto the scene, at a known angle –image of scene viewed from a different position AND-OR angle, reveals lateral displacements + frequency changes related to topological variations Mephisto 3D Scanner from 3D Dynamics SL hits from TI website 1.Application Report DLPA021 “Using the DLP Pico 2.0 Kit for Structured Light Applications” 2.Blog entry “3D Metrology and Structured Light”, by Dennis Doane DLP Other DLP based SL-Scanners ViaLUX, GFM, 3D3, ShapeQuest

Page  5 Problem : Cameras behave like low-pass filters because their impulse response is real non-negative finite bandwidth & resolution Objective of Optical Super-Resolution : Improve the resolution of a camera without altering its physical parameters: Optical Super-Resolution using Structured Light has revolutionized microscopy in recent years Principle : shift frequencies outside the passband into the passband How ? modulate the amplitude of a periodic pattern with scene information

Page  6 Optical Super-Resolution using Structured Light How is it different from Digital Super-Resolution ? Optical Super-Resolution See Optical Super-Resolution in action Digital Super-Resolution Recover spatial frequencies ( beyond the optical cutoff ) Recover spatial frequencies lost to aliasing ( but upto the optical cutoff )

Page  7 Optical Super-Resolution using Structured Light Perspective & De-magnification scene-dependent distortion ( useful for recovering depth but not OSR ) Perspective & de-magnification present a real challenge for macroscopic imaging /illumination systems such as commercial cameras/projectors Imaging & illumination systems in Structured Light-microscopy DO NOT experience significant perspective effects imaging parallel lines on railroad track How do we eliminate the scene-dependent distortion ?

Page  8 Solution-1 : Collocate the camera & projector, and illuminate the scene with a specific periodic pattern Daniel A. Vaquero, Ramesh Raskar, Rogerio S. Feris, & Matthew Turk. ”A Projector-Camera Setup for Geometry-Invariant Frequency Demultiplexing”. In IEEE Computer Vision and Pattern Recognition (CVPR'09) Macroscopic OSR using Structured Light Eliminating the scene-dependent distortion Are we really shifting frequencies outside the passband of the optics, into the passband ? Solution-2 : Coincide the camera & projector using a beam-splitter L. Zhang & S. K. Nayar, “Projection Defocus Analysis for Scene Capture and Image Display”, SIGGRAPH2006. “Macroscopic OSR” for imaging systems observing a 3D scene unsolved since 1963 W. Lukosz and M. Marchand, "Optischen Abbildung Unter Ueberschreitung der Beugungsbedingten Aufloesungsgrenze," Opt. Acta 10, (1963)

Page  9 Macroscopic OSR using Structured Light Our contributions –Identify a family of camera+projector setups that can realize OSR in macroscopic imaging, for arbitrary scenes –Unify existing embodiments of Structured Light –Single setup for recovering depth & realizing OSR raw image super-resolved image depth map Publications “Perspective Imaging under Structured Light”, accepted for publication in European Conference on Computer Vision, 2010 “Surpassing the Diffraction-limit of Digital Imaging Systems using Sinusoidal Illumination Patterns”, Computational Optical Sensing and Imaging, OSA Technical Digest (Optical Society of America), 2009 “A Method and Apparatus for Surpassing the Diffraction Limit in Imaging Systems”, filed patent

Page  10 Organization Imaging under “Structured Light” –what is “Structured Light” ? –estimating depth using “Structured Light” –Optical Super-Resolution : using Structured Light to overcome the lowpass nature of an imaging system –how is Optical Super-Resolution different from Digital Super-Resolution ? –what is wrong with state-of-the-art in Optical Super-Resolution ? Macroscopic OSR using Structured Light ( Uncalibrated ) Depth estimation using Structured Light ( Uncalibrated ) OSR + Depth estimation in a single setup ( Experimental Results )

Page  11 Identify the raw image and the exponentially modulated images Macroscopic OSR under Structured Light Complete Workflow Camera images under sinusoidal ilumination

Page  12 Identify the frequency of the modulating pattern After modulation, the DC component in shifts to the carrier frequency Macroscopic OSR under Structured Light Complete Workflow The DC component of the super-resolved image must have zero phase. Use this to identify Camera images under sinusoidal ilumination

Page  13 Aliasing Management avoid aliasing demodulated spatial frequencies that exceed the detector Nyquist frequency Macroscopic OSR under Structured Light Complete Workflow Camera images under sinusoidal ilumination

Page  14 Aliasing Management avoid aliasing demodulated spatial frequencies that exceed the detector Nyquist frequency Macroscopic OSR under Structured Light Complete Workflow How is it done ? (sinc-interpolation) symmetrically increase the size of the modulated images by prior to demodulation Camera images under sinusoidal ilumination

Page  15 Macroscopic OSR under Structured Light Complete Workflow Without aliasing management With aliasing management Aliasing Management avoid aliasing demodulated spatial frequencies that exceed the detector Nyquist frequency Camera images under sinusoidal ilumination

Page  16 Demodulation + Phase Compensation Macroscopic OSR under Structured Light Complete Workflow Any collocated/co-incident camera+projector setup can be used to recover spatial frequencies exceeding the bandwidth of an imaging system Quick Recap Camera images under sinusoidal ilumination

Page  17 Organization Imaging under “Structured Light” –what is “Structured Light” ? –estimating depth using “Structured Light” –Optical Super-Resolution : using Structured Light to overcome the lowpass nature of an imaging system –how is Optical Super-Resolution different from Digital Super-Resolution ? –what is wrong with state-of-the-art in Optical Super-Resolution ? Macroscopic OSR using Structured Light ( Uncalibrated ) Depth estimation using Structured Light ( Uncalibrated ) OSR + Depth estimation in a single setup ( Experimental Results )

Page  18 Recap : Depth from Structured Light Phase Measuring Profilometry Principle –project a sinusoidal (periodic) pattern onto the scene, at a known angle –image of scene viewed from a different position AND-OR angle, reveals lateral displacements + frequency changes related to topological variations Mephisto 3D Scanner from 3D Dynamics SL hits from TI website 1.Application Report DLPA021 “Using the DLP Pico 2.0 Kit for Structured Light Applications” 2.Blog entry “3D Metrology and Structured Light”, by Dennis Doane DLP Other DLP based SL-Scanners ViaLUX, GFM, 3D3, ShapeQuest

Page  19 Depth from Collocated Structured Light Complete Workflow To avoid ambiguities in phase unwrapping, 2 patterns ( 1 small frequency, 1 large frequency) are employed

Page  20 Organization Imaging under “Structured Light” –what is “Structured Light” ? –estimating depth using “Structured Light” –Optical Super-Resolution : using Structured Light to overcome the lowpass nature of an imaging system –how is Optical Super-Resolution different from Digital Super-Resolution ? –what is wrong with state-of-the-art in Optical Super-Resolution ? Macroscopic OSR using Structured Light ( Uncalibrated ) Depth estimation using Structured Light ( Uncalibrated ) OSR + Depth estimation in a single setup ( Experimental Results )

Page  21 Experimental Results Setup-1 : vertically collocated camera+projector

Page  22 Experimental Results - OSR Setup-1 : vertically collocated camera+projector OSR is possible only in the horizontal direction

Page  23 Experimental Results – Estimating depth Setup-1 : vertically collocated camera+projector

Page  24 Experimental Results - OSR Setup-2 : non-collocated camera+projector

Page  25 Experimental Results Setup-2 : non-collocated camera+projector Without aliasing management With aliasing management

Page  26 Closing Arguments & Open Issues Putting things in perspective It is possible to resolve detail exceeding the BW of a macroscopic imaging system There are camera+projector setups that can recover depth information + resolve detail exceeding the bandwidth of the imaging system Can we super-reslove when the optical axes of the camera and projector are crossed ? Can we accommodate aliasing during image capture ? Bar-code scanners Counterfeit Bill Detection Non-contact fingerprint scanning Non-contact archived document scanning Artwork authentication