Mosaic-Based 3D Scene Representation and Rendering Zhigang Zhu Visual Computing Lab Department of Computer Science City College and Graduate Center City University of New York C C V C L Allen Hanson Computer Vision Laboratory Computer Science Department University of Massachusetts

C C V C L Zhu & Hanson ICIP '05 Some Real World Problems Environmental monitoring (NSF) - airborne video of Amazon rain forest: far-view (1000 ft) Environmental monitoring (NSF) - airborne video of Amazon rain forest: far-view (1000 ft) City/Campus modeling (AFRL, NYSIA) - airborne video: far-view (1000 ft) Robot Navigation (DARPA, ARO)- ground video: medium range (~100ft) Under-vehicle inspection (ACT Inc.) - car drives over cameras: near-view (< 8 in)

C C V C L Zhu & Hanson ICIP '05 Objectives Input: 2D array of cameras Input: 2D array of cameras  multiple viewpoints; motion parallax Output: Image-based representation Output: Image-based representation  WFOV, Stereo, Occlusion Rep.

C C V C L Zhu & Hanson ICIP '05 Related Work 2D mosaics from a pure rotating camera 2D mosaics from a pure rotating camera  QuickTime VR, VideoBrush, Microsoft … Stereo mosaics from off-center rotating camera(s) Stereo mosaics from off-center rotating camera(s)  two cameras (Huang & Hung 1998)  one camera ( Peleg et al PAMI 01, Shum & Szeliski ICCV99 ) Panoramic imaging from a translating camera Panoramic imaging from a translating camera  Manifold projection (Peleg et al CVPR ’ 97, PAMI2000)  multiple-center-of-projection (Rademacher & Bishop, SigGraph'98)  EPI-based 3D reconstruction (Zhu, XU & Lin, CVPR99)  linear pushbroom camera (R. Gupta & R. Hartley, PAMI 97)  Parallel projection ( Chai & Shum, CVPR ’ 00) Geo-referenced mosaic Geo-referenced mosaic  Plane+Parallax+DEM (Kumar, Sawhney, et al, ICPR98, 2000)

C C V C L Zhu & Hanson ICIP '05 Outline Mosaic Representation Mosaic Representation Research Issues Research Issues Real Applications Real Applications  Air, Ground and Under-Vehicle Summary Summary

C C V C L Zhu & Hanson ICIP '05 Mosaic Representations Perspective Image Perspective Image notes: (one viewpoint, occlusion in both forward and backward direction) notes: (one viewpoint, occlusion in both forward and backward direction) Orthogonal Image (Nadir view) Orthogonal Image (Nadir view) notes: only nadir (or frontal) view - cannot see sides notes: only nadir (or frontal) view - cannot see sides Images with Oblique Parallel Projection Images with Oblique Parallel Projection Notes: multiple parallel - can see everything (generalization of orthogonal image) Notes: multiple parallel - can see everything (generalization of orthogonal image)

C C V C L Zhu & Hanson ICIP '05 Perspective Images One viewpoint O One viewpoint O  Narrow FOV Multiple viewing directions Multiple viewing directions invisible O

C C V C L Zhu & Hanson ICIP '05 Orthogonal Images Multiple viewpoints Multiple viewpoints  Wide FOV One viewing direction -Nadir view One viewing direction -Nadir view  Occlusion invisible

C C V C L Zhu & Hanson ICIP '05 Oblique Parallel Projections Multiple viewpoints – Wide FOV Multiple viewpoints – Wide FOV Multiple viewing directions Multiple viewing directions  Parallel rays in each image  Various oblique angles: invisible visible invisible visible Nadir‘Forward’‘Backward’

C C V C L Zhu & Hanson ICIP '05 Multiple parallel-perspective mosaics Multi-disparity stereo: Multi-disparity stereo:  Correspondence for 3D reconstruction Mosaic-based rendering without 3D Mosaic-based rendering without 3D  View selection and rendering

C C V C L Zhu & Hanson ICIP '05 Stereo Mosaics Mosaics with two different oblique angles Mosaics with two different oblique angles  Parallel projection  Adaptive baseline  Uniform depth resolution

C C V C L Zhu & Hanson ICIP '05 Stereo Mosaics Mosaics with two different oblique angles Mosaics with two different oblique angles  Parallel projection  Adaptive baseline  Uniform depth resolution

C C V C L Zhu & Hanson ICIP '05 Advantages Various Oblique Parallel Projections Various Oblique Parallel Projections  Occlusion representation Adaptive Baselines Adaptive Baselines  Uniform depth resolution Large FOV stereo Large FOV stereo  High quality 3D from stereo mosaics  Image-based rendering without 3D

C C V C L Zhu & Hanson ICIP '05 Main Research Issues Real-World Problem Real-World Problem  Uneven, sparse camera “array”  6 DOF motion of camera Two Main Research Issues Two Main Research Issues  Orientation Estimation (see real applications)  GPS, INS, Bundle Adjustment  Parallel Ray Generation  Ray Interpolation - PRISM

C C V C L Zhu & Hanson ICIP '05 PRISM: Ray Interpolation - 1D case Regular Camera with arbitrary orientation Regular Camera with arbitrary orientation

C C V C L Zhu & Hanson ICIP '05 PRISM: Ray Interpolation - 1D case Regular Camera with perspective projection Regular Camera with perspective projection

C C V C L Zhu & Hanson ICIP '05 PRISM: Ray Interpolation - 1D case Cameras with various known orientations Cameras with various known orientations

C C V C L Zhu & Hanson ICIP '05 PRISM: Ray Interpolation - 1D case Existing parallel rays with oblique angle  Existing parallel rays with oblique angle 

C C V C L Zhu & Hanson ICIP '05 PRISM: Ray Interpolation - 1D case Interpolate rays between frames by local match Interpolate rays between frames by local match

C C V C L Zhu & Hanson ICIP '05 PRISM: Ray Interpolation - 1D case … a dense parallel projection with angle  … a dense parallel projection with angle 

C C V C L Zhu & Hanson ICIP '05 Computation & Algorithm: distribute the computations in four steps Geo-Referencing Motion estimation: sparse tie points distributed in entire frames Seamless Mosaicing (PRISM) Process two narrow slices Stereo Matching stereo match only in two mosaics with adaptive baselines 3D Mapping/Visualization just a coordinate transformation and resampling

C C V C L Zhu & Hanson ICIP '05 Real Applications Aerial Video Surveillance Aerial Video Surveillance Under-Vehicle Inspection Under-Vehicle Inspection

C C V C L Zhu & Hanson ICIP '05 Watson Attitude & Heading Reference System Watson Attitude & Heading Reference System Profiling Laser Altimeter Canon XL1 DV camcorders Canon XL1 DV camcorders Duct Tape Instrumentation Package on an Airplane

C C V C L Zhu & Hanson ICIP '05 Objectives Rapidly create large FOV image mosaics that are Rapidly create large FOV image mosaics that are  geo-referenced,  with 3D (stereo) viewing and  with all dynamic targets identified As a light UAV flies over an area As a light UAV flies over an area

C C V C L Zhu & Hanson ICIP '05 Multiple parallel-perspective mosaics Multi-disparity stereo: Multi-disparity stereo:  Correspondence for 3D reconstruction Mosaic-based rendering without 3D Mosaic-based rendering without 3D  View selection and rendering

C C V C L Zhu & Hanson ICIP '05 Virtual Fly-Through Digital City/Campus

C C V C L Zhu & Hanson ICIP '05 Real Applications Aerial Video Surveillance Aerial Video Surveillance Under-Vehicle Inspection Under-Vehicle Inspection

C C V C L Zhu & Hanson ICIP '05 Camera Geometry of the UVIS Car drives over a 1D array of cameras… Car drives over a 1D array of cameras…

C C V C L Zhu & Hanson ICIP '05 X Y 1  32 cameras (images) per column – spatial “scan”  Car moves – temporal “scan”

C C V C L Zhu & Hanson ICIP '05 X Y

C C V C L Zhu & Hanson ICIP '05 X Y 12

C C V C L Zhu & Hanson ICIP '05 X Y 3 1 2

C C V C L Zhu & Hanson ICIP '05 X Y 3124

C C V C L Zhu & Hanson ICIP '05 X Y 31245

C C V C L Zhu & Hanson ICIP '05 X Y

C C V C L Zhu & Hanson ICIP '05 X Y

C C V C L Zhu & Hanson ICIP '05 X Y

C C V C L Zhu & Hanson ICIP '05 X Y

C C V C L Zhu & Hanson ICIP '05 X Y

C C V C L Zhu & Hanson ICIP '05 X Y

C C V C L Zhu & Hanson ICIP '05 X Y

C C V C L Zhu & Hanson ICIP '05 X Y  32 cameras (images) per column – spatial “scan”  48 columns (12 feet)- 4 inch separation in temporal “scan” direction  1536 virtual cameras (images) with 2D scans  Different viewpoints!

C C V C L Zhu & Hanson ICIP '05 UVIS: A Mosaicing example with 2D scan 1D array of 4 cameras ; car moves First pass: mosaicing along the columns (4 cameras) Second pass: mosaicing in the motion direction

C C V C L Zhu & Hanson ICIP '05 One camera to N stereo mosaics Very near range view Very near range view Distortion removal Distortion removal Motion estimation Motion estimation Stereo mosaicing Stereo mosaicing  anomaly detection

C C V C L Zhu & Hanson ICIP '05 UVIS: Dynamic Stereo Mosaics 6 mosaics with changing viewing directions Stereo mosaics: Each pair of 6 mosaics is a stereo pair Dynamic mosaics: Look around objects even without stereo

C C V C L Zhu & Hanson ICIP '05 UVIS: Dynamic Stereo Mosaics 6 mosaics with changing viewing directions Stereo mosaics: Each pair of 6 mosaics is a stereo pair Dynamic mosaics: Look around objects even without stereo

C C V C L Zhu & Hanson ICIP '05 UVIS: Dynamic Stereo Mosaics 6 mosaics with changing viewing directions Stereo mosaics: Each pair of 6 mosaics is a stereo pair Dynamic mosaics: Look around objects even without stereo

C C V C L Zhu & Hanson ICIP '05 UVIS: Dynamic Stereo Mosaics 6 mosaics with changing viewing directions Stereo mosaics: Each pair of 6 mosaics is a stereo pair Dynamic mosaics: Look around objects even without stereo

C C V C L Zhu & Hanson ICIP '05 UVIS: Dynamic Stereo Mosaics 6 mosaics with changing viewing directions Stereo mosaics: Each pair of 6 mosaics is a stereo pair Dynamic mosaics: Look around objects even without stereo

C C V C L Zhu & Hanson ICIP '05 Summary Stereo Mosaics with Oblique Parallel Projections Stereo Mosaics with Oblique Parallel Projections  Wide FOV  Occlusion Rep.  Stereo pairs Research Issues Research Issues  Orientation Estimation  Ray Interpolation ( Interframe match)  3D reconstruction and Moving Target Detection Real Applications Real Applications   Airborne Surveillance   Ground Robot Navigation   Under-Vehicle Inspection

C C V C L Zhu & Hanson ICIP '05 Questions? Zhigang Zhu: Allen Hanson

C C V C L Zhu & Hanson ICIP '05 Real Applications Aerial Video Surveillance Aerial Video Surveillance Ground Mobile Robot Navigation Ground Mobile Robot Navigation Under-Vehicle Inspection Under-Vehicle Inspection

C C V C L Zhu & Hanson ICIP '05 Panoramic Stereo Mosaics from Aerial Video Ideal model: Sensor motion is 1D translation, Nadir view Two “virtual” Pushbroom cameras Sensor Image Plane “Right” Mosaic “Left” Mosaic

C C V C L Zhu & Hanson ICIP '05 Re-Organizing the images…. Stereo pair with large FOVs and adaptive baselines

C C V C L Zhu & Hanson ICIP '05 Re-Organizing the images…. Stereo pair with large FOVs and adaptive baselines

C C V C L Zhu & Hanson ICIP '05 Re-Organizing the images…. Stereo pair with large FOVs and adaptive baselines

C C V C L Zhu & Hanson ICIP '05 Re-Organizing the images…. Stereo pair with large FOVs and adaptive baselines

C C V C L Zhu & Hanson ICIP '05 Re-Organizing the images…. Stereo pair with large FOVs and adaptive baselines

C C V C L Zhu & Hanson ICIP '05 Re-Organizing the images…. Stereo pair with large FOVs and adaptive baselines

C C V C L Zhu & Hanson ICIP '05 Real System Configuration GPS Receiver GPS Antenna Attitude & Heading Reference System Wide Angle Video Zoom Video Navigation Computer Data Acquisition Computer Mini-DV recorders Time code Generator RS-232 Laser Altimeter

C C V C L Zhu & Hanson ICIP '05 Watson Attitude & Heading Reference System Watson Attitude & Heading Reference System Profiling Laser Altimeter Canon XL1 DV camcorders Canon XL1 DV camcorders Duct Tape Instrumentation Package on an Airplane

C C V C L Zhu & Hanson ICIP '05 Stereo mosaics of Amazon rain forest 166-frame telephoto video sequence – by our Aerial Video package 166-frame telephoto video sequence – by our Aerial Video package

C C V C L Zhu & Hanson ICIP '05 Left Mosaic Stereo mosaics of Amazon rain forest 166-frame telephoto video sequence – by our Aerial Video package 166-frame telephoto video sequence – by our Aerial Video package 7056*944 stereo mosaics - by PRISM algorithm 7056*944 stereo mosaics - by PRISM algorithm Subpixel depth reconstruction – by TERREST system Subpixel depth reconstruction – by TERREST system Note the displacements inside the window

C C V C L Zhu & Hanson ICIP '05 Right Mosaic Stereo mosaics of Amazon rain forest 166-frame telephoto video sequence – by our Aerial Video package 166-frame telephoto video sequence – by our Aerial Video package 7056*944 stereo mosaics - by PRISM algorithm 7056*944 stereo mosaics - by PRISM algorithm Subpixel depth reconstruction – by TERREST system Subpixel depth reconstruction – by TERREST system Note the displacements inside the window

C C V C L Zhu & Hanson ICIP '05 Depth Map Stereo mosaics of Amazon rain forest 166-frame telephoto video sequence – by our Aerial Video package 166-frame telephoto video sequence – by our Aerial Video package 7056*944 stereo mosaics - by PRISM algorithm 7056*944 stereo mosaics - by PRISM algorithm Subpixel depth reconstruction – by TERREST system Subpixel depth reconstruction – by TERREST system Note the depths inside the window

C C V C L Zhu & Hanson ICIP '05 Stereo viewing Red: Right view; Blue/Green: Left view Red: Right view; Blue/Green: Left view

C C V C L Zhu & Hanson ICIP '05 END OF ALL SLIDES THAT I COULD USE