RANSAC-Assisted Display Model Reconstruction for Projective Display Patrick Quirk, Tyler Johnson, Rick Skarbez, Herman Towles, Florian Gyarfas, Henry Fuchs Department of Computer Science University of North Carolina at Chapel Hill Emerging Display Technologies 2006 – March 26, 2006

2 RANSAC-Assisted Display (Surface) Model Reconstruction for Projective Display Multi-Projector Display

3 RANSAC-Assisted Display (Surface) Model Reconstruction for Projective Display Perspectively Correct P

4 RANSAC-Assisted Display (Surface) Model Reconstruction for Projective Display Perspectively-correct Rendering P matrices Intrinsics and extrinsics of each projector Viewer’s location Display Surface Model New Method for Display Surface Estimation!

5 RANSAC-Assisted Display (Surface) Model Reconstruction for Projective Display Rendering Methods Rendering Algorithm 1-pass (homography) 2-pass (projective tex.) Display Shape Planar surfaceComplex shaped surface ReferencesRaskar99,Yang01, Sukthankar01, Chen02, Steele02, Wallace04 Raskar98, Yang-Welch01, Brown02, Cotting04

6 RANSAC-Assisted Display (Surface) Model Reconstruction for Projective Display Display Surface A Primer: 2-Pass Rendering 1st Pass Geometry: application defined Viewpoint: user’s position Result: “ideal image” 2nd Pass Geometry: screen surface Use “ideal image” as projected texture Viewpoint: projector’s position Result: projector’s image Viewer Projector

7 RANSAC-Assisted Display (Surface) Model Reconstruction for Projective Display Display Surface Estimation 3D Stereo Reconstruction Display Surface Model Generation 3D Point Cloud Tessellated Mesh

8 RANSAC-Assisted Display (Surface) Model Reconstruction for Projective Display Artifacts with Tessellated Meshes Inaccurate corner representations Sampling issue Texture mapping distortion on planar surfaces Reconstruction errors Holes in surface model Meshing errors

9 RANSAC-Assisted Display (Surface) Model Reconstruction for Projective Display Visual Artifacts P

10 RANSAC-Assisted Display (Surface) Model Reconstruction for Projective Display Rooms Are Piecewise Planar New display model estimation method Fit planes to point cloud Convert planes to polygons for rendering Advantage Corners can be accurately estimated (intersection of planes) Noiseless models eliminate texture distortion

11 RANSAC-Assisted Display (Surface) Model Reconstruction for Projective Display RANSAC Plane Fitting Random Sample Consensus Designed to work with many outliers Finds largest set of inliers Code from Peter Kovesi, Univ. Western Australia Hypothesis plane from 3 random points Finds plane with most inliers User specified fit tolerance Fits least-squares plane to these inliers

12 RANSAC-Assisted Display (Surface) Model Reconstruction for Projective Display Method FitPlane( C ) Fit one plane P to point cloud C using RANSAC Remove P inlier points from cloud C’ = C – P inliers Recursively loop using outliers from this pass as input for next plane

13 RANSAC-Assisted Display (Surface) Model Reconstruction for Projective Display RANSAC Plane Fitting

14 RANSAC-Assisted Display (Surface) Model Reconstruction for Projective Display Plane To Polygons Task: What planes intersect? Reduce complexity to a 2D problem Establish a “floor” plane orthogonal to the “wall” planes Project wall points onto this plane Casts problem as a line intersection problem

15 RANSAC-Assisted Display (Surface) Model Reconstruction for Projective Display Cast into a 2D Task Intersecting Line (Plane) Pairs Red – Orange Orange – Green Green – Cyan Cyan – Magenta Magenta – Black Black – Blue

16 RANSAC-Assisted Display (Surface) Model Reconstruction for Projective Display Results Less than 2-3% dimensional error Re-projection: Std. Dev. < 0.5 pixels

17 RANSAC-Assisted Display (Surface) Model Reconstruction for Projective Display Results Improved corner matching (< 2 pixels) Lack of texture distortion in application scenery P V

18 RANSAC-Assisted Display (Surface) Model Reconstruction for Projective Display FlightGear Demo PP

19 RANSAC-Assisted Display (Surface) Model Reconstruction for Projective Display Installations UNC Lab 3 walls with columns in each corner UNC lobby and I/ITSEC wall corner Naval Research Lab 3 walls with a column in one corner

20 RANSAC-Assisted Display (Surface) Model Reconstruction for Projective Display SUMMARY: New Surface Model Generation Method Recursively extract planes from 3D point cloud with RANSAC-assisted algorithm Removes outliers easily Fits noisy data well Convert Planes to Polygons for 2-pass rendering Results in simple, accurate model Much less distortion than with tessellated mesh

21 RANSAC-Assisted Display (Surface) Model Reconstruction for Projective Display Future Work Use optimal triangulation algorithm Rather than DLT; less noise in data Generalized Planes to Polygon solution Extract other surface functions Applicability of cylinders, quadrics, etc. Apply techniques to create larger virtual environments Continuous calibration/refinement during operation

22 RANSAC-Assisted Display (Surface) Model Reconstruction for Projective Display Thank You Funding Office of Naval Research Award N DARWARS Training Superiority programDARWARS Training Superiority program VIRTE – Virtual Technologies and Environments programVIRTE – Virtual Technologies and Environments program

23 RANSAC-Assisted Display (Surface) Model Reconstruction for Projective Display

24 RANSAC-Assisted Display (Surface) Model Reconstruction for Projective Display Planes to Polygons 1.Compute intersection line between two planes Align this with Y axis 2.Project down all points to get floorplan Line segments, based on range of inliers 3.Find closest 2D line intersections Determines which planes to intersect 4.Intersect the planes, make height uniform 5.Truncate non-intersecting planes (edges)

25 RANSAC-Assisted Display (Surface) Model Reconstruction for Projective Display Room Coordinate System Motivation Can easily estimate position of viewer Auto-alignment performed Incorporated into Planes to Polygons algorithm Chosen to be intersection of planes Also have tool to change coordinate system Extends to a tracked user

26 RANSAC-Assisted Display (Surface) Model Reconstruction for Projective Display 3D Stereo Reconstruction 1.Display feature set with projector(s) Checkerboard, Gray-coded stripes, Gaussian blobs, etc. 2.Image with stereo camera pair 3.Extract, identify, match feature points 4.Triangulate corresponding points Model represented by 3D point cloud!

27 RANSAC-Assisted Display (Surface) Model Reconstruction for Projective Display Plane 1

28 RANSAC-Assisted Display (Surface) Model Reconstruction for Projective Display Plane 2

29 RANSAC-Assisted Display (Surface) Model Reconstruction for Projective Display Plane 3

30 RANSAC-Assisted Display (Surface) Model Reconstruction for Projective Display Plane 4