Auto-Calibration of Multi-Projector Display Walls Andrew Raij (raij@cise.ufl.edu) UNC-Chapel Hill University of Florida Marc Pollefeys (marc@cs.unc.edu) UNC-Chapel Hill 12/1/2018 ICPR 2004

Multi-Projector Display Projector misalignment The tedious solution -- align by hand Many automatic camera-based methods [Raskar Viz1999, Chen Vis2000, Yang Viz2001, Raij Procams2003, …] ICPR 2004

Multi-Projector Display cont’d Undefined display area ICPR 2004

Related Work – Display Area Rectangle on Screen Require manual placement and / or interaction to achieve desired display area properties Yang Vis2001, Rehg CARV2002, Raij Procams2003, Ashdown CVPR2004 Raskar CVPR2001, Raskar SIGGRAPH2003 – Requires tilt sensors and rigidly attached projector-camera pairs Okatani ICCV2003 – Requires calibrated projectors ICPR 2004

Main Contribution We propose a fully automatic method for defining the display area of a planar multi-projector display Single camera No fiducials, interaction or additional sensors Uncalibrated projectors Note planar screen makes auto-calibration harder ICPR 2004

Our Testbed: PixelFlex2 Planar, casually-aligned, multi-projector display system developed at UNC-Chapel Hill 8 projectors in 4x2 configuration, roughly 12' x 6' image Calibration camera sees all projectors Original system uses 4 fiducials to define display area Note: without photometric correction ICPR 2004

Outline of our Approach Find camera-projector homographies Projector calibration with planar auto-calibration (TriggsECCV98) Reconstruct calibration camera, projectors and display plane. Define world-aligned frame in plane and choose display area. ICPR 2004

Step 1: Camera-Projector Homographies 2 5 1 6 7 4 8 3 Features Hcp 3.06 4.39e-02 -4.64e+02 -3.39e-02 2.76 -1.49e+03 1.69e-05 -2.17e-04 1.0 Proj ID (X, Y) 1 (185.5, 113.5) 2 (347.5, 113.5) … (…, …) Camera Proj ID (X, Y) 1 (196.70, 578.36) 2 (243.29, 579.07) … (…, …) 3.14 -1.06e-01 -1.79e+03 2.16e-03 2.83 -1.56e+03 4.55e-05 -2.34e-04 1.0 ICPR 2004

Step 2: Projector Auto-Calibration Projector ≡ Camera The image of the circular points must lie on the image of the absolute conic (IAC) in all views Ω∞ X π∞ πd xc ωc ωp Hcpxc Hcp C xcTωcxc = (Hcpxc)Tωp(Hcpxc) 0, p=1…n ω K-TK-1 P ICPR 2004

Projector Auto-Calibration cont’d n projectors (n ≥ 3) 2n+2 constraints n+5 unknowns Camera is calibrated - ωc is known Camera image of circ. points xc unknown Projector - fpi, yp unknown pixels are square principal point same in all projectors, offset vertically Note: Separate offset yields 2n+4 unknowns > 2n+2 Levenberg-Marquardt least squares minimization of xcTωcxc = (Hcpxc)Tωp(Hcpxc) 0, p=1…n (0, yp) n ||xcTωcxc ||2 + ∑||(Hcpxc)Tωp(Hcpxc)||2 p=1 ICPR 2004

Initialization to Minimization For each hypothesis on Kp and for each projector π1 π2 SVD-based Pose Estimation for Planar Scenes (TriggsECCV98) Projector Kp-1HcpKc Camera C xc1 xc2 ||xcTωcxc ||2 + ∑||(Hcpxc)Tωp(Hcpxc)||2 n p=1 ICPR 2004

Initialization (cont’d) Clear minima produced by initialization algorithm ICPR 2004

Estimated Projector Intrinsics Geometric Configuration Proj 1 2267.81 2173.30 2149.39 2095.71 2 2200.50 2206.80 2171.94 2154.88 3 2148.76 2223.58 2122.13 2133.44 4 2229.53 2301.02 2165.01 2156.52 5 2219.13 2181.43 2172.84 2138.11 6 2199.33 2194.60 2148.33 2128.62 7 2253.56 2258.69 2191.39 2230.51 8 2250.98 2267.79 2177.96 2152.01 YP 645.58 668.42 682.83 704.58 Intrinsics are consistent for different geometric configurations ICPR 2004

Step 3: Reconstruction K1-1Hc1Kc SVD Kn-1HcnKc π1 π2 K1-1Hc1Kc C P1 SVD π1 Kn-1HcnKc π2 C Pn Camera-projector baseline is normalized to 1 so n reconstructions are in different frames, up to scale Merge by normalizing by distance from camera to plane ICPR 2004

Step 4: Display Area Selection Display area is a world-aligned rectangle in the plane of desired aspect ratio Vertical = Horizontal V Plane Normal Assume camera x-axis is horizontal (i.e. allows tilting) and perform orthogonal projection on plane Can choose rectangle of desired aspect ratio since reconstruction up to scale ICPR 2004

Results Camera, projector array, and plane reconstruction. 16:9 max inscribed area viewport shown in yellow. Camera image with dotted lines extending towards vanishing points. Vanishing points found by projection of world direction basis into camera. ICPR 2004

Conclusions Contributions Automatic estimation of intrinsics of array of projectors projecting on a plane Automatic estimation of projector extrinsics and display plane Automatic selection of world-aligned viewport of proper aspect ratio In sum, fully automatic calibration of a planar multi-projector /camera system without physical calibration objects and/or interaction. ICPR 2004

Conclusions cont’d Discussion Future Work Possible even though only scene observed by the camera and projectors is a plane! “Metric" calibration is not highly accurate (intrinsics, etc.), but this is ok for the application. Perceptually, display area will appear to be a properly-aligned rectangle. Future Work Precise evaluation of calibration accuracy Improve calibration with max likelihood estimation based on bundle-adjustment. Calibrate for camera intrinsics, radial distortion in both camera and projectors. ICPR 2004

Acknowledgements This work was partially supported by: NSF Career award IIS 0237533 Department of Energy ASC VIEWS Program B519834 DARPA DARWARS program ONR N00014-03-1-0589. The UNC PixelFlex Display Research Team Henry Fuchs Herman Towles Chris Ashworth Sundeep Tirumalareddy ICPR 2004