Mitsubishi Electric Research Laboratories Raskar May 03 Seamless Multi-Projector Display on Curved Screens Jeroen van Bar, Thomas Willwacher, Srinivas Rao, Ramesh Raskar Mitsubishi Electric Research Labs Cambridge, MA USA

Mitsubishi Electric Research Laboratories Raskar May 03 Curved Screen Displays Multiple overlapping projectors on curved screens Goal : Replace single-proj Higher resoltn/brightness Sub-pixel auto-alignment Parametric solution Low cost infrastructure Markets Planetarium Curved screens

Mitsubishi Electric Research Laboratories Raskar May 03 Dome Projection Techniques Edge-Blended (Tiled/Mosaic) Display Edge-Blended (Tiled/Mosaic) Display Sub-Frames w/Spherical Mapping & Edge-Blends

Mitsubishi Electric Research Laboratories Raskar May Dome Screen

Mitsubishi Electric Research Laboratories Raskar May 03 Outline Seamless Curved Display Multi-projector low cost method Simplified Quadric Image Transfer Calibration with camera-pair Parametric Rendering solution

Mitsubishi Electric Research Laboratories Raskar May 03 Related Work Conventional Displays –Manual alignment, expensive infrastructure –[Jupiter,Trimensions, CAVE, Planetaria, Flight Simulators] Planar Screens –Camera in loop, auto calibration, low cost –Exploit homography parameters –[Raskar98,Surati99,Chen00,Brown02 ……] Curved Screens –Non-parametric solutions –[Jarvis97,Raskar98,Yang01 …] –Parametric –?, Siggraph 2003

Mitsubishi Electric Research Laboratories Raskar May 03 Parametric Approach Advantages –Lower camera resolution –Tolerance for pixel localization errors –Faster calibration –Efficient well-defined warping Avoid look up tables

Mitsubishi Electric Research Laboratories Raskar May 03 Parametric Image Transfer X i j Planar HomographyQuadric Transfer X i j

Mitsubishi Electric Research Laboratories Raskar May 03 Planar projective transfer What is homography ? –Two images of 3D points on a plane are related by a 3x3 matrix M i j Proj 1 Proj 2 j = A 3 x 3 i

Mitsubishi Electric Research Laboratories Raskar May 03 a1a2a3 b1b2b3 c1c2c3 What is homography ? Two images of 3D points on a plane Related by a 3x3 matrix Proj 1 A 3 x 3 Proj 2 i j jxjy1jxjy1 j = A 3 x 3 i ~ = ixiy1ixiy1 k j x = (a i) / (c i) j y = (b i) / (c i)

Mitsubishi Electric Research Laboratories Raskar May 03 Current Multi-Cube System MERL Projector Planar Mosaic Feature Comparison Many hoursTime to installFew minutes $20,000/projector cubeCost$5,000 per projector 15 kg/projector cube + heavy support Weight 3 kg/projector, no fixed support structure Many hoursAlignment 10 seconds ! Needs regular careMaintenanceVery low Small gapsImage quality Seamless !, Subpixel alignment Only high end: command and control, advertising Market Middle segment: Shops, schools, offices, trade shows Planar Displays

Mitsubishi Electric Research Laboratories Raskar May 03 Curved Screens View for a Sweet-spot Projector Sweet spot (Static user)

Mitsubishi Electric Research Laboratories Raskar May 03 Calibration for a Sweet-spot Camera at Sweet spot Projector

Mitsubishi Electric Research Laboratories Raskar May 03 Discretized non-parametric approach Projector Image Camera Image = p1p1 c1c1 c6c6 p6p6 Desired Image Camera at Sweet spot Projector

Mitsubishi Electric Research Laboratories Raskar May 03 Off-Axis Spherical Distortion Ideal Viewpoint Offset Viewpoint

Mitsubishi Electric Research Laboratories Raskar May 03 Fish-eye Projection Planetaria and Digital Dome Theaters Immersive Production Software –Spitz - PolyDome™ –SkySkan - DigiDome™ Immersive Production Software –Spitz - PolyDome™ –SkySkan - DigiDome™

Mitsubishi Electric Research Laboratories Raskar May 03 Outline Seamless Curved Display Multi-projector low cost method Simplified Quadric Image Transfer Calibration with camera-pair Parametric Rendering solution

Mitsubishi Electric Research Laboratories Raskar May 03 Curved projective transfer Quadric classification Projectively equivalent to sphere: Ruled quadrics: hyperboloids of one sheet hyperboloid of two sheets paraboloid sphere ellipsoid Degenerate ruled quadrics: conetwo planes

Mitsubishi Electric Research Laboratories Raskar May 03 Quadrics Q : 4x4 symmetric matrix, For 3D points X on Quadric Nine d.o.f In general 9 points in 3D define quadric X Q

Mitsubishi Electric Research Laboratories Raskar May 03 Quadric Image Transfer Quadratic image transfer function Quadric written as 21 params, 4 more than necessary ! X x x’ [Shashua97]

Mitsubishi Electric Research Laboratories Raskar May 03 Simplified Quadric Image Transfer Our Solution Based on observation.. X x x’ 17 param warp

Mitsubishi Electric Research Laboratories Raskar May 03 Simplified Quadric Image Transfer X x x’ 17 param warp Planar homography: 4 corresponding pixels Quadric transfer: 9 corresponding pixels

Mitsubishi Electric Research Laboratories Raskar May 03 Outline Seamless Curved Display Multi-projector low cost method Simplified Quadric Image Transfer Calibration with camera-pair Parametric Rendering solution

Mitsubishi Electric Research Laboratories Raskar May 03 Calibration of Quadric Screens

Mitsubishi Electric Research Laboratories Raskar May 03 Approach Calibration At each projector i, –Project structured pattern –View with stereo camera –Finding camera to projector quadric transfer, Run-time At each projector i, –Pre-warp input image using

Mitsubishi Electric Research Laboratories Raskar May 03 Low-res Camera 640x480 images But each Projector 1024x768 Calibration Finding relationship between camera and projector

Mitsubishi Electric Research Laboratories Raskar May 03 Non-linear Refinement Linear Estimation Error ~10 pixels NonLinear Refinement Error ~ 1.0 pixels

Mitsubishi Electric Research Laboratories Raskar May 03 Before Blending

Mitsubishi Electric Research Laboratories Raskar May 03 After Blending

Mitsubishi Electric Research Laboratories Raskar May 03 Projector Framebuffers Intensity Correction in Overlap

Mitsubishi Electric Research Laboratories Raskar May 03 Projector Framebuffers

Mitsubishi Electric Research Laboratories Raskar May 03 Projector Framebuffer Intensity Weights

Mitsubishi Electric Research Laboratories Raskar May 03 Outline Seamless Curved Display Multi-projector low cost method Simplified Quadric Image Transfer Calibration with camera-pair Parametric Rendering solution

Mitsubishi Electric Research Laboratories Raskar May 03 Rendering a 3D Scene Steps at each projector (Pre-distort vertex 3D location) For each triangle T with vertices {M j } –For each vertex M Find pixel m via VirtualViewProjection ( M ) Find warped pixel m ’ via quadricTransfer of m Replace M with m ’

Mitsubishi Electric Research Laboratories Raskar May 03 Vertex Shader for Quadric Transfer in Cg vertout main( appin IN, uniform float4x4 modelViewProj, uniform float4 constColor, uniform float3x3 A, uniform float3x3 E, uniform float3 e) { vertout OUT; float4 m1 = float4(IN.position.x, IN.position.y, IN.position.z, 1.0f ); float4 m, mi ;float3 m2,mp;float scale; m = mul( modelViewProj, m1); m2.x = m.x/m.w;m2.y = m.y/m.w;m2.z = 1; scale = mul(m2, mul(E,m2)); mp = mul(A,m2) + sqrt(scale)*e; mi.x = m.w * (mp.x)/(mp.z); mi.y = m.w * (mp.y)/(mp.z); mi.zw = m.zw; OUT.position = mi; OUT.color0 = IN.color0; // Use the original per-vertex color specified return OUT; } Parametric Warp

Mitsubishi Electric Research Laboratories Raskar May 03 Rendering 2D + 3D scene Concave DomeConvex Dome

Mitsubishi Electric Research Laboratories Raskar May 03 Details I Skipped.. Estimating camera and projector params –Internal and External params –Issue with near-planar 3D points Finding pixels weights for blending Non-linear optimization Rendering –Warping and Depth buffer issues

Mitsubishi Electric Research Laboratories Raskar May 03 Seamless Curved Display Multi-projector low cost method Simplified Quadric Image Transfer Complete Parametric calib+rendering solution More info :

Mitsubishi Electric Research Laboratories Raskar May 03 Projector Mailing List subscribe projector Projector bibliography

Mitsubishi Electric Research Laboratories Raskar May 03 Advantages Parametric warp –Lower camera resolution –Tolerance for pixel localization errors –Faster calibration –Efficient well-defined warping