Mitsubishi Electric Research Labs Raskar, vanBaar, Willwacher, Rao Quadric Curved Screens Quadric Transfer for Immersive Curved Screen Displays Ramesh Raskar, Jeroen van Bar, Thomas Willwacher, Srinivas Rao Mitsubishi Electric Research Labs Cambridge, MA USA

Mitsubishi Electric Research Labs Raskar, vanBaar, Willwacher, Rao Quadric Curved Screens Curved Screen Displays Multiple overlapping projectors Goal : Replace single-proj Higher resoltn Higher brightness Sub-pixel auto-alignment Parametric solution Planetarium Simulators

Mitsubishi Electric Research Labs Raskar, vanBaar, Willwacher, Rao Quadric Curved Screens Dome Projection Edge-Blended Display Edge-Blended Display Sub-Frames Spherical Warping

Mitsubishi Electric Research Labs Raskar, vanBaar, Willwacher, Rao Quadric Curved Screens

Mitsubishi Electric Research Labs Raskar, vanBaar, Willwacher, Rao Quadric Curved Screens Outline Seamless Curved Display Multi-projector methods Simplified Quadric Image Transfer Calibration with camera-pair Parametric Rendering solution

Mitsubishi Electric Research Labs Raskar, vanBaar, Willwacher, Rao Quadric Curved Screens Outline Seamless Curved Display Multi-projector methods Simplified Quadric Image Transfer Calibration with camera-pair Parametric Rendering solution

Mitsubishi Electric Research Labs Raskar, vanBaar, Willwacher, Rao Quadric Curved Screens Multi-Projector Displays TraditionalAuto Alignment Non- Parametric Parametric Head- tracked Planar DataWalls, Barco, Jupiter Surati99 Raskar99, Chen00, Brown02 Curved Flight simulators, Domes, SEOS, Barco, Jarvis97, Raskar98, Yang01 ?? Precise config = Costly setup Manual alignment = High maintenance

Mitsubishi Electric Research Labs Raskar, vanBaar, Willwacher, Rao Quadric Curved Screens Multi-Projector Displays ManualAuto Alignment Non- Parametric Parametric Head- tracked Planar DataWalls, Barco, Jupiter Surati99 Raskar99, Chen00, Brown02 Curved Flight simulators, Domes, SEOS, Barco, Jarvis97, Raskar98, Yang01 ??

Mitsubishi Electric Research Labs Raskar, vanBaar, Willwacher, Rao Quadric Curved Screens Multi-Projector Displays ManualAuto Alignment Non- Parametric Parametric Head- tracked Planar DataWalls, Barco, Jupiter Surati99 Raskar99, Chen00, Brown02 Curved Flight simulators, Domes, SEOS, Barco, Jarvis97, Raskar98, Yang01 ??

Mitsubishi Electric Research Labs Raskar, vanBaar, Willwacher, Rao Quadric Curved Screens Multi-Projector Displays ManualAuto Alignment Non- Parametric Parametric Head- tracked Planar DataWalls, Barco, Jupiter Surati99 Raskar99, Chen00, Brown02 Curved Flight simulators, Domes, SEOS, Barco, Jarvis97, Raskar98, Yang01 ??

Mitsubishi Electric Research Labs Raskar, vanBaar, Willwacher, Rao Quadric Curved Screens Planar Display with parametric approach Seconds R Raskar Camera

Mitsubishi Electric Research Labs Raskar, vanBaar, Willwacher, Rao Quadric Curved Screens Planar projective transfer: homography –Two images of 3D points on a plane –Related by a 3x3 matrix M x x’ Proj 1 Proj 2 x’ = A 3 x 3 x ~ Proj 1Proj 2 x x’ A 3 x 3

Mitsubishi Electric Research Labs Raskar, vanBaar, Willwacher, Rao Quadric Curved Screens Parametric Image Transfer X x x’ Planar HomographyQuadric Transfer X x x’x’

Mitsubishi Electric Research Labs Raskar, vanBaar, Willwacher, Rao Quadric Curved Screens Parametric Approach Calibration –Lower camera resolution –Tolerance for pixel localization errors –Faster calibration Rendering –Efficient well-defined warping –Avoids look up tables

Mitsubishi Electric Research Labs Raskar, vanBaar, Willwacher, Rao Quadric Curved Screens Multi-Projector Displays ManualAuto Alignment Non- Parametric Parametric Head- tracked Planar DataWalls, Barco, Jupiter Surati99 Raskar99, Chen00, Brown02 Curved Flight simulators, Domes, SEOS, Barco, Jarvis97, Raskar98, Yang01 ??

Mitsubishi Electric Research Labs Raskar, vanBaar, Willwacher, Rao Quadric Curved Screens Curved Screens View for a Sweet-spot Projector Sweet spot

Mitsubishi Electric Research Labs Raskar, vanBaar, Willwacher, Rao Quadric Curved Screens Calibration for a Sweet-spot Camera at Sweet spot Projector

Mitsubishi Electric Research Labs Raskar, vanBaar, Willwacher, Rao Quadric Curved Screens Discretized non-parametric approach Projector Image p1p1 c1c1 c6c6 p6p6 Camera Image = Desired Image

Mitsubishi Electric Research Labs Raskar, vanBaar, Willwacher, Rao Quadric Curved Screens Discretized Warping Software Spitz - PolyDome™ Discretized Warping Software Spitz - PolyDome™ Pre-Warped Projection

Mitsubishi Electric Research Labs Raskar, vanBaar, Willwacher, Rao Quadric Curved Screens Outline Seamless Curved Display Multi-projector methods Simplified Quadric Image Transfer Calibration with camera-pair Parametric Rendering solution

Mitsubishi Electric Research Labs Raskar, vanBaar, Willwacher, Rao Quadric Curved Screens Quadric classification Projectively equivalent to sphere: Ruled quadrics: hyperboloids of one sheet hyperboloid paraboloid sphere ellipsoid Degenerate ruled quadrics: conetwo planes

Mitsubishi Electric Research Labs Raskar, vanBaar, Willwacher, Rao Quadric Curved Screens Quadrics 4x4 symmetric matrix, Nine d.o.f 9 points in 3D define quadric X Q

Mitsubishi Electric Research Labs Raskar, vanBaar, Willwacher, Rao Quadric Curved Screens Quadric Image Transfer If 21 params, 4 more than necessary ! X x x’ [Shashua97],

Mitsubishi Electric Research Labs Raskar, vanBaar, Willwacher, Rao Quadric Curved Screens Simplified Quadric Image Transfer Based on.. X x x’ 17 param warp Homography with polar plane Projected conic

Mitsubishi Electric Research Labs Raskar, vanBaar, Willwacher, Rao Quadric Curved Screens Simplified Quadric Image Transfer X x x’ 17 param warp Planar homography: 4+ corresponding pixels Quadric transfer: 9+ corresponding pixels

Mitsubishi Electric Research Labs Raskar, vanBaar, Willwacher, Rao Quadric Curved Screens Outline Seamless Curved Display Multi-projector low cost method Simplified Quadric Image Transfer Calibration with camera-pair Parametric Rendering solution

Mitsubishi Electric Research Labs Raskar, vanBaar, Willwacher, Rao Quadric Curved Screens Dome Screen Calibration of Quadric Screens

Mitsubishi Electric Research Labs Raskar, vanBaar, Willwacher, Rao Quadric Curved Screens Approach Calibration At each projector i, –Project structured pattern –View with stereo camera –Finding camera to projector quadric transfer, Run-time At each projector i, –Pre-warp input image using

Mitsubishi Electric Research Labs Raskar, vanBaar, Willwacher, Rao Quadric Curved Screens Low-res Camera 640x480 images But each Projector 1024x768 Calibration Finding relationship between camera and projector

Mitsubishi Electric Research Labs Raskar, vanBaar, Willwacher, Rao Quadric Curved Screens Before Blending

Mitsubishi Electric Research Labs Raskar, vanBaar, Willwacher, Rao Quadric Curved Screens After Blending

Mitsubishi Electric Research Labs Raskar, vanBaar, Willwacher, Rao Quadric Curved Screens Projector Framebuffers Intensity Correction in Overlap

Mitsubishi Electric Research Labs Raskar, vanBaar, Willwacher, Rao Quadric Curved Screens Projector Framebuffers

Mitsubishi Electric Research Labs Raskar, vanBaar, Willwacher, Rao Quadric Curved Screens Projector Framebuffer Intensity Weights

Mitsubishi Electric Research Labs Raskar, vanBaar, Willwacher, Rao Quadric Curved Screens Outline Seamless Curved Display Multi-projector low cost method Simplified Quadric Image Transfer Calibration with camera-pair Parametric Rendering solution

Mitsubishi Electric Research Labs Raskar, vanBaar, Willwacher, Rao Quadric Curved Screens Rendering a 3D Scene Steps at each projector (Pre-distort vertex 3D location) For each triangle T with vertices {M j } –For each vertex M Find pixel m via VirtualViewProjection ( M ) Find warped pixel m ’ via quadricTransfer of m Replace M with m ’

Mitsubishi Electric Research Labs Raskar, vanBaar, Willwacher, Rao Quadric Curved Screens Vertex Shader for Quadric Transfer in Cg Parametric Warp vertout main( appin IN, uniform float4x4 modelViewProj, uniform float4 constColor, uniform float3x3 A, uniform float3x3 E, uniform float3 e) { vertout OUT; float4 m1 = float4(IN.position.x, IN.position.y, IN.position.z, 1.0f ); float4 m, mi ;float3 m2,mp;float scale; m = mul( modelViewProj, m1); m2.x = m.x/m.w;m2.y = m.y/m.w;m2.z = 1; scale = mul(m2, mul(E,m2)); mp = mul(A,m2) + sqrt(scale)*e; mi.x = m.w * (mp.x)/(mp.z); mi.y = m.w * (mp.y)/(mp.z); mi.zw = m.zw; OUT.position = mi; OUT.color0 = IN.color0; // Use the original per-vertex color specified return OUT; }

Mitsubishi Electric Research Labs Raskar, vanBaar, Willwacher, Rao Quadric Curved Screens 3D Terrain Rendering

Mitsubishi Electric Research Labs Raskar, vanBaar, Willwacher, Rao Quadric Curved Screens Parametric Rendering Benefits Head tracking support –Update quadric transfer per frame Single pass rendering –Avoid post-rasterized warp –Efficient rendering –Better image quality –Programmable hardware Distributed rendering –Runs unmodified 3D applications

Mitsubishi Electric Research Labs Raskar, vanBaar, Willwacher, Rao Quadric Curved Screens Head Tracked Single Pass Rendering

Mitsubishi Electric Research Labs Raskar, vanBaar, Willwacher, Rao Quadric Curved Screens Distributed Rendering with Unmodified Application

Mitsubishi Electric Research Labs Raskar, vanBaar, Willwacher, Rao Quadric Curved Screens Subpixel Accurate Registration

Mitsubishi Electric Research Labs Raskar, vanBaar, Willwacher, Rao Quadric Curved Screens Convex Dome

Mitsubishi Electric Research Labs Raskar, vanBaar, Willwacher, Rao Quadric Curved Screens Acknowledgements Mitsubishi Electric Research Labs –Paul Beardsley, Jay Thornton –Joe Marks Mitsubishi Electric, Japan –Masato Ogata, Hiroyuki Wada –Masatoshi Kameyama, Ashizaki

Mitsubishi Electric Research Labs Raskar, vanBaar, Willwacher, Rao Quadric Curved Screens Seamless Curved Display Multi-projector low cost method Simplified Quadric Image Transfer Complete Parametric calib+rendering solution Head tracking support, single pass rendering

Mitsubishi Electric Research Labs Raskar, vanBaar, Willwacher, Rao Quadric Curved Screens Details I Skipped.. Photometric Correction [Majumder03] Depth of field is limited Estimating camera and projector params –Internal and External params –Issue with near-planar 3D points Finding pixels weights for blending Non-linear optimization Rendering –Warping and Depth buffer issues

Mitsubishi Electric Research Labs Raskar, vanBaar, Willwacher, Rao Quadric Curved Screens Advantages Parametric warp –Lower camera resolution –Tolerance for pixel localization errors –Faster calibration –Efficient well-defined warping

Mitsubishi Electric Research Labs Raskar, vanBaar, Willwacher, Rao Quadric Curved Screens a1a2a3 b1b2b3 c1c2c3 What is homography ? Proj 1 A 3 x 3 Proj 2 i j jxjy1jxjy1 j = A 3 x 3 i ~ = ixiy1ixiy1 k j x = (a i) / (c i) j y = (b i) / (c i) ~ ~

Mitsubishi Electric Research Labs Raskar, vanBaar, Willwacher, Rao Quadric Curved Screens Rendering 2D + 3D scene Concave DomeConvex Dome

Mitsubishi Electric Research Labs Raskar, vanBaar, Willwacher, Rao Quadric Curved Screens Projector Mailing List subscribe projector Projector bibliography

Mitsubishi Electric Research Labs Raskar, vanBaar, Willwacher, Rao Quadric Curved Screens Vertex Shader for Quadric Transfer in Cg Parametric Warp vertout main( appin IN, uniform float4x4 modelViewProj, uniform float4 constColor, uniform float3x3 A, uniform float3x3 E, uniform float3 e) { vertout OUT; float4 m1 = float4(IN.position.x, IN.position.y, IN.position.z, 1.0f ); float4 m, mi ;float3 m2,mp;float scale; m = mul( modelViewProj, m1); m2.x = m.x/m.w;m2.y = m.y/m.w;m2.z = 1; scale = mul(m2, mul(E,m2)); mp = mul(A,m2) + sqrt(scale)*e; mi.x = m.w * (mp.x)/(mp.z); mi.y = m.w * (mp.y)/(mp.z); mi.zw = m.zw; OUT.position = mi; OUT.color0 = IN.color0; // Use the original per-vertex color specified return OUT; }

Mitsubishi Electric Research Labs Raskar, vanBaar, Willwacher, Rao Quadric Curved Screens Current Multi-Cube System MERL Projector Planar Mosaic Feature Comparison Many hoursTime to installFew minutes $20,000/projector cubeCost$5,000 per projector 15 kg/projector cube + heavy support Weight 3 kg/projector, no fixed support structure Many hoursAlignment 10 seconds ! Needs regular careMaintenanceVery low Small gapsImage quality Seamless !, Subpixel alignment Only high end: command and control, advertising Market Middle segment: Shops, schools, offices, trade shows Planar Displays

Mitsubishi Electric Research Labs Raskar, vanBaar, Willwacher, Rao Quadric Curved Screens Ideal Sweetspot Offset Sweetspot

Mitsubishi Electric Research Labs Raskar, vanBaar, Willwacher, Rao Quadric Curved Screens Quadric Transfer Convex or concave quadric surfaces Minutes rather than seconds due to necessary non-linear optimization of quadric transfer parameters Sub-pixel accuracy Current consumer-class graphics hardware allows fast rendering Convex Concave

Mitsubishi Electric Research Labs Raskar, vanBaar, Willwacher, Rao Quadric Curved Screens Non-linear Refinement Linear Estimation Error ~10 pixels NonLinear Refinement Error ~ 1.0 pixels

Mitsubishi Electric Research Labs Raskar, vanBaar, Willwacher, Rao Quadric Curved Screens Traditional Multi-Projector Setup Well-defined OverlapNo Overlap Precise configuration = Costly setup Manual alignment = High maintenance

Mitsubishi Electric Research Labs Raskar, vanBaar, Willwacher, Rao Quadric Curved Screens Related Work Conventional Displays –Manual alignment, expensive infrastructure –[Jupiter,Trimensions, CAVE, Planetaria, Flight Simulators] Planar Screens –Camera in loop, auto calibration, low cost –Exploit homography parameters –[Raskar98,Surati99,Chen00,Brown02 ……] Curved Screens –Non-parametric solutions –[Jarvis97,Raskar98,Yang01 …] –Parametric –?, Siggraph 2003