Improving the Speed of Virtual Rear Projection: A GPU-Centric Architecture Matthew Flagg, Jay Summet, James M. Rehg GVU Center College of Computing Georgia Institute of Technology

Matthew Flagg © Ubiquitous Interactive Displays Every flat surface can be an interactive display

Matthew Flagg © VRP: Shadow Elimination Single Projector Case

Matthew Flagg © Half power shadows Shadow Elimination Double Projector Case Passive VRP

Matthew Flagg © Shadow Elimination Boosting projector outputs Proportional feedback law

Matthew Flagg © Occluder Light Suppression Detecting occluded pixels

Matthew Flagg © Detecting occluded pixels Occluder Light Suppression

Matthew Flagg © Detecting occluded pixels Occluder Light Suppression

Matthew Flagg © Detecting occluded pixels Occluder Light Suppression Nonlinear feedback law

Matthew Flagg © Virtual Rear Projection Show ICCV’03 demo video

Matthew Flagg © Challenges for VRP  High image quality  Seams between display regions projected by different projectors  Photometric Uniformity  Fast Compensation  Avoid perception of shadows caused by system lag  Image processing required to ensure high image quality

Matthew Flagg ©  Camera view of screen must be unobstructed  Requires reference image capture before occlusion  Cannot be co-located with projector  Shadows still perceptible  Shadow detection image processing performed on CPU Limitations With Previous Work

Matthew Flagg ©  Detect Occluders, Not Shadows  Co-locate projector with camera  Active IR imaging  Based on work by Tan and Pausch CHI’02  Projector Roles:  Blinding Light Suppressor  Shadow Eliminator  Image Processing on GPU  Pixel Shaders  Render-To-Texture with DirectX9.0 New Approach

Matthew Flagg © New Approach  Detect Occluders, Not Shadows  Co-locate projector with camera  Active IR imaging  Based on work by Tan and Pausch CHI’02  Projector Roles:  Blinding Light Suppressor  Shadow Eliminator  Image Processing on GPU  Pixel Shaders  Render-To-Texture with DirectX9.0 IR backlit camera image

Matthew Flagg ©  Detect Occluders, Not Shadows  Co-locate projector with camera  Active IR imaging  Based on work by Tan and Pausch CHI’02  Projector Roles:  Blinding Light Suppressor  Shadow Eliminator  Image Processing on GPU  Pixel Shaders  Render-To-Texture with DirectX9.0 New Approach Turn off occluded pixels

Matthew Flagg © New Approach  Detect Occluders, Not Shadows  Co-locate projector with camera  Active IR imaging  Based on work by Tan and Pausch CHI’02  Projector Roles:  Blinding Light Suppressor  Shadow Eliminator  Image Processing on GPU  Pixel Shaders  Render-To-Texture with DirectX9.0 Occluder Light Suppression

Matthew Flagg ©  Detect Occluders, Not Shadows  Co-locate projector with camera  Active IR imaging  Based on work by Tan and Pausch CHI’02  Projector Roles:  Shadow Eliminator  Blinding Light Suppressor  Image Processing on GPU  Pixel Shaders  Render-To-Texture with DirectX9.0 New Approach Turn on occluded pixels with second projector

Matthew Flagg © New Approach  Detect Occluders, Not Shadows  Co-locate projector with camera  Active IR imaging  Based on work by Tan and Pausch CHI’02  Projector Roles:  Blinding Light Suppressor  Shadow Eliminator  Image Processing on GPU  Pixel Shaders  Render-To-Texture with DirectX9.0 Shadow Elimination

Matthew Flagg © New Approach  Detect Occluders, Not Shadows  Co-locate projector with camera  Active IR imaging  Based on work by Tan and Pausch CHI’02  Projector Roles:  Blinding Light Suppressor  Shadow Eliminator  Image Processing on GPU  Pixel Shaders  Render-To-Texture with DirectX9.0 Shadow Elimination and Occluder Light Suppression

Matthew Flagg © Fast Compensation: GPU-Centric Approach

Matthew Flagg © Fast Compensation: GPU-Centric Approach 1. Warping, background subtraction 2. Median filtering and dilation for inter-frame tolerance 3. Gaussian blur for blending 4. Compositing and warping

Matthew Flagg © Pixel Shader Pipeline camera texture background texture render texture 1 render texture 2 back buffer display image (A)(B)(C)

Matthew Flagg © Addressing Image Quality LAM for left projector LAM for right projector  Luminance Attenuation Maps (LAMs)  Simple feedback-based approach to accommodate non-linearities of projector- camera  Seam Blending seam – no blending seam – with blending

Matthew Flagg © Virtual Rear Projection Results Play Video

Matthew Flagg © Virtual Rear Projection Results System ComponentLatency Camera Capture to PC-Memory9.09ms PC-Memory to GPU-Memory1.70ms Pixel Shaders2.14ms Projector40.27ms Total Latency53.20ms  Image processing speed increased from 15Hz to 110Hz (camera capture rate), placing limit on the projector (85Hz refresh rate)  Projector latency accounts for 76% of total system latency!  With occluder movement tolerance of 5cm, shadows are imperceptible up to 94 cm/sec (fast walking)

Matthew Flagg © Conclusion  Presented new approach to VRP  Occluder Detection in IR spectrum  All processing moved to GPU  2 System Challenges Met  Display Image Quality  Shadow Perception Avoidance  Shadows eliminated fast enough to accommodate walking

Matthew Flagg © Future Work  Explore hardware solutions  Recent results show an LCD projector having ½ the latency of a DLP and LCOS  User Study  VRP currently used in Collaborative Design Lab in School of Aerospace Engineering  Replicate laboratory evaluation of passive VRP with new active VRP system  Improve Image Quality  Better seam blending