Supporting Beyond-surface Interaction for Tabletop Systems by Integrating IR Projections Hui-Shan Kao

Outline 2  Introduction  Related Work  System Design  System Calibration  Beyond Surface Implementation  Working with multi-touch  Application  Conclusion

Outline 3  Introduction  Related Work  System Design  System Calibration  Beyond Surface Implementation  Working with multi-touch  Application  Conclusion

Introduction 4  Regular interactive surface only support  Multi-Touch  Tangible input  Extend more possibility on interactive surface  Add another display  Enable intuitive 3D manipulation

Introduction 5  Beyond-Surface Interactions  Base on regular interactive surface  Embedded IR markers Invisible 6DOF of IR camera  pico projector  Tablet PC IR Projector Location & Orientation

Outline 6  Introduction  Related Work  Beyond-surface Interaction  Localization of Device  Invisible Projection  System Design  System Calibration  Beyond Surface Implementation  Working with multi-touch  Application  Conclusion

Beyond-surface Interaction 7  Second Light  Use an electronically switchable diffuser  Turn any translucent sheet above the surface into a mobile display via the second projection  With the camera that sees through the surface, it can localize a mobile panel in six-degrees. Izadi, S.etc Going beyond the display: a surface technology with an electronically switchable diffuser. In Proc UIST’08

Localization of Device 8  Enable 3D interaction on tabletop display  Need to recognize the 6DOF of device  The way to know 6DOF  Magnetic tracker Penlight  Vision based tracker Handheld projector JanusVF Marker based tracker Visible

Invisible projection 9  Invisible projection  Spectrum IR/Color polarization  Time high frequency Synchronization of camera and projector  Encoding in content Embedded code in color channel

Invisible projection 10  Hybrid Infrared and Visible Light Projection for Location Tracking  A projector capable of projecting visible images and infrared images  Using gray-coded pattern to locate the sensors.  Dynamic adaptation of projected imperceptible codes  Using high frequent temporal image modulation to project an invisible pattern Johnny Lee, etc. Hybrid Infrared and Visible Light Projection for Location Tracking In Proc UIST’07 A. Grundh¨ofer, etc.“Dynamic adaptation of projected imperceptible codes,” In Proc. ISMAR ’07

Outline 11  Introduction  Related Work  System Design  Hardware Configuration  Markers System  System Calibration  Beyond Surface Implementation  Working with multi-touch  Application  Conclusion

System Design 12  Goal:  Support multi-touch and multi-user DI based touch detection  Add beyond surface interaction Using invisible marker

Hardware Configuration 13 IR Projection Color Projection IR Projector IR Camera Color Projector Mirror Pico projector + IR Camera Tablet PC + IR Camera

Hardware Design 14  The order of glass layer and diffuse layer  diffuse layer should on top Not to degrade the luminance of pico projector The reflection of pico projector may offending the user

Hardware Configuration 15  Problem: IR rays will be reflected by the touch-glass and resulting in IR spot regions in camera views  Use two cameras to reduce the IR spot

Markers System 16  ARToolkitPlus  Fiducial marker Self-identify  Enable error correct bit  Localization  Camera and projector calibration  Camera pose estimation

Outline 17  Introduction  Related Work  System Design  System Calibration  Tabletop System Calibration  Projector and Camera System Calibration  Beyond Surface Implementation  Working with multi-touch  Application  Conclusion

Tabletop System Calibration 18  Original tabletop system calibration  Finding the homographs between table, camera, and projector  The corners are manually specified by the users Time-consuming Human intervention Pixel-level accuracy

Tabletop System Calibration 19  Only need four points to be manually specified  Adding an additional IR-Color camera  Project predefined markers for calibration IR Camera IR Projector Color Projector IR-Color Camera

Pico projector and camera calibration 20  The projector as an inverse camera  Mapping pixel from a 2D image into 3D rays  Using standard camera calibration procedure  Find the 3D points of the projected pattern and the 2D points of the image projected Rc, Tc Rp, Tp Rcp, Tcp

Outline 21  Introduction  Related Work  System Design  System Calibration  Beyond Surface Implementation  3D posture estimation  Multi-Resolution Markers  Working with multi-touch  Application  Conclusion

Beyond Surface Implementation 22 3D posture?

3D Posture Estimation 23  ARToolKitPlus for 3D estimation  Camera looks within image for markers  Encode identity  Allow recovery of camera pose relative to marker

Multi-Resolution Markers 24  Uni-resolution marker  Camera could observe the markers too small or big The marker with unfit size will not be recognized  Multi-resolution marker  System resizes the IR markers according to the camera’s posture

Marker Split and Merge 25  Marker Split  Not enough : split the markers into smaller size  Marker Merge  Too much : merge the markers for higher accuracy  How to re-arrange the layout of IR makers?  Ensure that camera will see at least 4 markers  Only re-arrange the layout in camera’s view field  The nearest camera will have high priority

3D Posture Estimation 26  To improve the marker detection  Adaptive threshold Non-uniform distribution of IR projection over-exposed of camera  Kalman Filtering smooth the estimation

Outline 27  Introduction  Related Work  System Design  System Calibration  Beyond Surface Implementation  Working with multi-touch  Foreground detection  Background Simulation  Software Synchronization  Application  Conclusion

Working with multi-touch 28  Traditional DI process  Take few frames for building background  subtract the background  obtain the foreground  IR markers projection will interrupt the traditional detection of multi-touch  Foreground can not be recognized

Foreground Detection 29  Define ROI  Simply subtract the background Some markers still remain  Background is not similar to real Simulate background

Marker On and Off 30  Divide the marker into two stages  Marker on for camera positioning  Marker off for finger detection  ROI detection control marker on/off With Marker OffWithout Marker Off ROI

31 IR Projector IR Cameras IR Camera of Mobile Device Simulated Background Observed Image Foregrounds Layout Manager Color Projector ROI Real scene ROI Generation

32 IR Projector IR Cameras IR Camera of Mobile Device Simulated Background Observed Image Foregrounds Layout Manager Color Projector ROI Real scene ROI Generation Background is not similar to real

Background Simulation 33  Off-line process of simulation  Since the layout of markers are predefined  Saving each marker as a patch image and record the position of marker  As the layout re-arrange, the simulating background can be built by the saving patch and their position

34 IR Projector IR Cameras IR Camera of Mobile Device Smoothing Simulated Background Observed Image Foregrounds Tangible Objects Finger Touches Layout Manager Prediction Color Projector ROI Applications Kalman Filtering Real scene Foreground Detection

Capture Image Simulate BG Real BG Software Synchronization 35  The camera and projectors are two independent systems  The simulated background will not synchronize to the capture image  Some of the markers will be treat as foreground

Software Synchronization 36  Keep the simulated backgrounds in a buffer by time  Find the most similar background by subtraction Background Candidate queue Real BG Capture Image BG Candidate Real BG

Outline 37  Introduction  Related Work  System Design  System Calibration  Beyond Surface Implementation  Working with multi-touch  Application  Conclusion

Application 38  3 types of the application provide intuitive and natural manipulation. iLampiFlashiView

iLamp  Goal: Project high-resolution content, bring more detailed and fine-grained information  Combine a Pico projector and an IR camera

iFlashlight  A mobile version of iLamp, can be moved easily.

iView  Tablet PC + IR Camera  An intuitive tool to see 3D content or augmented information of the 2D map from different perspectives.

iView 42  Problem in iView  The user will lose the connection with the surface.  Adding the boundary of surface  Instruct user to manipulate the surface for farther information.

Outline 43  Introduction  Related Work  System Design  System Calibration  Beyond Surface Implementation  Working with multi-touch  Application  Conclusion

Conclusion 44  A new interactive surface based on the programmable invisible markers.  Supporting both on-surface and above-surface interaction for any device outfitted with an IR camera.  Bring another level of information on interactive surface.

45

Bibliography 46  Otmar Hilliges, Shahram Izadi, Andrew D. Wilson,Steve Hodges, Armando Garcia-Mendoza and Andreas Butz, Interaction in the Air: Adding Further Depth to Interactive Tabletops. ACM Symposium on User Interface Software and Technology 2009.

Outline 47  1 Introduction  2 Related Work  2.1 3D interactive Tabletop display  2.2 Invisible projection  2.3 Self-identifying markers for 3D location  2.4 Projector and camera system  3 System Design  3.1 Hardware Configuration  3.2 Markers System  4 System Calibration  4.1 Tabletop System Calibration  4.2 Pico Projector and Camera System Calibration  5 Beyond Surface Implementation  5.1 Dynamic Multi-Resolution Markers  5.2 3D posture estimation  6 Working with multi-touch  6.1 Background Simulation  6.2 Foreground detection  6.3 Software Synchronization  7 Application  7.1 Multi-view map  7.2 Interactive 3D viewer  8 Conclusion

Software Architecture 48 IR Cam Warp Background Subtraction AR Layout Manager Warp Background Simulation Remote Device ARTag Detection Foreground Detection App Warp Display Synchronization Tag Detection Finger Detection

System Implementation 49  4.1 Dynamic Multi-Resolution Markers  4.2 Projector and camera calibration  4.3 3D posture estimation  4.4 Background simulation  4.5 Foreground detection  4.6 Background synchronization

50

51 Color Camera Color Projector Color-IR Camera IR Camera IR Projector User

Enhance Foreground Detection 52

53