Hybrid Infrared and Visible Light Projection for Location Tracking Johnny Lee, Scott Hudson, Paul Dietz Carnegie Mellon University Mitsubishi Electric Research Labs UIST 2007 – Newport, RI

Hybrid Projection one projector infrared visible But, why?

Projector-Based Location Discovery [Lee, UIST’04] Light sensors Projector Electronics & computer

Projector-Based Location Discovery calibration free: - no computer vision - no alignment - no manual input Scalable and robust

Moveable Surfaces calibration free no external tracker [Lee, UIST 2005] calibration free no external tracker - interactive content

Drawbacks Location Discovery [‘04] Moveable Surfaces [‘05] Incremental tracking Caustic B&W patterns Momentary movement

Ideally Full-screen Application Content Full-screen Location Patterns

Hybrid Projection one projector infrared visible for the computer for the human

Infrared & Visible Projection If we could make a projector the can emit IR images Could do full screen location discovery without the viewer’s knowledge Simultaneously project visible light application content. Chose green for infrared because that is how it is rendered in night-vision cameras.

Infrared & Visible Projection

Infrared & Visible Projection

Infrared & Visible Projection

Infrared & Visible Projection

Infrared & Visible Projection

Infrared & Visible Projection

Infrared & Visible Projection

Infrared & Visible Projection

Infrared & Visible Projection

How?

Light Source: Lamps Xenon Arc Lamp Lots of IR energy, sometimes 100 times But this is energy we can’t see so manufactures use IR cutoff filters Reduces heat and wear on the imaging element Even if the optics were changed Unmodulated light is hard to use (easy interference) Xenon Arc Lamp

Light Technologies You are probably familiar with these, but LED stands for light emitting diode Available in a wide variety of form factors, and power, and Most importantly color.

More Efficient and Better Lifespan The benefit of using an LED light source Lower heat Lower power Smaller – most of a projector is power supply and cooling Handheld projector becoming possible. Products emerging on market this year, 2007

IR and Visible Light LEDs 1000 LEDs/mm2 This is our LED light source. Notice only two color bands. Since I’m building these by hand, Don’t have manufacturing capability to create quad color LED arrays that are dense enough. But a commercial manufacturer could easily do this. So, for our proof-of-concept prototype we use visible red light and invisible infrared. IR and Visible Light LEDs 1000 LEDs/mm2 University of Strathclyde, Institute of Photonics

LED Array DMD Lens Projection optics Inside the DLP projector LED array Culminating lens which focuses light onto the DLP chip DLP reflect light into the barrel of projection optics. Projection optics

Our Dev Kit: 180 binary images/s High-Speed Dev Kit: 16,500 binary images/s Production Unit: +50,000 binary images/s We use a DLP Discovery development kit to control the mirror state on the chip. 1024x768 area = 20 binary images 60Hz tracking = 2.4% duty cycle of production DMD Required changes to commercial designs would be minimal.

Demo of Capability Uses a second projector for visible content

Inherent Multi-Stylus Tracking Inherent stylus support on any surface Multi simultaneous users with user ID Possible with a minor design change to upcoming generation of DLP projectors.

Non-Planar and Discontinuous Surfaces Inherent stylus support on any surface Multi simultaneous users with user ID Possible with a minor design change to upcoming generation of DLP projectors.

Static IR Patterns

Concept Applications Simulated using external tracking (calibration)

Hand-held projection with photosensitive tags In siggraph 2004, explored interaction of handheld projection Simulated using camera-based tracking, but enabled by IR projector-based tracking Selection Grouping Annotating [Siggraph 2004]

Foldable Interactive Displays [submitted to CHI]

Acknowledgements Johnny Chung Lee Funded in part by the National Science Foundation under grants IIS-0121560 and IIS-0325351 Funded in part by Mitsubishi Electric Research Labs Johnny Chung Lee johnny@cs.cmu.edu

Other ways to make invisible patterns Other non-visible wavelengths Steganography Color shifting Noise encoding Bit Timing Synchronization may be difficult

Camera-Based Tracking Requires calibration Requires markers for segmentation IR sensors + transmitter is less power than 4 IR LEDs Does not provide ID Limitation on the number of points Limitation on tracking rate Limitation on scene/target complexity Resolution is not as scalable Less optically robust Optical path geometry and variable illumination

Projector vs Camera Tracking Sensors provide point ID Independent of scene/surface complexity

Space-Labeling Projectors 11 Infrared LED slide projectors Potentially Low-cost Per axis: 500Hz tracking at 10-bits stacked of 11 LED slide projectors Requires cylindrical optics Can do tracking outdoors in complex environments And in dynamic lighting Tracking could be boosted to 25KHz. Outdoor motion tracking

Binary Gray 0000 0001 0010 0011 0100 0101 0110 0111 1000 1001 1010 1011 1100 1101 1110 1111 0000 0001 0011 0010 0110 0111 0101 0100 1100 1101 1111 1110 1010 1011 1001 1000

Binary Gray 0000 0001 0010 0011 0100 0101 0110 0111 1000 1001 1010 1011 1100 1101 1110 1111 0000 0001 0011 0010 0110 0111 0101 0100 1100 1101 1111 1110 1010 1011 1001 1000

Binary Gray

Binary Gray

Binary Gray

Binary Gray

Binary Gray

Binary Gray

Binary Gray

Binary Gray

Binary Gray

Binary Gray

Binary Gray

Binary Gray

Pixart Chip 1024x768 resolution 100Hz tracking 4 points Only 4 points High power LED points