Jay Summet -Intel Mentor: Rahul Sukthankar BurningWell Micro-controller based sensor localization and tracking system using projected light patterns GVU Brownbag – September 30 th, 2004

Motivation Displaying an image on an arbitrary surface requires that you know where the surface is within the projection area. Bottom photos courtesy of

Motivating Movie

Previous Methods Manual Calibration – Select control points manually. –Requires user intervention. –Slow. –Possible user error. Camera Based Calibration – Recognize object or detect fiducial marks. –Error Prone.

Our Solution Hardware light sensor embedded in the display surface. Structured light cast from the projector encodes a position. Sensor decodes it's own position. Position can then be broadcast via radio (or used locally).

Closely Related Work Automatic Projector Calibration with Embedded Light Sensors Johnny C. Lee, Paul H. Dietz, Dan Maynes-Aminzade, Ramesh Raskar and Scott Hudson -– To Appear, UIST RFIG Lamps: Interacting with a Self-describing World via Photosensing Wireless Tags and Projectors - Ramesh Raskar, Paul Beardsley, Jeroen van Baar, Yao Wang, Paul Dietz, Johnny Lee, Darren Leigh, Thomas Willwacher -Proceedings of SIGGRAPH No autonomous operation. Locates the sensors with a pure Gray code Disadvantages: May be wrong. Advantages: Always provides a location. No tracking.

Hardware Microchip® rfPIC 12F675 –4 Mhz RISC Processor. –1024 program words. –64 bytes of RAM. –128 bytes of EEPROM. –Built in Analog-to-Digital converter. –UHF ASK/FSK 10dBm radio transmitter. –Photo-transistor based light sensor.

Hardware (cont.) Microchip® ASK receiver module Microchip® 12F629 microprocessor. –Detects/decodes radio signals, converts to RS- 232 Serial signal. Hardware cost: Prototype: Sensor/Transmitter: $40 - Receiver $50 In quantities: Sensor/Transmitter: < $10 - Receiver < $15

Sensor Software Hand coded assembly (not by Russian programmers) bit program words (out of 1024 possible). Tightly integrated with the rfPic 12F675 hardware. –Analog-to-Digital converter. –Interrupt Timer. –Instruction Counter. –UHF Transmitter.

Sensor Software (cont) Sample Light Sensor. –Average of 16 samples. Sync to 60Hz signal. –Find edges. –Sample in the middle of bits. Threshold bit –Store sampled bits in a queue. Search for a valid data packet. –Looking for two different data packet types. –Using FEC error detection & signatures. Transmit data in background. –Interrupt driven while doing the above.

RS-232 Receiver PIC 12F629 –Samples Radio. –Decodes Data Packets. –Transmits via RS-232 at 19,200 bps.

Terms: Frame –Projectors can update what they project 60 times per second. Pattern –One frame of video. Packet –Sequence of bits (one/zero or black/white) spread over multiple frames.

Location Pattern Packets consist of: –Gray Code Patterns –SECDED checksum Patterns 21 bits - 16 data, 5 checksum 2.85 packets a second (3Hz)

Location Packet Check bits supports correction of 1 bit error, or detection of up to 3 bits of error. –Limited in complexity of error detection code by what can be implemented on an 8 bit micro- controller with RAM measured in bytes. –Current SECDED implementation uses ~115 program words and 5 bytes of RAM, including the 3 byte buffer which holds the last 24 bits received. Check bit Location Bit Signature Bit

Hexagonal Tracking Pattern 8 total bits –Inverse replication of 3 data bits to 6 bits plus 2 shared bits of framing. –End of last packet is beginning of next.

Tracking Packets 10 bits are cross checked –Framing bits must be the same (e.g. 11 or 00) and sequential frames must alternate (e.g ). –Check bits must be inverse of data bits. Still not as reliably distinguishable from noise as the location pattern, but much faster (7.5 / second). Frame bits Data Bit Check Bit

Pattern/Packet Generation C program using OpenGL and the GL Utility Toolkit (GLUT). –Allows easy image warping. RS232->USB-> /dev/ttyUSB0 serial interface used for tracking feedback.

Retrace Problem Example

Tracking Strategies Static –Sensor is located where it was last observed. Static + Variable Size –Resize tracking pattern depending upon confidence of sensor location. (Less confidence, larger size.) Linear –Linear model of sensor motion.

Demo Movie

Contributions Sensor able to operate independently. Error detecting code indicates when the location is successfully determined. Tracking of sensor while using rest of the display for output. Source code & hardware design.

Questions? Jay Summet