1. Mitsubishi Electric Research Labs Raskar, Beardsley, vanBaar, Wang, Dietz, Lee, Leigh, Willwacher R F I G Lamps Ramesh Raskar, Paul Beardsley, Jeroen van Baar, Yao Wang, Paul Dietz, Johnny Lee, Darren Leigh, Thomas Willwacher Mitsubishi Electric Research Labs (MERL), Cambridge, MA R F I G Lamps : Interacting with a Self-describing World via Photosensing Wireless Tags and Projectors

2. Mitsubishi Electric Research Labs Raskar, Beardsley, vanBaar, Wang, Dietz, Lee, Leigh, Willwacher R F I G Lamps Radio Frequency Identification Tags (RFID) microchip Antenna No batteries, Small size, Cost few cents

3. Mitsubishi Electric Research Labs Raskar, Beardsley, vanBaar, Wang, Dietz, Lee, Leigh, Willwacher R F I G Lamps Warehousing Routing Library Baggage handling Currency Livestock tracking

4. Mitsubishi Electric Research Labs Raskar, Beardsley, vanBaar, Wang, Dietz, Lee, Leigh, Willwacher R F I G Lamps Conventional Passive RFID

5. Mitsubishi Electric Research Labs Raskar, Beardsley, vanBaar, Wang, Dietz, Lee, Leigh, Willwacher R F I G Lamps Tagged Books in a Library Id List of books in RF range No Precise Location Data Are books in sorted order ? Which book is upside down ?

6. Mitsubishi Electric Research Labs Raskar, Beardsley, vanBaar, Wang, Dietz, Lee, Leigh, Willwacher R F I G Lamps Where are boxes with Products close to Expiry Date ?

7. Mitsubishi Electric Research Labs Raskar, Beardsley, vanBaar, Wang, Dietz, Lee, Leigh, Willwacher R F I G Lamps Conventional RF tag Photo-sensing RF tag

8. Mitsubishi Electric Research Labs Raskar, Beardsley, vanBaar, Wang, Dietz, Lee, Leigh, Willwacher R F I G Lamps Photosensor ? Compatible with RFID size and power needs Projector ? Directional transfer, AR with Image overlay

9. Mitsubishi Electric Research Labs Raskar, Beardsley, vanBaar, Wang, Dietz, Lee, Leigh, Willwacher R F I G Lamps b. Tags respond with date and precise (x,y) pixel location, Projector beams Ok at that location a. Photosensing RFID tags are queried via RF c. Multiple users can simultaneously work from a distance without RF collision

10. Mitsubishi Electric Research Labs Raskar, Beardsley, vanBaar, Wang, Dietz, Lee, Leigh, Willwacher R F I G Lamps

11. Mitsubishi Electric Research Labs Raskar, Beardsley, vanBaar, Wang, Dietz, Lee, Leigh, Willwacher R F I G Lamps RFID (Radio Frequency Identification) RFIG (Radio Frequency Id and Geometry)

12. Mitsubishi Electric Research Labs Raskar, Beardsley, vanBaar, Wang, Dietz, Lee, Leigh, Willwacher R F I G Lamps Prototype Tag RF tag + photosensor

13. Mitsubishi Electric Research Labs Raskar, Beardsley, vanBaar, Wang, Dietz, Lee, Leigh, Willwacher R F I G Lamps Outline Photosensing RF tags –Location sensing Geometric Operations –Multi-view analysis –Deformation Interactive Projection –Mouse-like interaction –Image stabilization

14. Mitsubishi Electric Research Labs Raskar, Beardsley, vanBaar, Wang, Dietz, Lee, Leigh, Willwacher R F I G Lamps Outline Photosensing RF tags –Location sensing Geometric Operations –Multi-view analysis –Deformation Interactive Projection –Mouse-like interaction –Image stabilization Applications

15. Mitsubishi Electric Research Labs Raskar, Beardsley, vanBaar, Wang, Dietz, Lee, Leigh, Willwacher R F I G Lamps Pattern MSB Pattern MSB-1 Pattern LSB Projected Sequential Frames Handheld Projector beams binary coded stripes Tags decode temporal code

16. Mitsubishi Electric Research Labs Raskar, Beardsley, vanBaar, Wang, Dietz, Lee, Leigh, Willwacher R F I G Lamps Pattern MSB Pattern MSB-1 Pattern LSB Projected Sequential Frames Handheld Projector beams binary coded stripes Tags decode temporal code

17. Mitsubishi Electric Research Labs Raskar, Beardsley, vanBaar, Wang, Dietz, Lee, Leigh, Willwacher R F I G Lamps Pattern MSB Pattern MSB-1 Pattern LSB Projected Sequential Frames Handheld Projector beams binary coded stripes Tags decode temporal code

18. Mitsubishi Electric Research Labs Raskar, Beardsley, vanBaar, Wang, Dietz, Lee, Leigh, Willwacher R F I G Lamps Pattern MSB Pattern MSB-1 Pattern LSB Projected Sequential Frames Handheld Projector beams binary coded stripes Tags decode temporal code

19. Mitsubishi Electric Research Labs Raskar, Beardsley, vanBaar, Wang, Dietz, Lee, Leigh, Willwacher R F I G Lamps Pattern MSB Pattern MSB-1 Pattern LSB Projected Sequential Frames Handheld Projector beams binary coded stripes Tags decode temporal code

20. Mitsubishi Electric Research Labs Raskar, Beardsley, vanBaar, Wang, Dietz, Lee, Leigh, Willwacher R F I G Lamps Pattern MSB Pattern MSB-1 Pattern LSB For each tag a.From light sequence, decode x and y coordinate b.Transmit back to RF reader (Id, x, y) For each tag a.From light sequence, decode x and y coordinate b.Transmit back to RF reader (Id, x, y) 0 0 1 1 1 1 0 0 0 0 X=12

21. Mitsubishi Electric Research Labs Raskar, Beardsley, vanBaar, Wang, Dietz, Lee, Leigh, Willwacher R F I G Lamps Visual feedback of 2D position a.Receive via RF {(x 1,y 1 ), (x 2,y 2 ), …} pixels b.Illuminate those positions a.Receive via RF {(x 1,y 1 ), (x 2,y 2 ), …} pixels b.Illuminate those positions

22. Mitsubishi Electric Research Labs Raskar, Beardsley, vanBaar, Wang, Dietz, Lee, Leigh, Willwacher R F I G Lamps Limitations Line of sight ~Surface patches ~Multiple tags per object Ambient light ~Modulated infrared Surface reflectance and shadows

23. Mitsubishi Electric Research Labs Raskar, Beardsley, vanBaar, Wang, Dietz, Lee, Leigh, Willwacher R F I G Lamps Outline Photosensing RF tags –Location sensing Geometric Operations –Multi-view analysis –Deformation Interactive Projection –Mouse-like interaction –Image stabilization Applications

24. Mitsubishi Electric Research Labs Raskar, Beardsley, vanBaar, Wang, Dietz, Lee, Leigh, Willwacher R F I G Lamps Find 3D coordinates a.Observe structured patterns via Camera b.Triangulate cam/proj view a.Observe structured patterns via Camera b.Triangulate cam/proj view

25. Mitsubishi Electric Research Labs Raskar, Beardsley, vanBaar, Wang, Dietz, Lee, Leigh, Willwacher R F I G Lamps 3D from 2 Projector Views (Structure from Motion) Two+ unknown projector views Correspondence is trivial Applications –Detect 3D deformations –Trajectory grouping

26. Mitsubishi Electric Research Labs Raskar, Beardsley, vanBaar, Wang, Dietz, Lee, Leigh, Willwacher R F I G Lamps Change Detection without fixed camera, in any lighting condition Compare with new coordinates from a different view Record coordinates of tags from one view BeforeAfter

27. Mitsubishi Electric Research Labs Raskar, Beardsley, vanBaar, Wang, Dietz, Lee, Leigh, Willwacher R F I G Lamps Texture Adaptation

28. Mitsubishi Electric Research Labs Raskar, Beardsley, vanBaar, Wang, Dietz, Lee, Leigh, Willwacher R F I G Lamps Outline Photosensing RF tags –Location sensing Geometric Operations –Multi-view analysis –Deformation Interactive Projection –Desktop-like interaction –Image stabilization Applications

29. Mitsubishi Electric Research Labs Raskar, Beardsley, vanBaar, Wang, Dietz, Lee, Leigh, Willwacher R F I G Lamps Desktop-like Interaction Selecting tags

30. Mitsubishi Electric Research Labs Raskar, Beardsley, vanBaar, Wang, Dietz, Lee, Leigh, Willwacher R F I G Lamps Support for handheld projection

31. Mitsubishi Electric Research Labs Raskar, Beardsley, vanBaar, Wang, Dietz, Lee, Leigh, Willwacher R F I G Lamps Mouse Simulation Cursor follows handheld projector motion Pre-warped image remains stable

32. Mitsubishi Electric Research Labs Raskar, Beardsley, vanBaar, Wang, Dietz, Lee, Leigh, Willwacher R F I G Lamps Image Quasi-Stabilization Eliminate hand jitter using inertial sensors+camera

33. Mitsubishi Electric Research Labs Raskar, Beardsley, vanBaar, Wang, Dietz, Lee, Leigh, Willwacher R F I G Lamps Absolute Stabilization Image stays registered with world features

34. Mitsubishi Electric Research Labs Raskar, Beardsley, vanBaar, Wang, Dietz, Lee, Leigh, Willwacher R F I G Lamps Image Stabilization

35. Mitsubishi Electric Research Labs Raskar, Beardsley, vanBaar, Wang, Dietz, Lee, Leigh, Willwacher R F I G Lamps Interactive Projection (Also in Emerging Technologies Booth)

36. Mitsubishi Electric Research Labs Raskar, Beardsley, vanBaar, Wang, Dietz, Lee, Leigh, Willwacher R F I G Lamps Adaptive Projection Paste : Geometric and Photometric compensation

37. Mitsubishi Electric Research Labs Raskar, Beardsley, vanBaar, Wang, Dietz, Lee, Leigh, Willwacher R F I G Lamps Prototype Handheld Projector

38. Mitsubishi Electric Research Labs Raskar, Beardsley, vanBaar, Wang, Dietz, Lee, Leigh, Willwacher R F I G Lamps Related Work Smart objects –Smart-its –FindIT Flashlight (Ma and Paradiso 2002) Location sensing –Multiple readers, large antennas –Olivetti Active Badge, Xerox PARCtab (Want 1995) Interaction and Display –Augmented Reality –Projector-based AR, Shaderlamps (Raskar 1998, Underkoffler 1999, Pinhanez 2001) –Handheld projector (Raskar 2003)

39. Mitsubishi Electric Research Labs Raskar, Beardsley, vanBaar, Wang, Dietz, Lee, Leigh, Willwacher R F I G Lamps Applications Single Tags –Authoring for AR, Store augmentation: display per object Multiple tags on objects –3D Pose –Packing and placement strategy –Robot navigation Interaction between tagged objects –History of geometric interaction –Orientation mismatch Distributed tags –Deformation –Interpolated values for temperature sensors

40. Mitsubishi Electric Research Labs Raskar, Beardsley, vanBaar, Wang, Dietz, Lee, Leigh, Willwacher R F I G Lamps Acknowledgements MERL –Cliff Forlines –Joe Marks, –Dick Waters, Kent Wittenburg –Vlad Branzoi –Rebecca Xiong, Debbi van Baar Mitsubishi Electric, Japan –Mamuro Kato, –Keiichi Shiotani

41. Mitsubishi Electric Research Labs Raskar, Beardsley, vanBaar, Wang, Dietz, Lee, Leigh, Willwacher R F I G Lamps Interacting with a Self-Describing World Hybrid optical and RF communication –Photosensing Wireless Tags + Projector illumination Geometric Analysis –Location, Selection, Augmentation At a distance Interactive Projection –Desktop like interaction –Stabilized images RFIG Applications beyond logistics –Identity, Geometry, History, Annotation www.MERL.com

42. Mitsubishi Electric Research Labs Raskar, Beardsley, vanBaar, Wang, Dietz, Lee, Leigh, Willwacher R F I G Lamps Find tag location using handheld Projector Photosensing Wireless Tags Many geometric ops R F I R F I D Interactive stabilized projection (Radio Frequency Id & Geometry) (Radio Frequency Id & Geometry) G

43. Mitsubishi Electric Research Labs Raskar, Beardsley, vanBaar, Wang, Dietz, Lee, Leigh, Willwacher R F I G Lamps (End of Presentation)

44. Mitsubishi Electric Research Labs Raskar, Beardsley, vanBaar, Wang, Dietz, Lee, Leigh, Willwacher R F I G Lamps Towards Passive RFID Photosensor –Most compatible with passive batteryless RFID –Power: near zero power for sensing –Size: Each photocell is tens of micrometer (smaller than RFID which are about 500 micrometer) –Very low cost, can be built with silicon of RFID microchip –RFID+Photosensor can be size of grain of rice –RFIG tag is visible to projector but not to humans –Current prototype uses battery because passive tags are difficult to program Other possibilities –Adding a light emitting diode (LED) Requires on board power Size has to be considerably large Size and power requirements are high

45. Mitsubishi Electric Research Labs Raskar, Beardsley, vanBaar, Wang, Dietz, Lee, Leigh, Willwacher R F I G Lamps Future Directions Sophisticated tags Light modulation Handheld projector

46. Mitsubishi Electric Research Labs Raskar, Beardsley, vanBaar, Wang, Dietz, Lee, Leigh, Willwacher R F I G Lamps Comparison with Bar-codes RFID –Barcodes take up more space –Long bit sequences –Multiple tags can be simultaneously read –Read distance is high RFIG –Multiple barcodes per object is cumbersome –Difficult to attach sensor (e.g. temperature)

47. Mitsubishi Electric Research Labs Raskar, Beardsley, vanBaar, Wang, Dietz, Lee, Leigh, Willwacher R F I G Lamps Locate Tags Precisely using a handheld scanner/projector Handheld device Eliminates multiple RF readers for triangulation or large antennas Projector indicates location of tag Illuminate tags with binary coded stripes with projector

48. Mitsubishi Electric Research Labs Raskar, Beardsley, vanBaar, Wang, Dietz, Lee, Leigh, Willwacher R F I G Lamps Photosensor added to RFID tags Hybrid communication via RF and via light Solve 3 main issues with conventional RFIDs 1.Locate tags precisely 2.Select a subset of tags for operation 3.Long distance operation without collision Applications beyond inventory management 1.Warehouse: locate objects for specific queries 2.Training and maintenance: interactive augmented reality 3.Surveillance: keep trail of object locations

49. Mitsubishi Electric Research Labs Raskar, Beardsley, vanBaar, Wang, Dietz, Lee, Leigh, Willwacher R F I G Lamps 2. Select subset of tags for operation Advantages Only tags which are illuminated respond to queries No RF collision between multiple tags Projector also use as a display to indicate result of operation Technique Illuminate subset of tags with flashlight or projector 3. Long distance operation without RF collision

50. Mitsubishi Electric Research Labs Raskar, Beardsley, vanBaar, Wang, Dietz, Lee, Leigh, Willwacher R F I G Lamps Futures We are used to fixed displays like desktops, laptops, and static projectors. Handheld devices with screens have become common over the last decade – but the displays are small. Handheld projectors offer a new display modality with its own characteristics - portable device size but with a large display. Add interaction to achieve the world as a desktop…

51. Mitsubishi Electric Research Labs Raskar, Beardsley, vanBaar, Wang, Dietz, Lee, Leigh, Willwacher R F I G Lamps Interactive Projection Desktop like interaction –Cursor follows use motion –Center pixel is the pointer –Rest of the image is stabilized –Selection Image stabilization –Absolute: camera and inertial sensor –Relative: world markers Copy-paste –Camera captures texture during copy –Projection after geometric/photometric adaptation