1. Ramesh Raskar Media Lab, MIT Cambridge, MA Second Skin

2. Second Skin and RFIG Second Skin –Sensing and Actuation Every mm at every ms –Impercetible, natural environment Overcoming the ‘Dark side of VR’ RFIG –Marker technology –Marker= RFID + photosensor (not barcode) –Locator= Projector (not camera) –Durable (10+ yrs), long range, imperceptible (zero area) Think inverse

3. Vicon Motion Capture High-speed IR Camera Body-worn markers Medical RehabilitationAthlete Analysis Performance CaptureBiomechanical Analysis

4. ‘Motion’ Capture ? Building a real time human model –Dense sampling over surface –Geometry with Id at every millisecond at every milimeter –Bio parameters Getting intimate –Cameras.. –Wearables –Second Skin (Sensor suit) –Tapping inside Close the loop in bio-I/O –Remote monitoring: Elderly care, training –Robot observation:learning, worker safety –Feedback for biomech/neuro interfaces

5. Tagged Books in a Library Id : List of books in RF range No Precise Location Data Are books in sorted order ?

6. RF Tag + Photosensor

7. Conventional RF tag Photo-sensing RF tag Projector + Photo-sensing RF tag

8. Pattern MSB Pattern MSB-1 Pattern LSB For each tag a.From light sequence, decode x and y coordinate b.Transmit back to RF reader (Id, x, y) For each tag a.From light sequence, decode x and y coordinate b.Transmit back to RF reader (Id, x, y) 0 0 1 1 1 1 0 0 0 0 X=12

9. Visual feedback of 2D position a.Receive via RF {(Id 1,x 1,y 1 ), (Id 2,x 2,y 2 ), …} b.Illuminate those positions a.Receive via RF {(Id 1,x 1,y 1 ), (Id 2,x 2,y 2 ), …} b.Illuminate those positions

10. Siggraph 2004 Where are products about to expire ?

11. Find tag location using handheld Projector Photosensing Wireless Tags Many geometric ops R F I R F I D Interactive stabilized projection (Radio Frequency Id & Geometry) (Radio Frequency Id & Geometry) G Siggraph 2004

12. Mitsubishi Pocket Projector

13. AR with Photosensing RFID and Handheld Projector

14. Handheld Projector iLamps 2002RFIG Lamps 2003-04 Pocket Projector 2004-05

15. Raskar, vanBaar, Beardsley, Willwacher, Rao, Forlines ‘iLamps: Geometrically Aware and Self-Configurable Projectors’, SIGGRAPH 2003

16. AR Issues Preprocessing: –Authoring Runtime: –Identification: Recognition of objects Using markers and visual tags –Registration: Finding relative pose of display device Dynamic estimate of translation and rotation Render/Warp images –Interaction: Widgets, Gesture recognition, Visual feedback

17. AR Issues Preprocessing: –Authoring Runtime: –Identification: Recognition of objects Using markers and visual tags –Registration: Finding relative pose of display device Dynamic estimate of translation and rotation Render/Warp images –Interaction: Widgets, Gesture recognition, Visual feedback RFID ?

18. AR Issues Preprocessing: –Authoring Runtime: –Identification: Recognition of objects Using markers and visual tags –Registration: Finding relative pose of display device Dynamic estimate of translation and rotation Render/Warp images –Interaction: Widgets, Gesture recognition, Visual feedback RFID Photosensing RFID Projector for visual feedback

19. Inside of Projector The Gray code pattern Focusing Optics Gray code Slide Condensing OpticsLight Source

20. Tag

21. 2D Location3D Location Y data X data Y data X data X2 data

22. Pattern MSB Pattern MSB-1 Pattern LSB For each tag a.From light sequence, decode x and y coordinate b.Transmit back to RF reader (Id, x, y) For each tag a.From light sequence, decode x and y coordinate b.Transmit back to RF reader (Id, x, y) 0 0 1 1 1 1 0 0 0 0 X=12

23. Imperceptible Tags under clothing, tracked under ambient light

24. Towards Second Skin Towards Second Skin Coded Illumination Motion Capture Clothing 500 Hz with Id for each Marker Tag Capture in Natural Environment –Visually imperceptible tags –Photosensing Tag can be hidden under clothes –Ambient lighting is ok Unlimited Number of Tags –Light sensitive fabric for dense sampling Non-imaging, complete privacy Base station and tags only a few 10’s $ Full body scan + actions –Elderly, patients, athletes, performers –Breathing, small twists, multiple segments or people –Animation Analysis

25. Second Skin and RFIG Second Skin –Sensing and Actuation Every mm at every ms –Impercetible, natural environment Overcoming the ‘Dark side of VR’ RFIG –Marker technology –Marker= RFID + photosensor (not barcode) –Locator= Projector (not camera) –Durable (10+ yrs), long range, imperceptible (zero area) Think inverse http://raskar.info

26. Acknowledgements MERL Jeroen van Baar, Paul Beardsley, Remo Ziegler, Thomas Willwacher, Srinivas Rao, Cliff Forlines, Paul Dietz, Joe Marks, Darren Leigh Office of the Future group at UNC Chapel Hill Greg Welch, Kok-lim Low, Deepak B’padhyay, Aditi Majumder, Michael Brown, Ruigang Yang Henry Fuchs, Herman Towles Wei-chao Chen

27. END

28. END

29. END

30. Mitsubishi Electric Research LaboratoriesSpecial Effects in the Real WorldRaskar 2006 Ramesh Raskar Mitsubishi Electric Research Labs (MERL) Cambridge, MA The Poor Man’s Palace: Special Effects in the Real World

31. Special Effects and Virtual Worlds –Photorealism around us ? Stays on screens –Does it affect daily life in real time ? Unlike other fields

32. Changing Appearance

33. Changing Virtual Illumination

35. Special Effects and Virtual Worlds –Photorealism around us ? Stays on screens –Does it affect daily life in real time ? Unlike other fields –Fusion: real world with graphics Next big challenge in CG/Second Life ? Believable, seamless co-existence

36. Changing Appearance Projector Virtual light source

37. Changing Virtual Illumination Raskar, Welch, Low, Bandyopadhyay, “Shader Lamps” (2000)

38. –Preprocessing Scan 3D object and create model Roughly align projector(s) Calibrate by finding pose –Run-time Render images of 3D model Warp/Correct

39. Virtual Motion

40. Raskar, Ziegler, Willwacher, “Cartoon Dioramas in Motion,” (NPAR 2002)

41. Dynamic Augmentation Projecting on Tracked Objects

42. d ( x ) 2 k ( x ) cos(  p ) Radiance Adjustment I p (x,  p ) = I d, k ( x ) > 0 L ( x,  ) VirtualReal Intensity correction Desired radiance Pixel intensity Reflectance

43. Virtual ReflectanceVirtual Illumination InteractionVirtual Motion ShaderLamps www.ShaderLamps.com

44. Projector-based Augmentation www.ShaderLamps.com Virtual ReflectanceVirtual Illumination InteractionVirtual Motion

45. Poor Man’s Palace

47. Maya: World is an Illusion Fake World –We all live in one Social Issues –Real-life Fakes Not just photos and videos but surroundings –Privacy X-reality/AR/Virtual Worlds –Delivers years of CG/Sensors/Robotics research into the real world –Time and Space shifting with non-linear distortions

48. Complex Reflectance –Specular or arbitrary BRDF surfaces –View-dependent appearance Participating Media –Simulating or in presence of smoke, fog Complex Geometry –Spaghetti Motion –Animation of real surfaces NPR, Distortions, Perceptual factors –Great thesis topics.. Beyond Gouraud Shading of White Objects

49. Pieces of the Puzzle Actuated Surfaces Actuated Workbench [Pangaro, Maynes-Aminzade, Ishii UIST 2002]

50. Displays Contenders Organic LED Light Emitting Polymers E-Ink

51. Recap Special Effects in Real World –Photorealism yet to impact daily life –Poor Man’s Palace Spatial Augmented Reality –Un tethered s olution for fusion –Geometry, Photometry, Id –Sense, Control, Compensate –Projectors, RFID, Sensors Open Problems –All senses: haptic, olfactory, proprioception –Natural phenomenon, complex BRDF, other displays Next Challenge: Photorealistic AR around us