Ramesh Raskar Media Lab, MIT Cambridge, MA Second Skin

Second Skin and RFIG Second Skin –Sensing and Actuation Every mm at every ms –Impercetible, natural environment Overcoming the ‘Dark side of VR’ RFIG –Marker technology –Marker= RFID + photosensor (not barcode) –Locator= Projector (not camera) –Durable (10+ yrs), long range, imperceptible (zero area) Think inverse

Vicon Motion Capture High-speed IR Camera Body-worn markers Medical RehabilitationAthlete Analysis Performance CaptureBiomechanical Analysis

‘Motion’ Capture ? Building a real time human model –Dense sampling over surface –Geometry with Id at every millisecond at every milimeter –Bio parameters Getting intimate –Cameras.. –Wearables –Second Skin (Sensor suit) –Tapping inside Close the loop in bio-I/O –Remote monitoring: Elderly care, training –Robot observation:learning, worker safety –Feedback for biomech/neuro interfaces

Tagged Books in a Library Id : List of books in RF range No Precise Location Data Are books in sorted order ?

RF Tag + Photosensor

Conventional RF tag Photo-sensing RF tag Projector + Photo-sensing RF tag

Pattern MSB Pattern MSB-1 Pattern LSB For each tag a.From light sequence, decode x and y coordinate b.Transmit back to RF reader (Id, x, y) For each tag a.From light sequence, decode x and y coordinate b.Transmit back to RF reader (Id, x, y) X=12

Visual feedback of 2D position a.Receive via RF {(Id 1,x 1,y 1 ), (Id 2,x 2,y 2 ), …} b.Illuminate those positions a.Receive via RF {(Id 1,x 1,y 1 ), (Id 2,x 2,y 2 ), …} b.Illuminate those positions

Siggraph 2004 Where are products about to expire ?

Find tag location using handheld Projector Photosensing Wireless Tags Many geometric ops R F I R F I D Interactive stabilized projection (Radio Frequency Id & Geometry) (Radio Frequency Id & Geometry) G Siggraph 2004

Mitsubishi Pocket Projector

AR with Photosensing RFID and Handheld Projector

Handheld Projector iLamps 2002RFIG Lamps Pocket Projector

Raskar, vanBaar, Beardsley, Willwacher, Rao, Forlines ‘iLamps: Geometrically Aware and Self-Configurable Projectors’, SIGGRAPH 2003

AR Issues Preprocessing: –Authoring Runtime: –Identification: Recognition of objects Using markers and visual tags –Registration: Finding relative pose of display device Dynamic estimate of translation and rotation Render/Warp images –Interaction: Widgets, Gesture recognition, Visual feedback

AR Issues Preprocessing: –Authoring Runtime: –Identification: Recognition of objects Using markers and visual tags –Registration: Finding relative pose of display device Dynamic estimate of translation and rotation Render/Warp images –Interaction: Widgets, Gesture recognition, Visual feedback RFID ?

AR Issues Preprocessing: –Authoring Runtime: –Identification: Recognition of objects Using markers and visual tags –Registration: Finding relative pose of display device Dynamic estimate of translation and rotation Render/Warp images –Interaction: Widgets, Gesture recognition, Visual feedback RFID Photosensing RFID Projector for visual feedback

Inside of Projector The Gray code pattern Focusing Optics Gray code Slide Condensing OpticsLight Source

Tag

2D Location3D Location Y data X data Y data X data X2 data

Pattern MSB Pattern MSB-1 Pattern LSB For each tag a.From light sequence, decode x and y coordinate b.Transmit back to RF reader (Id, x, y) For each tag a.From light sequence, decode x and y coordinate b.Transmit back to RF reader (Id, x, y) X=12

Imperceptible Tags under clothing, tracked under ambient light

Towards Second Skin Towards Second Skin Coded Illumination Motion Capture Clothing 500 Hz with Id for each Marker Tag Capture in Natural Environment –Visually imperceptible tags –Photosensing Tag can be hidden under clothes –Ambient lighting is ok Unlimited Number of Tags –Light sensitive fabric for dense sampling Non-imaging, complete privacy Base station and tags only a few 10’s $ Full body scan + actions –Elderly, patients, athletes, performers –Breathing, small twists, multiple segments or people –Animation Analysis

Second Skin and RFIG Second Skin –Sensing and Actuation Every mm at every ms –Impercetible, natural environment Overcoming the ‘Dark side of VR’ RFIG –Marker technology –Marker= RFID + photosensor (not barcode) –Locator= Projector (not camera) –Durable (10+ yrs), long range, imperceptible (zero area) Think inverse

Acknowledgements MERL Jeroen van Baar, Paul Beardsley, Remo Ziegler, Thomas Willwacher, Srinivas Rao, Cliff Forlines, Paul Dietz, Joe Marks, Darren Leigh Office of the Future group at UNC Chapel Hill Greg Welch, Kok-lim Low, Deepak B’padhyay, Aditi Majumder, Michael Brown, Ruigang Yang Henry Fuchs, Herman Towles Wei-chao Chen

END

END

END

Mitsubishi Electric Research LaboratoriesSpecial Effects in the Real WorldRaskar 2006 Ramesh Raskar Mitsubishi Electric Research Labs (MERL) Cambridge, MA The Poor Man’s Palace: Special Effects in the Real World

Special Effects and Virtual Worlds –Photorealism around us ? Stays on screens –Does it affect daily life in real time ? Unlike other fields

Changing Appearance

Changing Virtual Illumination

Special Effects and Virtual Worlds –Photorealism around us ? Stays on screens –Does it affect daily life in real time ? Unlike other fields –Fusion: real world with graphics Next big challenge in CG/Second Life ? Believable, seamless co-existence

Changing Appearance Projector Virtual light source

Changing Virtual Illumination Raskar, Welch, Low, Bandyopadhyay, “Shader Lamps” (2000)

–Preprocessing Scan 3D object and create model Roughly align projector(s) Calibrate by finding pose –Run-time Render images of 3D model Warp/Correct

Virtual Motion

Raskar, Ziegler, Willwacher, “Cartoon Dioramas in Motion,” (NPAR 2002)

Dynamic Augmentation Projecting on Tracked Objects

d ( x ) 2 k ( x ) cos(  p ) Radiance Adjustment I p (x,  p ) = I d, k ( x ) > 0 L ( x,  ) VirtualReal Intensity correction Desired radiance Pixel intensity Reflectance

Virtual ReflectanceVirtual Illumination InteractionVirtual Motion ShaderLamps

Projector-based Augmentation Virtual ReflectanceVirtual Illumination InteractionVirtual Motion

Poor Man’s Palace

Maya: World is an Illusion Fake World –We all live in one Social Issues –Real-life Fakes Not just photos and videos but surroundings –Privacy X-reality/AR/Virtual Worlds –Delivers years of CG/Sensors/Robotics research into the real world –Time and Space shifting with non-linear distortions

Complex Reflectance –Specular or arbitrary BRDF surfaces –View-dependent appearance Participating Media –Simulating or in presence of smoke, fog Complex Geometry –Spaghetti Motion –Animation of real surfaces NPR, Distortions, Perceptual factors –Great thesis topics.. Beyond Gouraud Shading of White Objects

Pieces of the Puzzle Actuated Surfaces Actuated Workbench [Pangaro, Maynes-Aminzade, Ishii UIST 2002]

Displays Contenders Organic LED Light Emitting Polymers E-Ink

Recap Special Effects in Real World –Photorealism yet to impact daily life –Poor Man’s Palace Spatial Augmented Reality –Un tethered s olution for fusion –Geometry, Photometry, Id –Sense, Control, Compensate –Projectors, RFID, Sensors Open Problems –All senses: haptic, olfactory, proprioception –Natural phenomenon, complex BRDF, other displays Next Challenge: Photorealistic AR around us