Mitsubishi Electric Research LaboratoriesSpecial Effects in the Real WorldRaskar 2006 Ramesh Raskar Mitsubishi Electric Research Labs (MERL) Cambridge, MA, USA Special Effects in the Real World

Mitsubishi Electric Research LaboratoriesSpecial Effects in the Real WorldRaskar 2006 Parent Organization: MELCO Mitsubishi Electric (MELCO), including subsidiaries –$32B in annual sales –125,000 people –10 business sectors: Space Development and Satellite Communications Communications and Information Processing Visual Equipment and Systems Electronic Devices Home Electronics Energy Industrial Equipment Public-Use Systems Transportation

Mitsubishi Electric Research LaboratoriesSpecial Effects in the Real WorldRaskar 2006 MERL Mitsubishi Electric Research Laboratories –Dick Waters, President & CEO Two labs, co-located in Cambridge, MA: –MERL Technology Lab Advanced systems development ~35 researchers –MERL Research Lab Joe Marks, Director Basic research in applied computing 23 researchers, interns, consultants CV, HCI, Multimedia, CG, AI

Mitsubishi Electric Research LaboratoriesSpecial Effects in the Real WorldRaskar 2006 Special Effects –Photorealism and Animation Stays on screens Yet to affect daily life in real time –Post-processed content –Unlike internet,databases, UI

Mitsubishi Electric Research LaboratoriesSpecial Effects in the Real WorldRaskar 2006 Changing Appearance Virtual light source

Mitsubishi Electric Research LaboratoriesSpecial Effects in the Real WorldRaskar 2006 Changing Virtual Illumination

Mitsubishi Electric Research LaboratoriesSpecial Effects in the Real WorldRaskar 2006

Mitsubishi Electric Research LaboratoriesSpecial Effects in the Real WorldRaskar 2006 Special Effects in the Real World –Photorealism and Animation Limited to display screens Yet to affect daily life in real time –Unlike other fields of info-tech –Fusion: real world with graphics Next big challenge in CG ? New opportunities in advt, gaming, animation Believable co-existence

Mitsubishi Electric Research LaboratoriesSpecial Effects in the Real WorldRaskar 2006 Changing Appearance Projector Virtual light source

Mitsubishi Electric Research LaboratoriesSpecial Effects in the Real WorldRaskar 2006 Changing Virtual Illumination Raskar, Welch, Low, Bandyopadhyay, “Shader Lamps” (2000)

Mitsubishi Electric Research LaboratoriesSpecial Effects in the Real WorldRaskar 2006 Virtual Motion

Mitsubishi Electric Research LaboratoriesSpecial Effects in the Real WorldRaskar 2006 Virtual Motion Raskar, Ziegler, Willwacher, “Cartoon Dioramas in Motion,” (NPAR 2002)

Mitsubishi Electric Research LaboratoriesSpecial Effects in the Real WorldRaskar 2006 Dynamic Augmentation Projecting on Tracked Objects

Mitsubishi Electric Research LaboratoriesSpecial Effects in the Real WorldRaskar 2006 Virtual ReflectanceVirtual Illumination InteractionVirtual Motion ShaderLamps

Mitsubishi Electric Research LaboratoriesSpecial Effects in the Real WorldRaskar 2006 Projector-based Augmentation Virtual ReflectanceVirtual Illumination InteractionVirtual Motion

Mitsubishi Electric Research LaboratoriesSpecial Effects in the Real WorldRaskar 2006 Poor Man’s Palace Surround CG for Daily Environment

Mitsubishi Electric Research LaboratoriesSpecial Effects in the Real WorldRaskar 2006 Maya: World is an Illusion Fake World –We all live in one Real-life Fakes –Not just photos and videos but surroundings Privacy

Mitsubishi Electric Research LaboratoriesSpecial Effects in the Real WorldRaskar 2006 Augmented Reality (AR) With See-through Head-mounted Display © Andrei State Superman X-ray vision

Mitsubishi Electric Research LaboratoriesSpecial Effects in the Real WorldRaskar 2006 hand-held display spatial see-through display head- mounted display retinal display Head-attached Body-attached spatial projector Spatial Augmented Reality Classification of Augmented Reality

Mitsubishi Electric Research LaboratoriesSpecial Effects in the Real WorldRaskar 2006 Being There Recreate Large Environments –‘BeingThere’, walk-around –Museums, Exhibitions, Training Kok-Lim Low, Greg Welch, Anselmo Lastra, Henry Fuchs. “Life-Sized Projector-Based Dioramas,” Proc. ACM Symposium on Virtual Reality Software and Technology 2001 (VRST 2001), November 2001.

Mitsubishi Electric Research LaboratoriesSpecial Effects in the Real WorldRaskar 2006 Desired Virtual Model © Andrei State Projected Guidance for Placement

Mitsubishi Electric Research LaboratoriesSpecial Effects in the Real WorldRaskar 2006

Mitsubishi Electric Research LaboratoriesSpecial Effects in the Real WorldRaskar 2006 Head-mounted DisplayProjectors © Andrei State Augmented Reality

Mitsubishi Electric Research LaboratoriesSpecial Effects in the Real WorldRaskar 2006 Opportunities Entertainment -Animating objects, Museums, Special Effects, Laser Shows Advertising -Car showrooms, Furniture store Design and Ergonomics -Prototyping products, visualize material changes Information Augmentation -Office, factory or assembly Training

Mitsubishi Electric Research LaboratoriesSpecial Effects in the Real WorldRaskar 2006 Spatially Augmented Reality (SAR) HMD-VR Spatially Immersive-VR AR using HMD Spatially Augmented Reality e.g. CAVE Video or Optical see-through SAR, Shaderlamps

Mitsubishi Electric Research LaboratoriesSpecial Effects in the Real WorldRaskar 2006 Steps –Preprocessing Scan 3D object Roughly align projectors Compute correction –Run time Render images of 3D model Correct

Mitsubishi Electric Research LaboratoriesSpecial Effects in the Real WorldRaskar 2006 Steps –Preprocessing Scan 3D object and create virtual model Faro arm

Mitsubishi Electric Research LaboratoriesSpecial Effects in the Real WorldRaskar 2006 Steps –Preprocessing Scan 3D object and create model Approximately position projector(s) Calibrate by finding pose

Mitsubishi Electric Research LaboratoriesSpecial Effects in the Real WorldRaskar 2006 Steps –Preprocessing Scan 3D object and create model Approximately position projector(s) Calibrate by finding pose –Run-time Inverse Rendering of 3D Models

Mitsubishi Electric Research LaboratoriesSpecial Effects in the Real WorldRaskar 2006 Shape cue allows perceptual unwrapping

Mitsubishi Electric Research LaboratoriesSpecial Effects in the Real WorldRaskar 2006 ‘Wallpaper the surface’ Shape Adaptive Projection Minimum Stretch Display

Mitsubishi Electric Research LaboratoriesSpecial Effects in the Real WorldRaskar 2006 Shape Adaptive Projection Minimum Stretch Display Raskar, vanBaar, Beardsley, Willwacher, Rao, Forlines ‘iLamps: Geometrically Aware and Self-Configurable Projectors’, SIGGRAPH 2003

Mitsubishi Electric Research LaboratoriesSpecial Effects in the Real WorldRaskar 2006 Spatially Augmented Reality Raskar, vanBaar, Beardsley, Willwacher, Rao, Forlines ‘iLamps: Geometrically Aware and Self-Configurable Projectors’, SIGGRAPH 2003

Mitsubishi Electric Research LaboratoriesSpecial Effects in the Real WorldRaskar 2006 Mitsubishi Pocket Projector

Mitsubishi Electric Research LaboratoriesSpecial Effects in the Real WorldRaskar 2006

Mitsubishi Electric Research LaboratoriesSpecial Effects in the Real WorldRaskar 2006 A new way to play tic-tac-toe Handheld projector. Grid projects to a fixed position on the wall. Cursor is guided by pointing the projector.

Mitsubishi Electric Research LaboratoriesSpecial Effects in the Real WorldRaskar 2006 Pocket Projector Games

Mitsubishi Electric Research LaboratoriesSpecial Effects in the Real WorldRaskar 2006 Swarms

Mitsubishi Electric Research LaboratoriesSpecial Effects in the Real WorldRaskar 2006 Warehousing Routing Library Baggage handling Currency Livestock tracking RFID

Mitsubishi Electric Research LaboratoriesSpecial Effects in the Real WorldRaskar 2006 Conventional Passive RFID

Mitsubishi Electric Research LaboratoriesSpecial Effects in the Real WorldRaskar 2006 Tagged Books in a Library Id : List of books in RF range No Precise Location Data Are books in sorted order ? Which book is upside down ?

Mitsubishi Electric Research LaboratoriesSpecial Effects in the Real WorldRaskar 2006 Prototype Tag RF tag + photosensor

Mitsubishi Electric Research LaboratoriesSpecial Effects in the Real WorldRaskar 2006 Find tag location using handheld Projector Photosensing Wireless Tags Many geometric ops R F I R F I D Interactive stabilized projection (Radio Frequency Id & Geometry) (Radio Frequency Id & Geometry) G Siggraph 2004

Mitsubishi Electric Research LaboratoriesSpecial Effects in the Real WorldRaskar 2006 AR with Photosensing RFID and Handheld Projector

Mitsubishi Electric Research LaboratoriesSpecial Effects in the Real WorldRaskar 2006 Visual feedback of 2D position a.Receive via RF {(x 1,y 1 ), (x 2,y 2 ), …} pixels b.Illuminate those positions a.Receive via RF {(x 1,y 1 ), (x 2,y 2 ), …} pixels b.Illuminate those positions

Mitsubishi Electric Research LaboratoriesSpecial Effects in the Real WorldRaskar 2006 RFID (Radio Frequency Identification) RFIG (Radio Frequency Id and Geometry)

Mitsubishi Electric Research LaboratoriesSpecial Effects in the Real WorldRaskar 2006 CurvedPlanar Non-planar Single Projector Multiple Projectors Projector j Use r : T ? Pocket-Proj Objects Projectors and Surfaces

Mitsubishi Electric Research LaboratoriesSpecial Effects in the Real WorldRaskar 2006 Augmentation of “five senses” Visual Displays Auditory Displays Haptic Displays Locomotion Interfaces Olfactory Displays Taste Displays ? Sixth Sense.. Extending sensory modalities

Mitsubishi Electric Research LaboratoriesSpecial Effects in the Real WorldRaskar 2006 Sense of Balance Galvanic Vestibular Stimulation Masahiko Inami (UEC)

Mitsubishi Electric Research LaboratoriesSpecial Effects in the Real WorldRaskar 2006 Electric Cutaneous Display Skin Nerve Electrode Dr. H. Kajimoto, U Tokyo Masahiko Inami (UEC)

Mitsubishi Electric Research LaboratoriesSpecial Effects in the Real WorldRaskar 2006 Smart Touch Masahiko Inami (UEC)

Mitsubishi Electric Research LaboratoriesSpecial Effects in the Real WorldRaskar 2006 Scent Display Yasuyuki Yanagida, ATR Spatio-Temporal Controlled Delivery of Scent

Mitsubishi Electric Research LaboratoriesSpecial Effects in the Real WorldRaskar 2006 Air Cannon Generating a doughnut-shaped vortex ring Aperture Push

Mitsubishi Electric Research LaboratoriesSpecial Effects in the Real WorldRaskar 2006 Displays Contenders Organic LED Light Emitting Polymers E-Ink

Mitsubishi Electric Research LaboratoriesSpecial Effects in the Real WorldRaskar 2006 Holodust Perlin et al

Mitsubishi Electric Research LaboratoriesSpecial Effects in the Real WorldRaskar 2006 Fake World

Mitsubishi Electric Research LaboratoriesSpecial Effects in the Real WorldRaskar 2006 Poor Man’s Palace Surround CG for Daily Environment

Mitsubishi Electric Research LaboratoriesSpecial Effects in the Real WorldRaskar 2006 Recap Special Effects in Real World –Photorealism yet to impact daily life –Poor Man’s Palace Spatial Augmented Reality –Solution for fusion: Projectors, RFID, Sensors –Games for pocket projector –Display anywhere for advertising Future Problems –Natural phenomenon, complex reflectance, other displays –All senses: haptic, olfactory, preprioception Next Challenge in Real World Animation –Movies and games in our surroundings –3D Advertising

Mitsubishi Electric Research LaboratoriesSpecial Effects in the Real WorldRaskar 2006 Spatial Augmented Reality Graphics –Photorealism yet to impact daily life –Special effects in real world Spatial Augmentation –Solution for fusion –Needs hardware components –Projectors, RFID, Sensors Next Challenge in Animation raskar.com