Mobile Augmented Reality Dieter Schmalstieg Graz University of Technology, Austria

Mobile Augmented Reality Lecture Overview Motivation Hardware requirements Hardware requirements Tracking Tracking Environmental modeling Environmental modeling

Mobile AR – Motivation Mobile, wearable computing opens up new possibilities location-aware/situated computing Now, the interface is everywhere AR is a powerful UI for this type of computing

Mobile AR – Motivation Mobile AR Applications: Navigational aids Communication aids Personal situated information DB General UI for appliances Tourism Journalism Maintenance and construction Military training and warfighting

Mobile AR – Background Post-WIMP interfaces: 3D Desktop 3D, Desktop VR, Fishtank VR Projection-based VR Head-mounted VR Situated Mobile, Wearable, Handheld Multi-Device, Pervasive Tangible, Embodied Multimodal Speech, Gestures, Audio, Haptic Ubiquitous Computing Mobile AR

Mobile AR – Background Steps Toward Wearable Computing Computer Form Factor RoomWallDeskBoxLaptopPalmtopClothingUserRelationshipSubmitShare Sit at... and carry before/after... and carry before/afterHoldWear

Implications of Wearability (after S. Mann, B. Rhodes, T. Starner) Mobility usable/used indoors and outdoorsIntimacy sense the wearer’s body, communicate privately Context sensitivity take into account changing environmentConstancy Permeation of UI into wearer’s life

Mobile AR – Background Situated Computing Ubiquitous computing (Weiser ’89) PARCTab (1993) Hull et al. (’97) state that “situated computing concerns the ability of computing devices to detect, interpret, and respond to aspects of the user’s local environment” “situated computing concerns the ability of computing devices to detect, interpret, and respond to aspects of the user’s local environment”

Mobile AR – Background WorldBoard 1990s: many researchers started to co- locate information with physical space J. Spohrer 1996: What comes after the World Wide Web? Information in place. The world as a repository of information. (Imagined as a service Apple Computer, Inc. would provide.)

What is Mobile AR? Ways of augmenting a mobile user’s environment wearable display, no tracking whatsoever body-stabilized wearable display (orientation tracking only) location-dependent audio augmentation (with or without spatialized audio) location-dependent screen-stabilized augmentation (possibly monocular) location-dependent body-stabilized augmentation (on a projection cylinder/sphere surrounding the user) stereo head-tracked, position tracked, AR with full overlay registration

Ideal Mobile AR Device + A pair of stylish sunglasses + Hi-res stereo 3D graphics + Built-in computer with wireless network + Highly accurate 6DOF tracking + All for $ Doesn‘t exist (yet?)

Challenges of Mobile AR Challenges of mobile computing Limited resources Size, weight Battery live Ruggedness Challenges of mobile AR Tracking, 3D graphics, real-time performance Challenges of outdoor environment Lighting conditions no instrumented environment possible

Components of Mobile AR Overview: Computing platforms DisplaysTracking

Mobile computer form factors Notebook Tablet PC Wearable PC PDASmartphone

AR Backpack Examples Columbia Touring Machine (2002 ) Rockwell vest (1999) AT&T Sentient AR (2001) 1997

Display platforms Minolta Forgettable Display MicroVision Nomad retinal scanning display MicroOptical EG 7 Sony Glasstron Stereo optical see-thru Head-mounted display Monocular, monoscopic, stereoscopic Optical/video see-thru, see-around Handheld display (PDA etc.) Audio display

Tracking Requirements Provide position and orientation (separately?) Untethered, large working volume Indoor vs outdoor Indoor: Can instrument environment Outdoor: Self-contained or satellite based

Indoor Tracking Ultrasonic beacon array AT&T Bat, Intersense IS900 Infrared LED array UNC HiBall, MIT‘s locust swarm Outside-in computer vision Observer cameras + passive IR targets (e.g., ARTTrack - medium range) Inside-out computer vision Fiducials (e.g. ARToolKit at Uni.SA) Instrument environment with fiducials Dead-reckoning techniques

Outdoor tracking GPS, enhanced GPS differential GPS RTK Pseudolites Dead-reckoning Inside-out computer vision Natural features, e.g. USC, TU-Graz Still challenges in performance + robustnessOrientation Gyroscopes, magnetometers, inclinometers

Some Pioneers of Outdoor AR Columbia University: MARS ‘97 Uni.SA: Tinmith Naval Research Lab: BARS

Receiver chain VGA Serial port q inertial = [(q x, q y, q z ), q w ] Inertial tracker Receivers Sentient Computing Server Case Study: AT&T Cambridge’s Sentient AR Memory prod Maintainence Everywhere GUI

Case Study: Studierstube Mobile AR System HMD DGPS Inertial sensor camera notebook WLAN GPRS modem tracked touchpad Wide Area Tracking - DGPS (outdoors) - ARToolKit (indoors) Indoor + outdoor

Navigation & Browsing indoors outdoors

Ergonomic Considerations How to make wearable AR ergonomically and socially acceptable?

Surprising Answer We can start today! „Handheld Augmented Reality“ Platform: PocketPC Fully self contained vision tracking + 3D graphics

Environmental Modeling Unless we are attaching information to markers in the scene only, the computer needs a model of the environment For annotating detailed infrastructure: need geometrical model Access to DB of environmental information

Environmental Modeling Model urban infrastructure from 2D topographic maps and aerial photographs Modeling from laser range finder data Modeling from a combination of a set of photographs and geometrical constraints (Berkeley Façade, Canoma)