1. Mobile Augmented Reality Dieter Schmalstieg Graz University of Technology, Austria

2. Mobile Augmented Reality Lecture Overview Motivation Hardware requirements Hardware requirements Tracking Tracking Environmental modeling Environmental modeling

3. Mobile AR – Motivation Mobile, wearable computing opens up new possibilities location-aware/situated computing Now, the interface is everywhere AR is a powerful UI for this type of computing

4. Mobile AR – Motivation Mobile AR Applications: Navigational aids Communication aids Personal situated information DB General UI for appliances Tourism Journalism Maintenance and construction Military training and warfighting

5. Mobile AR – Background Post-WIMP interfaces: 3D Desktop 3D, Desktop VR, Fishtank VR Projection-based VR Head-mounted VR Situated Mobile, Wearable, Handheld Multi-Device, Pervasive Tangible, Embodied Multimodal Speech, Gestures, Audio, Haptic Ubiquitous Computing Mobile AR

6. Mobile AR – Background Steps Toward Wearable Computing Computer Form Factor RoomWallDeskBoxLaptopPalmtopClothingUserRelationshipSubmitShare Sit at... and carry before/after... and carry before/afterHoldWear

7. Implications of Wearability (after S. Mann, B. Rhodes, T. Starner) Mobility usable/used indoors and outdoorsIntimacy sense the wearer’s body, communicate privately Context sensitivity take into account changing environmentConstancy Permeation of UI into wearer’s life

8. Mobile AR – Background Situated Computing Ubiquitous computing (Weiser ’89) PARCTab (1993) Hull et al. (’97) state that “situated computing concerns the ability of computing devices to detect, interpret, and respond to aspects of the user’s local environment” “situated computing concerns the ability of computing devices to detect, interpret, and respond to aspects of the user’s local environment”

9. Mobile AR – Background WorldBoard 1990s: many researchers started to co- locate information with physical space J. Spohrer 1996: What comes after the World Wide Web? Information in place. The world as a repository of information. (Imagined as a service Apple Computer, Inc. would provide.)

10. What is Mobile AR? Ways of augmenting a mobile user’s environment wearable display, no tracking whatsoever body-stabilized wearable display (orientation tracking only) location-dependent audio augmentation (with or without spatialized audio) location-dependent screen-stabilized augmentation (possibly monocular) location-dependent body-stabilized augmentation (on a projection cylinder/sphere surrounding the user) stereo head-tracked, position tracked, AR with full overlay registration

11. Ideal Mobile AR Device + A pair of stylish sunglasses + Hi-res stereo 3D graphics + Built-in computer with wireless network + Highly accurate 6DOF tracking + All for $99.90 - Doesn‘t exist (yet?)

12. Challenges of Mobile AR Challenges of mobile computing Limited resources Size, weight Battery live Ruggedness Challenges of mobile AR Tracking, 3D graphics, real-time performance Challenges of outdoor environment Lighting conditions no instrumented environment possible

13. Components of Mobile AR Overview: Computing platforms DisplaysTracking

14. Mobile computer form factors Notebook Tablet PC Wearable PC PDASmartphone

15. AR Backpack Examples Columbia Touring Machine (2002 ) Rockwell vest (1999) AT&T Sentient AR (2001) 1997

16. Display platforms Minolta Forgettable Display MicroVision Nomad retinal scanning display MicroOptical EG 7 Sony Glasstron Stereo optical see-thru Head-mounted display Monocular, monoscopic, stereoscopic Optical/video see-thru, see-around Handheld display (PDA etc.) Audio display

17. Tracking Requirements Provide position and orientation (separately?) Untethered, large working volume Indoor vs outdoor Indoor: Can instrument environment Outdoor: Self-contained or satellite based

18. Indoor Tracking Ultrasonic beacon array AT&T Bat, Intersense IS900 Infrared LED array UNC HiBall, MIT‘s locust swarm Outside-in computer vision Observer cameras + passive IR targets (e.g., ARTTrack - medium range) Inside-out computer vision Fiducials (e.g. ARToolKit at Uni.SA) Instrument environment with fiducials Dead-reckoning techniques

19. Outdoor tracking GPS, enhanced GPS differential GPS RTK Pseudolites Dead-reckoning Inside-out computer vision Natural features, e.g. USC, TU-Graz Still challenges in performance + robustnessOrientation Gyroscopes, magnetometers, inclinometers

20. Some Pioneers of Outdoor AR Columbia University: MARS ‘97 Uni.SA: Tinmith Naval Research Lab: BARS

21. Receiver chain VGA Serial port q inertial = [(q x, q y, q z ), q w ] Inertial tracker Receivers Sentient Computing Server Case Study: AT&T Cambridge’s Sentient AR Memory prod Maintainence Everywhere GUI

22. Case Study: Studierstube Mobile AR System HMD DGPS Inertial sensor camera notebook WLAN GPRS modem tracked touchpad Wide Area Tracking - DGPS (outdoors) - ARToolKit (indoors) Indoor + outdoor

23. Navigation & Browsing indoors outdoors

24. Ergonomic Considerations How to make wearable AR ergonomically and socially acceptable?

25. Surprising Answer We can start today! „Handheld Augmented Reality“ Platform: PocketPC Fully self contained vision tracking + 3D graphics

26. Environmental Modeling Unless we are attaching information to markers in the scene only, the computer needs a model of the environment For annotating detailed infrastructure: need geometrical model Access to DB of environmental information

27. Environmental Modeling Model urban infrastructure from 2D topographic maps and aerial photographs Modeling from laser range finder data Modeling from a combination of a set of photographs and geometrical constraints (Berkeley Façade, Canoma)