A Survey of Augmented Reality Ronald T. Azuma, 1997 A Survey of Augmented Reality

2 Summary 1)What is Augmented Reality (AR)? 2)Applications 3)Characteristics 4)Problems 5)Solving problems using sensing 6)Future directions 7)Conclusions

3 1.1 What is Augmented Reality? Augmented Reality (AR) is a variation of Virtual Environments (VE), or Virtual Reality. These technologies completely immerse a user inside a synthetic environment. AR allows the user to see the real world (directly or indirectly), with virtual objects superimposed upon or composited with the real world. Augmented Reality enhances a user's perception of and interaction with the real world. Fred Brooks calls AR Intelligence Amplification (IA): using the computer as a tool to make a task easier for a human to perform. AR supplements reality, rather than completely replacing it.

4 1.2 What is Augmented Reality? Some researchers define AR in a way that requires the use of Head- Mounted Displays (HMDs). To avoid limiting AR to specific technologies, Ronald Azuma defines AR as systems that have the following three characteristics:  1) Combines real and virtual  2) Interactive in real time  3) Registered in 3D This definition allows other technologies besides HMDs while retaining the essential components of AR. There are three major display techniques for Augmented Reality:  Head Mounted Displays  Handheld Displays  Spatial Displays AR does not include film,2D overlays, or any non-interactive technologies.

5 2.1 Applications - Summary At least six classes of potential AR applications have been explored:  medical visualization,  assembly, maintenance and repair,  annotation and visualization,  robot path planning,  entertainment,  military aircraft navigation and targeting.

6 2.2 Medical (MRI+CT+US) + AR = “X-ray vision” AR surgery simulator

7 2.3 Manufacturing & Repair For complex machinery:  assembly,  maintenance,  repair. Superimposed 3D drawings can be animated, making directions even more explicit for a manual. Technology implies smaller costs and storage space. BMW Augmented Reality

8 2.4 Annotation and visualization Annotations for parts of an engine a library with handheld device, mapping (Bing maps 3D), real-life labels, architecture, os interface (p-ism). AR Tourist Guide

9 2.5 Robot path planning teleoperation for time-delay and human-induced oscillations – planning and previewing tool. the ARGOS system has demonstrated that stereoscopic AR is an easier and more accurate way of doing robot path planning than traditional monoscopic interfaces Augmented Reality - Helicopter iPhone Control

Entertainment used in 3D television sets. could be used in theater. computer games (cargocollective.com/moreyellow). tabletop games (SCOPE) arts clothes shopping AR Pool

Military Aircraft Head-up displays Helmet-mounted sights Navigation and flight information Aiming / Targeting Apache Targeting

Characteristics - Augmentation AR also has the ability to remove objects. Could also be applied to sound. Upgraded with Haptics.

Optical vs. Video Advantages:  Simple to design;  Nanoseconds delay;  No resolution reduction;  Safe in case of no power;  No eye offset; Disadvantages:  Reduces amount of light;  Can’t display solid virtual objects because of brightness differences.  Can’t perform “full” AR immersion.

Optical vs. Video Advantages:  Flexibility in blending;  Additional registration strategies;  Wide field of view;  Can solve distortions;  Easy to match brightness. Disadvantages:  Delay;  Camera position might confuse the user;  Expensive;  Dangerous in case of power failure;  More difficult to design.

Focus and Contrast Focus in optical systems is difficult to achieve because the virtual image is projected at a distance away from the user. This distance should be adjusted in order for the projected objects to be able to view both elements clearly. Contrast with optical systems is difficult because if the brightness of the virtual elements is not matched to the one of the original environment, then it could either be invisible or remove the environment. Focus for video systems can be tricky because the graphics should be rendered to simulate a limited depth-of-field, and the video camera should have an autofocus lens. Contrast is not such a big problem for video systems because the cameras themselves have limited dynamic response and everything has to be adjusted for the monitor.

Requirements Portability is not such a big problem anymore because devices have been invented that are light enough for people to wear. Also AR has been implemented into hand-held devices. Scene generators are not a big problem because usually the elements to not need to have such a great quality (for example annotations). Display devices are deal with the rendering of the images, for optical it is not a problem, but for video it might reduce the resolution. Tracking and sensing is a big requirement for AR.

Registration (tracking system) The objects in the real and virtual worlds must be properly aligned with respect to each other. Is a big problem for video systems because visual information tends to override all other senses (visual capture). Static errors Dynamic errors are the ones that have no effect until either the viewpoint or the objects begin moving. End to end system delay occurs especially in the video systems. Solutions: Reducing system lag by sacrificing throughput for delay. Reduce apparent lag using image deflection Match temporal streams Prediction is used to compute the movement, better with inertial sensors. cause registration errors even when the user's viewpoint and the objects in the environment remain completely still. Optical distortion  Additional optics  Digital compensation  Predistortion functions Errors in the tracking system  Discussed later Mechanical misalignments Incorrect viewing parameters  For optical based systems can be achieved through various calibration techniques.  For video systems it is solved by taking different angle pictures and computing parameters.

Vision-based techniques Using image processing or computer vision techniques to aid registration. Closed- loop feedback: feature detection in the video stream. On some applications fiducials (markers) are used in the environment – only relative position. Template matching searches the images for stored images. Using laser-rangefinders to acquire an initial depth-map. Asking the user for corrections of the differences between fusion of real and virtual.

19 5 Solving problems using sensing Accurate tracking of the user's head and sensing the locations of other objects in the environment. Input variety and bandwidth:  CT, MRI, US, IR and UV, remote sensors would let users view hidden objects.  Anything not detectable by human senses but detectable by machines is good for AR.  Range data is vital for many AR applications (laser, sonar, stereo vision) – difficult in real time.  Annotation systems require database as input. Higher accuracy:  Sensors that are accurate to around a millimeter and a tiny fraction of a degree, across the entire working range.  Every sensor type has disadvantages so combining them can compensate from each other. Longer range:  A scalable system is one that can be expanded to cover any desired range, simply by adding more modular components to the system.  GPS might be useful as a long-range tracker for AR systems, though accuracy is low.

20 5 Future directions Hybrid approaches:  Using multiple techniques to cover more weaknesses;  Combining for example vision-based techniques with prediction. Real-time systems and time-critical computing:  AR systems need to be build to synchronize the real world which runs in real time with the virtual part;  Problem occurs mostly for video-based systems. Perceptual and psychophysical studies:  How much delay, registration error, etc can a user detect?  How do we deal with possible optical illusions? Portability:  How to make the systems portable, even for outdoors.  There are video eyewear for mobile phones and pda’s, gaming consoles, etc. (Vuzix iWear).Vuzix iWear Multimodal displays:  Adding information for more of senses like sound and touch. Social and political issues:  Implications relating to the use of certain technologies in AR like lasers;  The way people perceive AR, as a threat or as a tool to Augment Intelligence.

21 6 Conclusions We need both optical and video based systems because they both have advantages and disadvantages. Research should go on to better optical technologies to allow more blending capabilities. These are more useful for civilian, business and military applications. Video based systems are also useful because they can be the only solution for some difficult medical tasks with teleoperated robots. The ultimate goal would be to generate virtual objects that are indistinguishable from real objects. (Photorealism) Head-up-displays (HUD) and Helmet-mounted-displays (HMD) are currently in use in military applications and decrease significantly the amount of collateral damage. There are applications for the iPhone for example that use annotations when the user uses the camera. (tourism for example) AR has also been used in cars to enhance visibility at night with IR. Several setups have been used in museums to enhance the expositions.