Chapter 6 – Direct Manipulation and Virtual Environments

6.1 Introduction Good interfaces produce positive feelings Desirable: –Visibility of objects –Visibility of actions –Rapid, reversible, incremental actions –Direct manipulation of objects of interest

6.2 Examples of Direct Manipulation Interfaces Windows environment (Xerox, Apple, Microsoft) Air Traffic Control Automobile? WYSIWYG Word Processors Mapping and GIS Modern computer games CAD Programming of industrial robots by moving robot by hand (actions recorded)

5.3 Explaining Direct Manipulation Beneficial attributes: –Novices learn quickly –Experts work rapidly –Intermittent users can retain concepts –Error messages are rarely needed –Users see if their actions are furthering their goals –Users experience less anxiety –Users gain confidence and mastery - encourages exploration

Explaining Direct Manipulation Problems with direct manipulation –Spatial or visual representations can be too spread out –Users must learn the graphical representations –The visual representation may be misleading –Typing commands with the keyboard may actually be faster –Choosing the right objects, actions, metaphors is not easy –May need greater system resources –History and other tracing may be difficult to maintain –Visual impaired users may have difficulty

Explaining Direct Manipulation Relationship to Object-Action Interface Model –Objects of interest are displayed –Interface actions are close to high level task domain

Icons An icon is an image, picture, or symbol representing a concept Icon-specific guidelines Represent the object or action in a familiar manner Limit the number of different icons Make icons stand out from the background Consider three-dimensional icons Ensure a selected icon is visible from unselected icons Ensure harmony and distinctiveness Design the movement animation Add detailed information Explore combinations of icons to create new objects or actions

Icons Five levels of icon design: –Lexical qualities –Syntactics –Semantics –Pragmatics –Dynamics Design starting with quick sketches Evaluate designs via testing with users

Direct Manipulation Programming Robots Programmed by workers leading them through task once (e.g. painting motion) Excel macros programmed via doing the tasks by hand MS Access Query by Example

Programming in the User Interface Five challenges of programming in the user interface: –Sufficient computational generality –Access to the appropriate data structures and operators –Ease in programming and editing programs –Simplicity in invocation and assignment of arguments –Low risk Possible alternative to Agents

6.6 Home Automation Remote control of devices is being extended to: –Channel audio and video –lawn watering –video surveillance and burglar alarms –Multiple-zone environmental controls –Maintenance records Providing direct-manipulation with rich feedback is vital in these applications Many direct-manipulation actions take place on a display of the floor plan ON and OFF can have many representations and present problems with choosing the appropriate one Controlling complex home equipment by direct manipulation reshapes how we think of homes and residents

6.7 Remote Direct Manipulation Examples: –Telemedicine –Robotic Space exploration –Home automation Complicating factors in the architecture of remote environments: –Time delays –Incomplete feedback –Feedback from multiple sources –Unanticipated interferences

6.8 Virtual Environments Virtual reality Augmented reality Situational awareness

Virtual Reality Examples –Architecture – actually walk into building and look around (instead of more traditional direct manipulation looking at it from various directions and perspectives) –Possible - medicine –On the boundary – real flight simulator

Example: Firefighter TrainingFirefighter Training This system will simulate the progress of a fire in a single family dwelling will respond to actions made by the user to rescue occupants and put the fire out. The user of the VE will be a Fire Company Officer being trained or evaluated for his/her skills at commanding a fire crew. In the VE, the user will speak commands that are translated by an operator into a predetermined animation sequence in the virtual environment. As the fire company officer issues commands, the virtual fire crew will go through animations reflecting these commands, fire burns in response to virtual crew actions.

Example: Meditation ChamberMeditation Chamber The goal of this research is design and build an immersive virtual environment that uses visual, audio, and tactile cues to create, guide, and maintain a patient's guided relaxation and meditation experience. The use of meditation and guided imagery is well established as helpful in the treatment and prevention of a number of diseases The possibility of increasing the effectiveness and repeatability of this type of therapy This project is aimed at creating a working prototype of this system

Example: Virtual Geographic Information System VGIS (Virtual Geographic Information System) is a large, multifaceted project to allow navigation of and interaction with very large and high resolution, dynamically changing databases while retaining real- time display and interaction. The system allows users to navigate accurate geographies with sustained frame rates of frames per second. The user can not only see these terrains from any viewing angle but also buildings, roads, high resolution imagery draped on the terrain, and other features

Example: Virtual Reality Phobia Therapy Virtual Reality Phobia Therapy Virtual Reality Exposure involves exposing the patient to a virtual environment containing the feared stimulus in place of taking the patient into a real environment or having the patient imagine the stimulus, which is what traditional exposure therapy usually involves.

Example: Helping Burn Patients Cope with Pain using immersive VR for pain control (in addition to pain medicine). Their first virtual world used was SpiderWorld. Spiderworld was originally designed to treat spider phobics, but has also proved quite distracting for burn patients.treat spider phobics now developing several new virtual environments specifically designed for treating pain (e.g., especially attention-grabbing virtual environments). SnowWorld has been developed with support from the Paul Allen Foundation for Medical Research. Patients fly through an icy canyon with a river and frigid waterfall. Patients shoot snowballs at snowmen and igloos (with animated impacts). Since patients often report re-living their original burn experience during wound care, SnowWorld was designed to help put out the fire.

Example: Virtual Gorilla Exhibit Virtual Gorilla Exhibit being developed to explore techniques for using Virtual Reality to present information to users experientially that would otherwise be difficult for them to learn. Based upon actual data from the Zoo Atlanta gorilla exhibit, modeling an environment where the user can explore areas that are normally off limits to the casual visitor.

Maybe not quite virtual reality "Macys.com gets rights to 3-D model software" by Reuters, CNET News.com, September 2, 1999"Macys.com gets rights to 3-D model software"CNET News.com Software developer Broderbund, sell its new Cosmopolitan Fashion Makeover software exclusively through Macys.com.BroderbundMacys.com With the Cosmopolitan software women can create their own three-dimensional model based on their own body measurements and digitally ``try on'' brand name clothing. Users can also link via the makeover software directly to the Macys.com online shopping site, where they can buy the clothing online.Macys.com

Virtual Reality Headgear

(1999)

Alternative to Headgear

Exploring using the CAVE

Virtual Environments Successful virtual environments depend on the smooth integration of: –Visual Display –Head position sensing –Hand-position sensing –Force feedback –Sound input and output –Other sensations –Cooperative and competitive virtual reality

1. Virtual Retinal Display (VRD) 2. Shared Space 3. Learning in Virtual Environments 4. PAIN MAN 5. Virtual Motion Controller 6. Interactive VRD 7. Virtual Pilot 8. Greenspace 9. Virtual Chess 10. Starship 11. New Media 11. Expert Surgical Assistant 13. Tactile Augmentation 13. Geoscientific Visualization 13. Motion Sickness 16. FLIGHT 17. Blocksmith 18. SS Working Group 19. Parkinson's Project 20. SS Working Group 21. Driving Simulator 22. Two-Handed User Interface 22. Motion Sickness 24. Wearable Interfaces 25. Virtual Classroom 26. Situation Awareness 27. CEDeS Lab 27. Endoscopic Surgery Simulator 27. Virtual Playground 27. Virtual Reality Toolkit 31. Engineering Study of an Endoscope Design 31. Virtual Mirrors 31. Starship 31. Human-Computer Symbiote 31. Virtual Chess 36. 4d mouse 36. collaborative mixed reality 36. VR Interaction Techniques 39. Collaboration through Wearables 39. Phobia Desensitization 41. VRD 41. architecture and virtual reality 41. Multimodal Interfaces 41. situation awareness 41. Medical Robotic Interfaces 46. VRD Emulator 46. laparoscopic surgical simulator 48. Flicker Sensitivity 48. knowledge base project 48. / 48. self-motion perception 52. Design for a Low Vision Aid 52. PRISM 54. LIMIT 54. Interface Sickness 54. design for a low vision aid using a scanned laser display 57. visual-inertial nulling: cross-over asymmetry 57. design for a low vision aid 57. Measures for Presence 60. functional effects of refractive surgery on driving performance U Washington Virtual Reality Projects

Progress on Visual Display The Virtual Retinal Display (VRD) team has been focused on developing improvements to the current prototype systems and on creating the parts needed for future prototypes. The VRD, based on the concept of scanning an image directly on the retina of the viewer's eye, was invented at the HIT Lab in The development program began in November 1993 with the goal of producing a full color, wide field-of- view, high resolution, high brightness, low cost virtual display.

Progress on Movement Sensing For some applications, a hands-free, body-operated walking interface is ideal; the UW HIT Lab has been developing prototypes of "sufficient-motion" interfaces, which allow the user to interact by using a subset of the real-world kinesthetic inputs. Though the ranges of motion are less than full, these inputs are sufficient to convince the user that he or she is moving in the virtual world. Development of these interface devices is called the Virtual Motion Controller (VMC) Project.

Virtual Motion Controller The HIT Lab's VMC working prototype measures body position over the working surface with an arrangement of four weight sensors The curved working surface provides important feedback to the user about his or her physical location, and therefore body locomotion input to the device.

Progress on Cooperative Augmented Reality The Shared Space interface demonstrates how augmented reality, the overlaying of virtual objects on the real world, can radically enhance face-to-face and remote collaboration. For remote collaboration, system allows life-sized live virtual video images of remote user to be overlaid on the local real environment, supporting spatial cues and removing the need to be physically present at a desktop machine to conference. computer vision techniques are used to precisely register virtual images with physical objects, extending the currently popular "Tangible Interface" metaphor. work in the context of a collaborative card-game application that allows face-to-face and remote users to collaboratively interact with each other and virtual animations.

Science Fiction? Most of this stuff won’t be in an office near you next year … But the future comes quickly in computer science !

End Chapter 6