What is a 3D User Interface / History and Roadmap of 3D UI’s Kurt Andersen

Important Terms Human-Computer Interaction (HCI) – Field of study that examines all aspects of the interplay between people and interactive technologies User Interface (UI) –The medium in through which we interact with computers Input Device – Hardware device allowing communication from user to computer Output Device – Hardware device allowing communication from computer to user Degrees of Freedom (DOF) – Number of independent dimensions of the motion of a body

More Important Terms Interaction Technique – Method allowing a user to accomplish a task via the UI Usability – Characteristics of an artifact that affect the user’s use of the artifact. i.e. ease of use, user task performance, or user comfort User Experience (UX) – broad concept encompassing a user’s entire relationship with an artifact, including usability and emotional factors UX Evaluation – Assessment or measurement of aspects of the UX 3D Interaction – User tasks performed in a real or virtual 3D spatial context.

Even More Terms (I Promise This is it) 3D UI – A UI in 3 dimensions (wow!) Virtual Environment (VE) – A synthetic 3D world seen from a first person point of view Virtual Reality (VR) – Interchangeable with VE. Augmented Reality (AR) – Real world environment with synthetic objects Mixed Reality (MR) – Combination of VR and AR Ubiquitous Computing (UbiComp) – Notion that computing devices and infrastructure is mobile or scattered throughout the real world. Telerobotics – Ability to control a robot that is geographically separated from the user

Components of a User Interface Input Devices such as but not limited to mouse or touchscreen Output devices such as a monitor or some sort of display screen Interaction techniques such as drag and drop or pinch to zoom Interface Widgets like pull-down menus, or UI metaphors like Windows, Icons, Menus, Pointer (WIMP)

3D UI Applications (more in depth on these later) Design and prototyping Heritage and tourism Gaming and Entertainment Simulation and training Education Art Visual data analysis Architecture and construction Medicine and psychiatry Robotics

3D interaction is relevant to real-world tasks It gives us a sense of actually being there Use natural skills to perform interactions More direct and immediate than a 2D UI Shorter cognitive distance between action and response

Technology in 3D UI’s Input Output Spatial input devices 3D pointing devices Whole hand devices allowing gestural input Output Stereoscopic projection displays High-resolution head mounted display Spatial audio systems Haptic Devices

Why 3D UI’s are Difficult Should be simple because we live in a 3D world Our world gives us a lot more feedback for every action Constructing similar cues in a virtual environment to mimic the real world Restricted to less senses when in a virtual environment

3D UI’s are Straightforward or Lack Usability For entertainment applications it is easy to evaluate its use Head turning is the only interaction (for VR films) For other applications i.e. simulations, scientific visualization, modeling, or training Difficult to design and evaluate Leads to a lack of usability or poor user experience

Brief History of GUI’s Pre 1980 – All interaction was done through the command line Early 1980’s – inexpensive graphics rasterizer displays Utilities to interact with graphical display such as mouse Xerox Star one of the first computers with GUI Birth of GUI was the start of research into HCI Draws on knowledge of perception, cognition, linguistics, human factors, ethnography, sociology, graphic design, industrial design, and other areas.

Brief History of VR 1960’s – Ivan Sutherland develops a head mounted display for simulated reality. Capable of both VR and AR. Tracked head position Precise but cumbersome mechanical head tracker 1980’s-1990’s – VR systems became more practical 3D stereoscopic graphics on mini CRT displays Interaction devices such as the VPL DataGlove Early thoughts on how to enable users to interact with the world through tracked hands and tools.

Combination of 3D and 2D GUIs Due to early research in GUI’s for personal computers, the field was already reasonably matured as research geared towards 3D GUIs became more relevant The application of existing HCI knowledge to 3D interfaces helped improve their usability

Roadmap to 3D UI’s (page 15 for a better look)

Theoretical Background Comes from work done in: human spatial perception, cognition, action, visualization design, HCI and UI design. Users are acting in a real and/or virtual 3D space; therefore, psychology and human factors knowledge about spatial perception, cognition, human navigation, movement, and manipulation in a 3D space contain critical information for the design of 3D UI’s

Technological Background Interactive 3D Graphics – images rendered in real time so users can interact with them directly Fast line polygon rendering algorithms, texture mapping, real-time lighting methods, dedicated graphics processor, algorithms for drawing stereoscopic images, shader algorithms, high level graphics software toolkits(Unity) 3D Display Devices – stereoscopic displays for desktop computers, walkaround 3D displays, head worn displays for VR/AR, 3D spatial sound, projection based spatially immersive displays 3D input devices – Position and orientation tracking sensors, posture and gesture sensing devices (mostly for hands), multiple DOF joysticks

Technological Background Continued Simulator systems – precursor to VR. Large systems with an immersive display of 3D graphics. Used for many applications such as flight, space, or driving simulators. Telepresence Systems – allows users to feel as if they are in a real- world location. Uses a combination of sensors on a remote with a display and interactive controls. Similar to VR but uses real world data rather than synthetic data. VR/AR systems – Immersive synthetic environment/projecting synthetic objects into the real world. VPL’s HMD and DataGlove, Cave Automatic Virtual Environment, Spatial AR, handheld mobile AR on smartphones

Popular Media There has been a lot of influence from media sources such as sci-fi movies, fiction books, and other forms of media These examples can be seen in: Star Trek: The Next Generation(Holodeck), Iron Man 2(3D interaction with data based on in air gestures), Disclosure (VR system with natural physical movement and language interaction)

3D UI Subareas Each of the following will be gone over more in depth later in the book. These are bread and butter of what 3D UI’s are comprised of. Interaction Techniques for Universal Tasks - Interaction Techniques for Composite and Application-Specific Tasks, 3D UI Widgets and Tools, 3D Interaction Techniques using 2D Devices 3D UI Design Approaches - Hybrid Interaction Techniques, Two- Handed Interaction, Multimodal Interaction 3D UI Software Tools -Specialized and Development Tools for 3D Applications, 3D Modeling Tools 3D UI Evaluation - Evaluation of Devices, Interaction Techniques, Complete 3D UIs or Applications, Methodologies, Study of Phenoomena Particuilar to 3D UIs

Areas Impacted by 3D UIs List mentioned back on slide 6 Education – Newton’s Laws can be experimented with in a highly interactive virtual world Visual Data Analysis – Rather than just observing numbers, we are able to visually see data and pick out points to analyze better. Medicine and Psychiatry – 3D applications are used for telemedecine(remote diagnosis and treatment), or 3D visualizations of MRI’s.

Standards of 3D UIs As of now there are no documented standards. However, there are some areas that have some standardization of a 3D UI On Interactive 3D Graphics - World Wide Web Consortium has international standards for many aspects of the internet for interactive 3D graphics. On HCI – What metrics should be used to study the user experience? Generally we measure speed, accuracy. Satisfaction, and perceived ease of use. In 3D we also need to consider physical comfort and presence.

What Else Will Be Discussed in This Book? Will focus on the content that is specific to 3D UIs. Basically more in depth of the aforementioned sub areas Will cover presence/cybersickness, design or workings of various devices, graphics algorithms and techniques for rendering 3D environments and 3D toolkits/APIs. Numerous case studies will be covered throughout the book as well.

Conclusion Covered what a 3D UI is Covered a brief history of 3D UIs Briefly touched on all the subareas with the design of 3D UIs Talked about what we can expect from this book