Tatyana Povalyaeva Important Aspects of the Annotation of Data in Virtual Environments Seminar

Introduction Scientific Data Analysis Interaction Visualization Conclusion & Outlook Tatyana PovalyaevaImportant Aspects of the Annotation of Data in Virtual Environments 2 Overview

Introduction Scientific Data Analysis Interaction Visualization Conclusion & Outlook Tatyana PovalyaevaImportant Aspects of the Annotation of Data in Virtual Environments 3 Overview

Virtual environments: a copy of reality as realistic as possible (simulation) existed in the past or do not yet exist cannot be entered or perceived in a real world cannot be real in principal (artificial reality) 4 Introduction – Virtual Reality Tatyana PovalyaevaImportant Aspects of the Annotation of Data in Virtual Environments

Virtual reality: is a computer-simulated, virtual, 3D interactive world allows for the immersion of the user in the generated virtual world is achieved by several components:  a high-performance computer graphics system  a stereoscopic projection system  user-centered projection (tracking)  3D interaction 5 Introduction – Virtual Reality Tatyana PovalyaevaImportant Aspects of the Annotation of Data in Virtual Environments

6 Introduction – Virtual Reality Hardware Tatyana PovalyaevaImportant Aspects of the Annotation of Data in Virtual Environments head-mounted display (HMD ) room-mounted display (e.g. CAVE)

Tracking: determines position and orientation of the user mechanical electromagnetic optical etc. 7 Introduction – Virtual Reality Hardware Tatyana PovalyaevaImportant Aspects of the Annotation of Data in Virtual Environments

Interaction devices: 8 Introduction – Virtual Reality Hardware Tatyana PovalyaevaImportant Aspects of the Annotation of Data in Virtual Environments flystick space mouse force-feedback haptic device

Scientific data visualization & exploration: has applications in engineering and natural sciences represents the data obtained through the simulation of technical, physical, and natural processes in a visual form enables scientists to  gain insights into the data  test hypotheses  discover unexpected phenomena  present the results in a clear and effective way 9 Introduction – Visualization Tatyana PovalyaevaImportant Aspects of the Annotation of Data in Virtual Environments

Scientific data visualization & exploration: 10 Introduction – Visualization Tatyana PovalyaevaImportant Aspects of the Annotation of Data in Virtual Environments

Introduction Scientific Data Analysis Interaction Visualization Conclusion & Outlook Tatyana PovalyaevaImportant Aspects of the Annotation of Data in Virtual Environments 11 Overview

Scientific data analysis is a complex and ongoing process Challenges in virtual environments:  how to make notes  display systems  input devices The possibility to interactively annotate the data is very important 12 Scientific Data Analysis Tatyana PovalyaevaImportant Aspects of the Annotation of Data in Virtual Environments

Annotations: are an accumulation of metadata that combine to create a source of information which references to a particular object within the virtual environment can contain text, images or voice 13 Annotations Tatyana PovalyaevaImportant Aspects of the Annotation of Data in Virtual Environments

Integration into a virtual reality system Interaction with the system Suitable visualization technique 14 Annotations - Requirements Tatyana PovalyaevaImportant Aspects of the Annotation of Data in Virtual Environments

Introduction Scientific Data Analysis Interaction  Selection  System Control  Metadata Input Visualization Conclusion & Outlook Tatyana PovalyaevaImportant Aspects of the Annotation of Data in Virtual Environments 15 Overview

Main tasks: Selection of objects and annotations System control and transmission of commands to the system Input of different types of metadata into annotations 16 Interaction Tatyana PovalyaevaImportant Aspects of the Annotation of Data in Virtual Environments

Introduction Scientific Data Analysis Interaction  Selection  System Control  Metadata Input Visualization Conclusion & Outlook Tatyana PovalyaevaImportant Aspects of the Annotation of Data in Virtual Environments 17 Overview

Selection involves acquiring or identifying one or more particular objects or annotations from the entire set of the available objects Possible problems with:  small, moving or occluded objects  objects that are close to each other  dynamic virtual environment 18 Interaction - Selection Tatyana PovalyaevaImportant Aspects of the Annotation of Data in Virtual Environments

19 Interaction - Selection Techniques Tatyana PovalyaevaImportant Aspects of the Annotation of Data in Virtual Environments

The user points at objects with a virtual ray Simple and efficient selection technique Allows to select objects at any distance Disadvantages:  can be hard if the object is occluded or moving  jittering 20 Selection – Ray Casting Tatyana PovalyaevaImportant Aspects of the Annotation of Data in Virtual Environments

Uses a conic selection volume and an object scoring function Advantages:  reduced jittering  interactive object disambiguation 21 Selection – IntenSelect Tatyana PovalyaevaImportant Aspects of the Annotation of Data in Virtual Environments

Introduction Scientific Data Analysis Interaction  Selection  System Control  Metadata Input Visualization Conclusion & Outlook Tatyana PovalyaevaImportant Aspects of the Annotation of Data in Virtual Environments 22 Overview

System control provides a way to issue commands to the system to perform actions like creating or modifying the annotations System control techniques:  graphical menus  menus accessed via voice commands  command sets accessed via gesture  virtual objects with an implicit function or mode 23 Interaction – System Control Tatyana PovalyaevaImportant Aspects of the Annotation of Data in Virtual Environments

The 2D menu is placed on a tracked physical surface (tablet) The user interacts with the menu components using a tracked stylus This technique is faster and more precise as the floating pull- down menus 24 Interaction – Pen and Tablet Tatyana PovalyaevaImportant Aspects of the Annotation of Data in Virtual Environments

A flexible cloth glove with a conductive cloth sewn integrated in each of the fingers When two or more fingers come into contact with one another, a signal is sent back to the system allows for the system control without direct eye contact with the input device Tatyana PovalyaevaImportant Aspects of the Annotation of Data in Virtual Environments 25 Interaction – Pinch Gloves

26 Interaction – Pie-Menus Tatyana PovalyaevaImportant Aspects of the Annotation of Data in Virtual Environments A circular menu that is made of several slices around a center The pointer can be moved in different radial directions selection of items depends not on distance but on direction In general faster and more reliable than linear menus

Introduction Scientific Data Analysis Interaction  Selection  System Control  Metadata Input Visualization Conclusion & Outlook Tatyana PovalyaevaImportant Aspects of the Annotation of Data in Virtual Environments 27 Overview

Textual information is an important part of annotations Problems in immersive virtual environments:  devices such as conventional keyboards are unavailable  input of textual content requires other devices and techniques than the conventional text input There exists a wide range of possible input techniques, e.g.:  keyboard-based  pen-based  speech-based 28 Interaction - Metadata Input Tatyana PovalyaevaImportant Aspects of the Annotation of Data in Virtual Environments

Introduction Scientific Data Analysis Interaction Visualization Conclusion & Outlook Tatyana PovalyaevaImportant Aspects of the Annotation of Data in Virtual Environments 29 Overview

The visualization of an annotation must be properly adapted to its specific content The relation between an annotation and its reference object should be as clear as possible It should be decided:  what metadata have to be visualized and in which way it can be done  which representation form is the most legible  how an annotation will be positioned within the virtual environment  how annotations can be arranged into a layout 30 Visualization – General Aspects Tatyana PovalyaevaImportant Aspects of the Annotation of Data in Virtual Environments

Annotations contain images and textual descriptions Objects in the scene are represented by the axis-oriented bounding rectangles This approach avoids occlusion and allows for the efficient use of free display areas 31 Visualization – Implementation Examples Tatyana PovalyaevaImportant Aspects of the Annotation of Data in Virtual Environments

Visualization – Implementation Examples This method is based on dynamic potential fields The accumulated attractive and repulsive forces are used to determine a set of label placement candidates Not real-time capable 32 Tatyana PovalyaevaImportant Aspects of the Annotation of Data in Virtual Environments

Visualization – Implementation Examples This concept uses the shadow metaphor Functional correlations and textual information are presented on additional projection layers The approach is well suited for compact 3D models 33 Tatyana PovalyaevaImportant Aspects of the Annotation of Data in Virtual Environments

Visualization - Layout Problems Duplication of annotations in a multy-screen display system The use of the whole image space of the immersive virtual environment is not efficient 34 Tatyana PovalyaevaImportant Aspects of the Annotation of Data in Virtual Environments

Visualization – Implementation Examples An interactive annotation system that allows for an automated, real-time capable, and clear annotations layout, which prevents common layout problems Resolves occlusions using visibility volume The approach is based on the Interactive Data Exploration and Annotation framework (IDEA) 35 Tatyana PovalyaevaImportant Aspects of the Annotation of Data in Virtual Environments

Introduction Scientific Data Analysis Interaction Visualization Conclusion & Outlook Tatyana PovalyaevaImportant Aspects of the Annotation of Data in Virtual Environments 36 Overview

37 Conclusion & Outlook Tatyana PovalyaevaImportant Aspects of the Annotation of Data in Virtual Environments Aim: A web interface that allows for a database connection in order to store / load / show / modify annotations outside the virtual environment.

Thank you for your attention! 38 Tatyana PovalyaevaImportant Aspects of the Annotation of Data in Virtual Environments

[BFH01] Blaine Bell, Steven Feiner, and Tobias Höllerer. View Management for Virtual and Augmented Reality. In UIST '01: Proceedings of the 14th Annual ACM Symposium on User Interface Software and Technology, ACM. [BKLP05] Doug A. Bowman, Ernst Kruij, Joseph J. LaViola Jr., and Ivan Poupyrev. 3D User Interfaces: Theory and Practice. Addison-Wesley, [BW01] Doug A. Bowman and Chadwick A. Wingrave. Design and Evaluation of Menu Systems for Immersive Virtual Environments. In VR '01: Proceedings of the Virtual Reality 2001 Conference, page 149, Washington, DC, USA, IEEE Computer Society [CHWS88] J. Callahan, D. Hopkins, M. Weiser, and B. Shneiderman. An Empirical Comparison of Pie vs. Linear Menus. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, ACM. [dHKP05] Gerwin de Haan, Michal Koutek, and Frits H. Post. IntenSelect: Using Dynamic Object Rating for Assisting 3D Object Selection. In Proceedings of the 9th Internationial Immersive Projection Technology Workshop (IPT) and 11th Eurographics Workshop on Virtual Environment (EGVE), [HAS04] Knut Hartmann, Kamran Ali, and Thomas Strothotte. Floating Labels: Applying Dynamic Potential Fields for Label Layout. In Smart Graphics, Springer Verlag, T. Kuhlen: Course on Virtual Reality. Virtual Reality Group, RWTH Aachen University. [PHT+10] Sebastian Pick, Bernd Hentschel, Irene Tedjo-Palczynski, Mark Wolter, and Torsten Kuhlen. Automated Positioning of Annotations in Immersive Virtual Environments. Joint Virtual Reality Conference of EuroVR { EGVE { VEC, [PWBI98] I. Poupyrev, S. Weghorst, M. Billinghurst, T. Ichikawa. Egocentric Object Manipulation in Virtual Environments: Empirical Evaluation of Interaction Techniques. Eurographics, Vol. 17, References Tatyana PovalyaevaImportant Aspects of the Annotation of Data in Virtual Environments