1. Display Techniques in Information-Rich Virtual Environments Nicholas F. Polys PhD Research Proposal August 18, 2004 Committee: Dr. Doug Bowman, VT Dr. Chris North, VT Dr. Scott McCrickard, VT Dr. Ken Livingston, Vassar College Dr. Don Brutzman, Naval Postgraduate School

2. Display Techniques in Information-Rich Virtual Environments Proposal Outline Problem scope & Statement Background Research Questions & Goals Information Design Dimensions Approach, Method, Measures Experimental Program Significance

3. Display Techniques in Information-Rich Virtual Environments Complex systems typically span multiple scales and involve heterogeneous data types (objects, spatial relations, attributes) Engineers, researchers, and analysts need to access, manage, and understand a wide variety of information and inter- relationships General Problem: Integrated Information Spaces

4. Display Techniques in Information-Rich Virtual Environments Fundamental Data Types Spatial / perceptual data: geometry, colors, textures, lighting Abstract data / world & object attributes: nominal, ordinal, quantitative Temporal data / behaviors: states, dynamics

5. Display Techniques in Information-Rich Virtual Environments Information-Rich Domains Simulation and design applications require spatial/perceptual fidelity and information enhancement: Engineering / CAD Construction / architecture Medicine / biology Science / research Education / training …

6. Display Techniques in Information-Rich Virtual Environments

8. User Problem: Integrated Information Spaces Multiple applications and fragmented views make it difficult to understand the relationships between information types Next generation information interfaces must unify display and interaction spaces for: –Exploration –Search –Comparison and Pattern recognition

9. Display Techniques in Information-Rich Virtual Environments Problem Statement: We lack precise definitions, development tools, and systematic research as to how perception and cognition operate in information-rich interfaces and environments: Combining Virtual Environments and Information Visualizations is currently ad hoc and application-specific There are competing models of Vision and Working Memory that may apply An experimental methodology and theory is required to assess, design, and deliver ‘appropriate’ displays

10. Display Techniques in Information-Rich Virtual Environments Background: Information Psychophysics Edward Tufte, Envisioning Information (1983, 1990) Jaques Bertin, Semiology of Graphics (1983) Donald Norman, Cognitive Engineering (1986) Joseph Goguen, Semiotic Morphisms (2000) Colin Ware, Perception for Design (2003)

11. Display Techniques in Information-Rich Virtual Environments Background: Information Visualization Visual display of abstract information Visual Markers (Cleveland & McGill, 1984; Mackinlay, 1986; Card et al, 1999) Multiple Views (North, 2001; North et al, 2002; Convertino et al, 2003) Zoom-able Interfaces (Bederson et al, 1996, 2000; Woodruff et al, 1998a-c)

12. Display Techniques in Information-Rich Virtual Environments Background: Virtual Environments Visual display of spatial and perceptual data Immersive and Desktop Platforms Conceptual Learning (Salzman et al, 1999) Navigating space (Darken et al, 1996, 2002) Naturalism & Performance (Bowman, 2002; Bowman et al, 2004) Image Plane (Pierce et al, 1997)

13. Display Techniques in Information-Rich Virtual Environments Background: Augmented Reality Enhancing perceptual scenes with additional abstract information Feiner et al. Windows on the World (1993) Bell et al. Dynamic Space Management, View Management (2000, 2001)

14. Display Techniques in Information-Rich Virtual Environments Background: Multimedia & Comprehension Co-references between text and images (Chandler & Sweller, 1990; Faraday & Sutcliffe, 1997, 1998) Task Knowledge Structure (Sutcliffe & Faraday, 1994; Sutcliffe 2003) Meaningful Learning: troubleshooting, redesigning, deriving principles (Mayer, 2002)

15. Display Techniques in Information-Rich Virtual Environments Background: Architecture of WM Componentized WM (Baddeley, 2003) –phonological loop, visuospatial sketchpad, episodic buffer, central executive Individual differences in WM Capacity (Just & Carpenter: 1996) Short and Long term WM (Ericsson & Kintsch: 1995)

16. Display Techniques in Information-Rich Virtual Environments Working Memory (Baddeley, 2003) Central Executive Phonological Loop Episodic bufferVisuospatial sketchpad Visual semantics Episodic LTM Language Fluid system Crystallized system

17. Display Techniques in Information-Rich Virtual Environments Background: Visuospatial WM Capacity 3-5 ‘items’ Functional units & chunking –Objects & features (Vogel et al, 2001) –Visual indices & dynamic feature binding (Saiki, 2003) Subsystems: form & color, space & movement (Logie 1995) Relation to Central Executive (Miyake et al. 2001)

18. Display Techniques in Information-Rich Virtual Environments Background: Interpreting Linegraphs Computational model: Understanding Cognitive Information Engineering [ UCIE] (Lohse, 1991) –similar to GOMS (Card, Moran, & Newell, 1983), and ACT* (Anderson, 1983) Processing –encoding visual patterns –inferring and retrieving functional relations via graph schema –associating / labeling referents

19. Display Techniques in Information-Rich Virtual Environments Proposal: Information-Rich Virtual Environments (IRVEs) We need to understand: How spatial/perceptual information and abstract information can be combined and displayed What makes the combinations effective What makes them usable and How users think and act when using them

20. Display Techniques in Information-Rich Virtual Environments Research Question for IRVE Information Design: Where and How should enhancing abstract information be displayed relative to its perceptual referent so that the respective information can be understood together and separately?

21. Display Techniques in Information-Rich Virtual Environments Research Goals: To understand how the respective design techniques of Virtual Environments and Information Visualization can be combined and balanced To enable stronger mental associations between spatial and abstract information while preserving the models of each type of information.

22. Display Techniques in Information-Rich Virtual Environments Research Goal 1: Define a theoretical framework for Information-Rich Virtual Environments (IRVEs) as the solution to the problem of integrated information spaces Results: –Terminology –Definitions –Bowman et al, 2003

23. Display Techniques in Information-Rich Virtual Environments Research Goal 2: Enumerate the design space for IRVE tasks and display techniques through Usability Engineering process and literature review Results: –IRVE Task Space, Information Design Space, Interaction Design Space –Polys & Bowman, 2004; Polys et al, 2004c

24. Display Techniques in Information-Rich Virtual Environments Research Goal 3: Prototype information-rich application interfaces to identify problems and generate hypotheses regarding optimal IRVE information design per task Results: –IRVE Display Prototypes –Claims Analysis 1 (heuristic evaluation, informal user studies, pluralistic walkthroughs) –Polys, 2003; Polys et al, 2004a, 2004b

25. Display Techniques in Information-Rich Virtual Environments Research Goal 4: Describe IRVE display configurations in a concise XML DTD and Schema and use this display description to generate runtime components Results [pending]: –IRVE Testbed

26. Display Techniques in Information-Rich Virtual Environments Research Goal 5: Provide a quantitative basis by which to characterize the density and distribution of information in an IRVE Results [pending]: –Assessment method for IRVE information sets

27. Display Techniques in Information-Rich Virtual Environments Research Goal 6: Identify tradeoffs and guidelines for the IRVE display design space using –Empirical usability evaluations and –Metrics for individual cognitive differences Results [pending]: –Empirical data relating design techniques, information sets, and user performance

28. Display Techniques in Information-Rich Virtual Environments Research Question re-phrased: For the combination of abstract and perceptual visualizations in IRVEs, association should be maximized and interference (such as occlusion and crowding by layouts) should be minimized… What are the best ways to manage layout space and association cues so that perceptual and abstract information can be understood together and separately?

29. Display Techniques in Information-Rich Virtual Environments Goals & Results Summary Survey IRVE information design space Prototype IRVE display techniques and applications Understand and quantify how IRVE information sets vary Enumerate information design heuristics and guidelines for IRVEs

30. Display Techniques in Information-Rich Virtual Environments IRVE Information Design Challenges Visibility Legibility Association Occlusion Aggregation

31. Display Techniques in Information-Rich Virtual Environments Abstract information design parameter Psychological process Usability impact Visual attributes: - color - fonts - size - background - transparency Perception- Legibility - Readability - Occlusion Layout attributes: - location - association - density Interpretation- Relating abstract and perceptual information - Conceptual categories & abstractions - Occlusion Aggregation: - level of information encoding - type of visualization Making Sense- Comparison & Pattern Recognition - Effectiveness - Satisfaction IRVE Information Design Dimensions

32. Display Techniques in Information-Rich Virtual Environments Layout Space (Locations) The layout space of abstract information in IRVEs is described by the coordinate system it is resident in: Object World User Viewport Display

33. Display Techniques in Information-Rich Virtual Environments Object Space Object space is relative to an object’s location in the environment (e.g. Semantic Objects).

34. Display Techniques in Information-Rich Virtual Environments World Space World space is relative to an area, region, or location in the environment.

35. Display Techniques in Information-Rich Virtual Environments User Space User space is relative to the user’s location but not their viewing angle.

36. Display Techniques in Information-Rich Virtual Environments Viewport Space Viewport space-is the image plane where HUDs or overlays may be located.

37. Display Techniques in Information-Rich Virtual Environments Display Space Display layout space where abstract visualizations are located outside the rendered view in some additional screen area.

38. Display Techniques in Information-Rich Virtual Environments Association The association dimension of IRVE information design is delineated by the Gestalt principles:

39. Display Techniques in Information-Rich Virtual Environments Approach (Goals 4-6) Develop methods to describe and generate IRVE display components Develop quantitative methods to characterize IRVE datasets Use the above to control testbed environment and stimuli and run empirical usability evaluations

40. Display Techniques in Information-Rich Virtual Environments IRVE Testbed Configuration Syntax The composition of an IRVE display will be defined under an XML DTD and Schema. The DTD and Schema will provide syntactic and semantic production rules for IRVE display spaces. In order to instantiate an IRVE display, the testbed will read the information mapping configuration from the XML and generate the X3D code for that IRVE.

41. Display Techniques in Information-Rich Virtual Environments IRVE Stimuli & Metrics Exocentric vs. Egocentric: Exocentric metrics have an advantage in that they may be employed on a data set independent of the user or rendering; egocentric metrics may be more advantageous to find dynamic, perspective-specific layouts. This research proposal will investigate exocentric metrics for IRVE data characteristics such as the quantity, density, and distribution of abstract information in the VE.

42. Display Techniques in Information-Rich Virtual Environments IRVE Layout & Association Dimensions Common Region Proximity Connected ness SimilarityCommon Fate Object xxxxx Worldxxxxx Userxxxxx Viewportxxxxx Displayxxxxx Experiment 1 Experiments 2 & 3

43. Display Techniques in Information-Rich Virtual Environments Evaluation Method The usability experiments will have a factorial, within-subjects, counterbalanced design Cognitive battery test will be used control between- subject variance and may provide insight into how display techniques work with different visuospatial abilities. These experiments will allow us to explore tradeoffs in IRVE information design and identify guidelines and design patterns for integrated information spaces.

44. Display Techniques in Information-Rich Virtual Environments Procedure Subjects from VT population Demographic questionnaire & Cognitive battery tests Training session for spatial navigations Experimental Protocol & dependent measures

45. Display Techniques in Information-Rich Virtual Environments Measures Time to Completion Correctness Satisfaction / Ease of Use User strategy Cognitive Battery

46. Display Techniques in Information-Rich Virtual Environments Perceptual Speed / Closure Flexibility Spatial Visualization Spatial Orientation Cognitive Battery

47. Display Techniques in Information-Rich Virtual Environments IRVE Tasks IRVE Search Tasks require subjects to: find a piece of abstract information based on some perceptual criteria find a piece of perceptual information based on some abstract criteria. IRVE Comparison Tasks require subjects to: compare perceptual attributes of two items with a given abstract criteria compare abstract attributes of two items with a given perceptual criteria

48. Display Techniques in Information-Rich Virtual Environments Experiment 1: Object space layouts can be drawn at a fixed orientation or billboarded to always face the user. Is one technique better than the other for common tasks and navigation modes? Here we will investigate Search and Comparison tasks combined with flying or terrain-following navigation.

49. Display Techniques in Information-Rich Virtual Environments Object space: Billboard vs. Fixed Search & Comparison Tasks Terrain following and Flying navigation Fixed orientations provide additional wayfinding cues Billboarded orientations require less spatial navigation for legibility In these circumstances, is one display technique better that the other?

50. Display Techniques in Information-Rich Virtual Environments Experimental UnitHuman Subject (n = 8)  Tests: Flexibility of Closure, Perceptual Speed, Spatial Orientation, Factor 1 Object space layout Level 1 Billboarded Level 2 Fixed Factor 2Navigation type Level 1 Walk Level 2 Fly Factor 3Task: Search Level 1 Spatial -> Abstract Level 2 Abstract -> Spatial Factor 4Task: Comparison Level 1 2 Spatial items Level 2 2 Abstract items Response VariablesPerformance: Time to Completion, Correctness Satisfaction: Ease of Use 16 conditions

51. Display Techniques in Information-Rich Virtual Environments Experiment 2: Abstract information is associated to perceptual information through different cues in Object and Viewport spaces. How do the Gestalt principles rank in IRVEs? We will examine connectedness, proximity, and common region for Search tasks between these two layout spaces.

52. Display Techniques in Information-Rich Virtual Environments Association in Object vs. Viewport spaces Search Task Relating abstract and perceptual information between layout spaces and association parameters Do depth cues of object space enable a tighter coupling of information types or is the occlusion unneccessary?

53. Display Techniques in Information-Rich Virtual Environments Experimental UnitHuman Subject (n = 8)  Tests: Flexibility of Closure, Perceptual Speed, Spatial Orientation, Factor 1 Layout space Level 1 Object Level 2 Viewport Factor 2Gestalt principle Level 1 Connectedness Level 2 Proximity Level 3 Common Region Factor 3Task: Search Level 1 Spatial -> Abstract Level 2 Abstract -> Spatial Response VariablesPerformance: Time to Completion, Correctness Satisfaction: Ease of Use 12 conditions

54. Display Techniques in Information-Rich Virtual Environments Experiment 3: The theory behind IRVEs must account for a wide range of content. How does content determine an appropriate layout space if at all? We will examine Object and Viewport spaces for Comparison tasks in different environments. We will factorially compare environments of high and low density and distribution.

55. Display Techniques in Information-Rich Virtual Environments Density and Distribution thresholds for Object vs. Viewport spaces Comparison Task Can environment characteristics predict advantageous display techniques and ground design guidelines?

56. Display Techniques in Information-Rich Virtual Environments Experimental UnitHuman Subject (n = 16)  Tests: Flexibility of Closure, Perceptual Speed, Spatial Orientation, Digit Span Factor 1 Layout Space Level 1 Object Level 2 Viewport Factor 2Task: Comparison Level 1 2 Spatial items Level 2 2 Abstract items Factor 3Information Density Level 1 Low Level 2 High Factor 4Information Distribution Level 1 Homogenous Level 2 Clustered Response VariablesPerformance: Time to Completion, Correctness Satisfaction: Ease of Use 12 conditions

57. Display Techniques in Information-Rich Virtual Environments Significance Establishes a research area combining Information Visualization and Virtual Environments Investigates how search and comparison tasks can be supported with IRVE display techniques Relates display techniques to models of perception and working memory

58. Display Techniques in Information-Rich Virtual Environments Significance (cont’d) Supports and extends international standards technology (e.g. X3D, XML) Provides tools for IRVE developers Provides guidelines for IRVE designers Environment characteristics and display guidelines may also transfer to Augmented Reality

59. Display Techniques in Information-Rich Virtual Environments Timeline & Milestones September 2004IRVE Task Space Enumeration IRVE Data Metrics October 2004IRVE Display Language Testbed completed November 2004Experiment 1 January 2004Experiment 2 March 2005Experiment 3 Fall 2005Research Defense Spring 2006Dissertation Defense

60. Display Techniques in Information-Rich Virtual Environments Thank You Questions & Discussion npolys@vt.edu http://www.virtuworlds.com/X3D/npolys_PhD_Proposal.doc http://www.virtuworlds.com/media_02/index.html#wares