1. First Principles in Information Visualization Design Mary Czerwinski Microsoft Research

2. 6/20/2015Info Vis 2001--Microsoft Research2 Colleagues Colleagues in this area of research at Microsoft include: –George Robertson, –Daniel Robbins, –Maarten van Dantzich –Eric Horvitz Numerous psychologists, researchers, designers and visionaries, e.g.,Ben, Robert

3. 6/20/2015Info Vis 2001--Microsoft Research3 Roadmap What is the information problem? Call to action—”Toward a true science…” Theories and Principles –Attention and Spatial Displays Preattentive features 2 & 3D depth cues 3D, cognitive maps and navigation Visual Momentum –2 & 3D Visualization Examples Do they exemplify design from 1 st principles? Concluding Remarks

4. 6/20/2015Info Vis 2001--Microsoft Research4 Toward a Science of Information Visualization… A scientific approach to information visualization requires –Theory of perception and cognition; “cognition in the wild” combined with the theory of computer graphics Has this community paid adequate attention to the former theoretical perspectives? How to begin?

5. 6/20/2015Info Vis 2001--Microsoft Research5 Basic Proposition Details of an IV create a basic orientation to data & the functionality provided to the user Properties of the perceptual system provide constraints on IV design Theories from cognitive science can be applied in recognition of constraints 3D visualizations may invoke perceptual responses different from those of 2D

6. 6/20/2015Info Vis 2001--Microsoft Research6 Design from 1 st Principles Phenomena: What perceptual, mental and social phenomena are potentially relevant? Measurement: How can those phenomena be quantified to discover general patterns, principles, or laws? Theories: What formal theories can be developed to predict cognitive behavior based on those principles? Design: How can designers make use of the theories and principles to produce effective visualizations? Evaluation: How can principles and designs be systematically tested? Can’t do it all today: focus on attention & navigation

7. 6/20/2015Info Vis 2001--Microsoft Research7 Focus on Visualization…Why? Increasing amount of data is becoming available for better or worse Web has increased our ability to maintain awareness of and inspect vast amounts of data peripherally We need to make sense of it all yet do work!

8. 6/20/2015Info Vis 2001--Microsoft Research8 Human Visual Perception Important to look at what’s known about human perception and attention—depends on task Humans have remarkable perceptual abilities: –to scan, recognize, and recall images rapidly –to automatically detect patterns and changes in size, colour, shape, movement, or texture Textual interfaces require cognitive effort to understand their information content Information visualization seeks to use design to offload cognitive work to the perception system

9. 6/20/2015Info Vis 2001--Microsoft Research9 Preattentive Processing We know that a limited set of visual properties are processed preattentively, without the need for focused attention (<250 msec) These features can be used to highlight important IV characteristics Experiments in psychology have used these features to perform the following preattentive visual tasks: –target detection, detect the presence or absence of a "target“ –boundary detection, detect a texture boundary between two groups –counting, count or estimate the number of elements in a display

10. 6/20/2015Info Vis 2001--Microsoft Research10 2D & 3D Pop-Out Features Line (blob) orientation, Julesz & Bergen [1983]; Wolfe et al. [1992] Length, Triesman & Gormican [1988] Width, Julesz [1985] Size, Triesman & Gelade [1980]curvatureTriesman & Gormican [1988] Number, Julesz [1985]; Trick & Pylyshyn [1994] Terminators, Julesz & Bergen [1983] Intersection, Julesz & Bergen [1983] Closure, Enns [1986]; Triesman & Souther [1985] Color (hue), Nagy & Sanchez [1990, 1992]; D'Zmura [1991];Kawai et al. [1995]; Bauer et al. [1996] Intensity, Beck et al. [1983]; Triesman & Gormican [1988] Flicker, Julesz [1971] Direction of motion, Nakayama & Silverman [1986];Driver & McLeod [1992] Binocular lustre, Wolfe & Franzel [1988] Stereoscopic depth, Nakayama & Silverman [1986] 3-D depth cues, Enns [1990] Lighting direction, Enns [1990]

11. 6/20/2015Info Vis 2001--Microsoft Research11 Preattentive Processing

12. 6/20/2015Info Vis 2001--Microsoft Research12 Conjunction Search (a)(b)

13. 6/20/2015Info Vis 2001--Microsoft Research13 3D Preattentive Features (Enns & Rensink, 1990)

14. 6/20/2015Info Vis 2001--Microsoft Research14 Motion and Depth (Nakayama & Silverman, 1986) Demonstrated empirically that motion and stereoscopic depth are preattentive Stereo and color are conjoined Stereo and motion are conjoined Motion and color are not when conjoined—big one to point out

15. 6/20/2015Info Vis 2001--Microsoft Research15 3D Depth Cues-Object-Centered Many visual cues can be preattentive Object-centered cues (pictorial cues) –Linear perspective- 2 converging lines, assume 2 parallel lines receding in depth –Interposition- We assume that obscured objects are more distant –Height in the plane- We assume that objects higher in our visual field are farther away

16. 6/20/2015Info Vis 2001--Microsoft Research16 3D Object-Centered Depth Cues Cont’d Light and shadow –When objects are lighted from one direction, they have shadows giving clues about orientation & 3D shape Relative size –Of objects known to be the same size, those subtending a smaller visual image on the retina are assumed to be farther away Textural Gradients –When the plane of a texture is oriented toward the line of sight the grain will grow finer at greater distance Proximity-luminance covariance –Objects and lines are brighter as they are closer to us, and continuous reductions in illumination and intensity are assumed to signal receding aerial perspective - fog Motion parallax – objects closer to us appear to move faster

17. 6/20/2015Info Vis 2001--Microsoft Research17

18. 6/20/2015Info Vis 2001--Microsoft Research18

19. 6/20/2015Info Vis 2001--Microsoft Research19 Observer-Centered Cues –Binocular disparity- Images received by the 2 eyes, at slightly different points in space, are disparate. Stimulates unique pairs of corresponding points on the retina. –Convergence- Cross-eyed pattern of the eyes, required to focus on objects as they come closer, is necessary to bring the image on to the detail-sensitive retina of both eyes. –Accommodation- A cue provided to the brain by the eye muscles. The muscles adjust the shape of the lens to bring the image into focus on the retina.

20. 6/20/2015Info Vis 2001--Microsoft Research20 Visual Cues Summary Gestalt cues like similarity, proximity, closure, etc. Not all cues are equally effective from all distances All depth cues work redundantly to provide the same information in a weighted additive fashion (still debated)—some cues more effective at some distances and for some tasks See Cutting & Vishton graph of cues by distance

21. 6/20/2015Info Vis 2001--Microsoft Research21 A Little Theory (I promise!)

22. 6/20/2015Info Vis 2001--Microsoft Research22 Treisman’s Feature Integration Theory

23. 6/20/2015Info Vis 2001--Microsoft Research23 Texton Theory (Julesz, ’81, ’83, ’84)

24. 6/20/2015Info Vis 2001--Microsoft Research24 Similarity Theory (Duncan & Humphreys, 1988) Duncan and Humphreys’ model of preattentive processing Model assumes that search ability depends on both the type of task and the display conditions [Duncan 1989, Muller 1990] –Search time is based on two criteria: T-D similarity and D-D similarity –as T-D similarity increases, search efficiency decreases and search time increases –as D-D similarity decreases, search efficiency decreases and search time increases –T-D similarity and D-D similarity are related; decreasing D-D similarity has little effect if T-D similarity is low; increasing T-D similarity has little effect if D-D similarity is high

25. 6/20/2015Info Vis 2001--Microsoft Research25 Some Preattentive Features Are More Important than Others Callaghan (1990) –Interference asymmetries suggest some preattentive features may be "more important" than others –The visual system reports information on 1 type of feature over and above other features that may also be present in the display –Callaghan's experiments suggest that brightness overrides hue information and that hue overrides shape information during boundary detection

26. 6/20/2015Info Vis 2001--Microsoft Research26 Jacques Bertin (1983) Semiology of Graphics –Systematically classified the use of visual elements to display data and relationships –System consisted of 7 visual variables: Position, Form, Orientation, Color, Texture, Value and Size Joined with visual semantics for linking data attributes to visual elements Note similarity of visual variables to preattentive feature list

27. 6/20/2015Info Vis 2001--Microsoft Research27 Great Bertin (1981) Quote “ Items of data do not supply the information necessary for decision-making. What must be seen are the relationships which emerge from consideration of the entire set of data. In decision-making the useful information is drawn from the overall relationships of the entire set.” My addition: “..which potentially requires several user interactions.” Key issue here: How many interactions are too many for a given task?

28. 6/20/2015Info Vis 2001--Microsoft Research28 Principles—Spatial Info Vis Benefits of Spatial Information Displays –Can leverage spatial memory, comes for free –Can leverage preattentive processing and Gestalt perceptual organization principles See Palmer (1992): proximity, similarity, common fate, good continuation, closure, etc.

29. 6/20/2015Info Vis 2001--Microsoft Research29 Principles—Spatial Displays Problems with Spatial Displays –Vincente, Hayes & Williges (1987) - spatial ability was strongest predictor of performance in spatial db retrieval, with participants of lower spatial abilities most likely to get lost Spatial organization of an information space should match the user’s mental organization Need to note how subjectively related 2 pieces of information are—if high cognitive similarity, information should be spatially closer together Programs like Pathfinder and KNOT can be used to judge similarity of terms/features/functions

30. 6/20/2015Info Vis 2001--Microsoft Research30 Transition Slide…2D-3D Now we’ll focus primarily on 3D IV and navigation, but much applies to 2D as well If you’ve got 3D data to display, users will likely have to navigate it How to balance user nav with information presentation is a research topic….like query refinement, average users might not like to do the work required for rich data exploration

31. 6/20/2015Info Vis 2001--Microsoft Research31 Why 3D? Best for leveraging spatial memory Problem with 2D, linear time-based data –Extent of the time domain is usually large relative to the number of projects being tracked (Robertson, Card & Mackinlay, 1993). –People aren’t that good at memory for time… Ziegarnik effects, biases, etc. 2D v. 3D performance advantage research is inconclusive; users generally prefer 3D Hardware can now accommodate Has large “wow!” factor, users love it

32. 6/20/2015Info Vis 2001--Microsoft Research32 Principles—3D Vis Techniques 3D visualization techniques have the greatest advantage in tasks requiring the use of the large or entire data sets But…use of a fully ego-centered frame of reference to show a 3D space can distort our perception of virtual space –Appropriate use of stereopsis (Wickens, Merwin & Lin, 1994) or monocular cues (Brown & Gallimore, 1995; Gillan, 2000) can help reduce these problems –Use of a surface mesh to connect data points in 3D also provides small accuracy advantages in some cases (Liu & Wickens, 1992) –The principle of visual momentum (Woods, 1984) or nav guildelines (following) should be followed as the user navigates the info space –Leverage spatial memory (it comes for free)

33. 6/20/2015Info Vis 2001--Microsoft Research33 Visual Momentum—Woods (’84) Visual Momentum (Woods, 1984): borrowed from cinematographic techniques Neale (1996 & 1997) has empirically validated the success of using these techniques during navigation tasks in complex, VEs –Use consistent representations –Use graceful transitions (e.g., dynamic rotations of cone trees when selecting a new node of interest) –Highlight anchors—“You are here” maps (Levine, 1982) do this –Display continuous world maps, always from a preferred (your current) perspective; games excel at this

34. 6/20/2015Info Vis 2001--Microsoft Research34 Mental or Cognitive Maps Essential for wayfinding Term proposed by Tolman to describe studies with rats in mazes (1948) Cognitive or spatial maps are products of psychological processes of acquisition, coding, storage, recall, and decoding information about locations and attributes of phenomena in spatial environment (Downs & Stea) Research shows internal form probably hierarchical (Stephens, ‘76; Hirtle & Jonides, ‘85)

35. 6/20/2015Info Vis 2001--Microsoft Research35 Types of Spatial Knowledge Users have three types of spatial knowledge, acquired sequentially for a domain (Thorndyke and Hayes-Roth) – landmark knowledge - recognition of familiar spatial features (0-D) – route knowledge - ability to follow a known path to a destination; can infer new landmarks (1-D) – survey knowledge - map-like knowledge of a spatial domain; can infer new routes

36. 6/20/2015Info Vis 2001--Microsoft Research36 Cognitive Maps (Insfield) Experience with a route influences its representation and priming effects Better memory for routes with many salient landmarks Directly experienced representations align better and are more flexible than “secondary” ones (derived from pictures or maps)

37. 6/20/2015Info Vis 2001--Microsoft Research37 Principles—Navigation (Vinson, CHI ’99) Essential that VEs contain landmarks –Use all 5 types (paths, edges, districts, nodes, landmarks) –Make your landmarks distinctive (significant height, complex shape, bright exterior, large, visible signs, well maintained and expensive materials, free standing, surrounded by landscaping, or unique exterior color, texture—preattentive features) –Use concrete objects, not abstract –Landmarks should be visible at all scales

38. 6/20/2015Info Vis 2001--Microsoft Research38 Principles—Navigation (Vinson, CHI ’99) A landmark should be easy to distinguish from other landmarks and objects –The sides of a landmark must differ from each other (going and coming) –Landmark distinctiveness can be increased by placing other objects nearby –Landmarks must carry a common element to distinguish them as a group from data elements Place landmarks on major paths and path junctions –Arrange paths and edges to form a grid –Align the landmarks’ main axes with the path/edge grid’s main axes –Align each landmark’s main axes with those of the other landmarks

39. 6/20/2015Info Vis 2001--Microsoft Research39 So What? We talked about the attentional and spatial cognition phenomena Info Vis designers need to know about Do they matter for the design of novel information visualizations?

40. 6/20/2015Info Vis 2001--Microsoft Research40 Examples & 1 st Principles How many of the IV community’s designs follow 1 st principles? Posing the question, suggesting design alternatives No one design has been truly born out of 1 st principles successfully yet Need a prescriptive way to map principles to designs by task (e.g., Foltz’ talk)

41. 6/20/2015Info Vis 2001--Microsoft Research41 Perspective Wall (Copyright © by the ACM)

42. 6/20/2015Info Vis 2001--Microsoft Research42 Table Lens (Rao & Card, 1994)

43. 6/20/2015Info Vis 2001--Microsoft Research43 Lifestreams (Freeman & Fertig, 1995 (Copyright ©by the ACM) )

44. 6/20/2015Info Vis 2001--Microsoft Research44 Hierachy Vis--TreeMap (Johnson & Shneiderman, 1991)

45. 6/20/2015Info Vis 2001--Microsoft Research45 Tree Maps (Wattenberg, 1999)

46. 6/20/2015Info Vis 2001--Microsoft Research46 Hyperbolic Browser (Lamping & Rao,1994) Space-filling focus+context technique, see several levels of the hierarchy at once Layout on hyperbolic plane, which is then mapped to the unit disk. Each child places its children in a wedge of space.

47. 6/20/2015Info Vis 2001--Microsoft Research47 Cone Trees—Robertson et al., 1991)

48. 6/20/2015Info Vis 2001--Microsoft Research48 Focus + Context Examples Fisheye Views –Geometric distortion like a fisheye lens or magnifying glass over the area of interest (Furnas, 1981; 1986)

49. 6/20/2015Info Vis 2001--Microsoft Research49 Visual Cues Leveraged by Fisheyes Full detail of item of interest, less info about items further away Closer items brighter, distant items are “grayed out” Fisheye views aid user performance (Hollands, Carey, Matthews & McCann, (1988) Fisheye adds egocentricity to a exocentric display; more effective in large info spaces (Vincow & Wickens, 1998) Especially useful in 3D environments Need to investigate acceptable distortion

51. 6/20/2015Info Vis 2001--Microsoft Research51

52. 6/20/2015Info Vis 2001--Microsoft Research52 MSR Examples Data Mountain –Leverages spatial memory, landmarks (poor), spatialized audio, animation, shadows, user control of semantic spatial layout and text labeling; bring info to user Task Gallery –3D replacement for Windows shell, based on user task and spatial layout, user annotation –Used 3D cues of depth, transparency, relative size, shadows, landmarks, etc.; constrained navigation

53. 6/20/2015Info Vis 2001--Microsoft Research53 Examples: Multiple Depth Cues Spatial memory Occlusion Animation Relative Size Passive landmarks Spatialized audio Data Mountain

54. 6/20/2015Info Vis 2001--Microsoft Research54 Data Mountain Usability Spatial memory works in virtual environments! 26% faster than IE4 Favorites Spatial memory persists for a long time 2x faster with Implicit Query

55. 6/20/2015Info Vis 2001--Microsoft Research55 Task Management Task = User Activity –Tasks laid out spatially –Simple task switch –Use 3D to engage spatial memory –Distinctive landmarks (not a key axes though) –Distinctive wallpaper for task ID –Constrained nav

56. 6/20/2015Info Vis 2001--Microsoft Research56 Occlusion Selected windows occlude others –Tried moving loose stack (confusing) –Use transparency instead

57. 6/20/2015Info Vis 2001--Microsoft Research57 Shadows and Landmarks Shadows aid in depth perception Landmarks aid in spatial orientation and navigation

58. 6/20/2015Info Vis 2001--Microsoft Research58 Constrained Navigation Users Do Not Get Lost Simple, forward- back navigation On-screen navigation controls (discoverable) Control glances

59. 6/20/2015Info Vis 2001--Microsoft Research59 User Study Results –Four iterative usability tests completed –Numerous usability problems identified and fixed –Very high satisfaction ratings

60. 6/20/2015Info Vis 2001--Microsoft Research60 1 st Principles for IV Design Use distinctive, preattentive visual features, (e.g., boundaries, textures, colors, shapes and animations) to highlight data of interest Make sure non-target data is uniform and not similar to target data for given task (after zoom?) Ensure good navigability; distinctive landmarks, grid the system, overview map Offset user’s cognitive task to a perceptual task –How much to offset is an open question….

61. 6/20/2015Info Vis 2001--Microsoft Research61 Conclusion Challenge: Leverage principles of perception and cognition in information visualization to cross Ben’s “chasm” IV community needs to map principles to design “Natural” information visualization helps: –Some information tasks become automatic –Free up attentional resources for cognitive work –Users are faster, make better decisions or have more fun! –Still leaves a place for creativity and design

62. 6/20/2015Info Vis 2001--Microsoft Research62 Valuable Resources…. Keith Andrews online course materials Marc Green’s article at http://www.ergogero.com/dataviz/dviz0.ht ml http://www.ergogero.com/dataviz/dviz0.ht ml InfoVisualRes at http://www.cs.man.ac.uk/~ngg/InfoViz/Cou rses_and_Tutorials/ http://www.cs.man.ac.uk/~ngg/InfoViz/Cou rses_and_Tutorials/ Thank you!