Understanding Visualization through Spatial Ability Differences Maria C. Velez, Deborah Silver and Marilyn Tremaine Rutgers University 2005

2 What this Talk is About ä Different people have been shown to have a lot of trouble with 3D visualizations ä To investigate this issue further, we ran an experiment comparing measured spatial skills to abilities to understand visualizations ä The results suggested key problems individuals were having and ways in which we can make the visualizations understandable by a larger audience

3 Motivation ä Issues in Visualization Understanding Classic 3D visualizations (2D projection and slices) have been found to be suboptimal for tasks like understanding shape and 3D space layout. Examples of conventional visualization displays used in medical and weather imaging

4 Previous Solutions to Visualization Difficulties Improvements have been proposed: Exovis Cube Corner Training methods have been developed: “Pool of water” metaphor used in cutting planes training Improvements are ad hoc rather than theory-based Mental Rotation Test used to study spatial comprehension

5 Focus – What Makes a Visualization Difficult? ä We use differences in human spatial abilities to understand the problems that affect people’s understanding of a visualization Controlling for human variability makes effects detectable Looking at extremes helps us understand normal behavior ä Our questions Does everyone solve problems similarly? Do they make the same error? How does diversity in the population affect performance in the visualization? Do the solutions proposed help everyone equally?

6 Research Approach I. Select a set of cognitive skills that are likely to play a role in visualization understanding II. Measure these cognitive skills with standardized tests using a group of subjects selected for their variability III. Measure the level of visualization understanding of the subjects (via one type of “prototypical” visualization test) IV. Match the visualization performance results to the standardized test results V. Examine the properties of the visualization for both successful and unsuccessful comprehensions for each spatial ability subgroup VI. Examine the error distributions of the wrong answers for each spatial ability subgroup

7 Human Spatial Abilities ä What are spatial abilities? Skills involving the retrieval, retention and transformation of visual information in a spatial context. ä Are there other relevant cognitive factors ? Spatial Orientation Spatial Location Memory Targeting Spatial Visualization Disembedding Spatial Perception Visual Memory Perceptual Speed I Determine which cognitive skills that might play a role in visualization understanding

8 Standardized Tests ä Measuring spatial abilities We measured spatial abilities through the Kit of Factor-Referenced Cognitive Tests available at ETS. II Measure a subset of those cognitive skills with standardized tests using a group of subjects selected for their variability Spatial Orientation: Cube Comparison Test Spatial Visualization: Paper Folding Test Disembedding: Hidden Patterns Test Visual Memory: Shape Memory Test Perceptual Speed: Identical Figures Test

9 The Visualization Test III Measure the level of visualization understanding of the subjects (via a “prototypical” visualization test) ä Goal: Examine the comprehension of a “prototypical” visualization: orthogonal projection Basic visualization without bells and whistles Easy to learn by untrained experiment participants Use geometrical (geon-like and compounded) and common realistic objects Geon-like Compounded Realistic

10 The Visualization Test – Screen 1 Mentally form an image of the object and its alignment

11 The Visualization Test – Screen 2 Select the object that represents the object creating the projections This should be your answer

12 Experiment Design ä Measures of performance Accuracy: percentage of correct answers Analysis Time : Time spent analyzing the object’s projections (seconds) Selection Time : Time spent selecting the answer (seconds) Analysis Time and Selection Time are measured independently ä Experiment Participants (selected for variability) 56 paid participants, 50% percent female Average age: 21 years (range: 18 to 31 years) 84% undergraduate, 16% graduate students

13 Experimental Method ä Experiment Procedure Five paper-based cognitive factor tests General instructions and five practice questions Computer-based visualization test : 60 minutes to complete 38 questions Debriefing explaining the purpose of the experiment

14 Road Map to Analysis Spatial test results Standardized cognitive test Performance Analysis of subject’s errors (Case by case analysis) Correlation Visualization properties results Spatial ability groups results Correlation Differences

15 Expected Relation Between Spatial Ability and Visualization Performance Shape Memory Cube Comparison Paper Folding Pattern Matching Identical Figures AccuracyLow Positive / No correlation + High + LowNo correlation Time of Analysis Low Positive / No correlation + High + Low- High Time of Selection Low Positive / No correlation + High + Low- High + Positive correlation - Negative correlation

16 Spatial Test Results ä Analysis Pearson correlation analysis between performance and scores in standardized tests IV Match the visualization performance results to the standardized test results ä Results Shape Memory Cube Comparison Paper Folding Pattern Matching Identical Figures Accuracy+ Low+ High No Corr.- Low Time of AnalysisNo Corr. - Low Time of SelectionNo Corr. - Medium- Low ä Implications Visualization comprehension on diverse populations affected by spatial ability diversity Paper tests were time constrained which may have affected the time correlations Not Expected Expected

17 Visualization Properties Results ä For the object’s properties (i.e. surfaces, edges and vertices) we calculated: Total count in the original 3D object Distinct properties that would be visible in a wireframe rendering of the projection. Visible properties in a uniformly shaded object e1e1 e2e2 e3e3 e4e4 e5e5 e6e6 e7e7 e1e1 e2e2 e3e3 e4e4 e5e5 e6e6 e7e7 e1e1 e 24 e 35 e6e6 Total count of edges: 12 Distinct edges: 7 Visible edges: 4

18 Visualization Properties Results ä Analysis Pearson correlation analysis between performance and property counts and ratios ä Implications The hidden geometric properties make visualization understanding cognitively harder and thus, more time consuming Rotation of objects and animation will help users’ comprehension Complex objects require slower animations to give viewer time to extract information Counts Ratios Visualized / Distinct EdgesVerticesSurfaces AccuracyNo Corr.+ High+ Medium Time of Analysis+ Medium- High Time of SelectionNo Corr. ä Results

19 Other Visualization Properties Results ä A learning curve was not detected (see figure  ) ä Accuracy was affected by choices that differed from the correct answer by small differences in orientation ä No significant performance differences were found between geometric and realistic objects

20 Spatial Ability Groups Results ä No significant property differences between: All questions with high percentage of correct answers All questions with high percentage of incorrect answers V Examine the properties of the visualization for both successful and unsuccessful comprehensions for each spatial ability subgroup BUMMER! Our Next Step is to Look at the Data in More Detail ä Divide-up participants 3 groups and selected the High Spatial (HS) and Low Spatial (LS) ability participants (based on Paper Folding Test). Knowing a source of variability and looking at the extremes helps to make the effect visible

21 Spatial Ability Groups Results ä Analysis Are there properties that only high spatial people use? Compare Properties of Questions Answered Correctly by high spatial participants to Properties of all Questions ä Results Total Number of Edges and Total Number of Vertices were found significantly higher in questions which the high spatial participants answered correctly The Ratio of Distinct to Visualized Surfaces was found significantly higher in questions answered correctly by high spatial participants ä Implications High spatial participants understand more complex objects and can process a higher number of hidden properties

22 Analysis of Subject’s Errors VI Examine the error distributions of the wrong answers for each spatial ability subgroup ä Analysis Create a bar graph showing distribution of answers for each question Analyze the questions where distributions clearly not evenly distributed ä Interesting results for further analysis (possible strategies) Frequency of answers High spatial ability 6 Low spatial ability 8 High spatial ability Low spatial ability

23 Issues ä Experiment trials organized according to what was believed to be trial difficulty. This organization was wrong. ä There was ambiguity in the answers that participants had to choose between, in particular because participants were allowed to rotate the answers, they were not able to see the differences in orientation between two possible answers ä Only projection visualization was studied and thus, the results cannot be readily extrapolated to many other 3D visualizations ä The object properties manipulated in the questions were horizontal and vertical alignment. Future studies will include properties such as size, shape (sides), aspect ratio.

24 Summary of Results ä Spatial abilities are related to 3D visualization comprehension ä Problem solution time was not found to be related to visualization accuracy ä Counts of geometric properties affected visualization accuracy for low spatial subjects, and time of analysis for everyone ä The “hidden” geometric properties in the visualization affect visualization accuracy for low spatial subjects ä Small rotation differences are difficult to detect in a visualization ä A case by case analysis suggests that high spatial and low spatial ability participants use different strategies

25 Larger Implications of Research ä Visualization designers can use measures of cognitive ability to help understand what makes visualizations hard/easy to comprehend ä Using interactive rotation and animations is likely to help users better understand visualizations ä Visualization difficulty may be highly variable for a diverse population ä There exist educated people who cannot understand simple 3D visualizations

26 Acknowledgments ä Thanks to our reviewers for their comments ä This research is supported by the National Science Foundation through the SGER grant #

27