1. Indented Tree or Graph? A Usability Study of Ontology Visualization Techniques in the Context of Class Mapping Evaluation Presenter: Thomas Rodenhausen

2. OVERVIEW Controlled experimental usability study on ontology visualization techniques Indented Tree Graph Goal: Understand the semantics in ontologies Task: Evaluate class mappings Quantitative and Qualitative analysis Effectiveness Efficiency Workload Satisfaction Results: Both with strengths and weaknesses

3. Indented Tree Nodes: Ontological entities Indentation illustrates super-subclass relationships One and only one path between any pair of nodes Used Implementation: Protégé’s

4. Indented Tree Nodes: ontological entities Edges: Relationships between them Used Implementation: D3’s

5. EXPERIMENT Evaluate a set of mappings between pairs of ontologies Requires an understanding of the semantics of the mapped entities in their respective ontologies Gained by interacting with the visualizations

6. EVALUATE CLASS MAPPINGS Identify Correct mappings Incorrect mappings Create Missing mappings

7. DATASETS Ontologies accompanied by mapping gold standards Conference Ontologies: Ontology Alignment Evaluation Initiative 2012 Simpler: smaller, not very deep, only single inheritance BioMed Ontologies: Organism domain More complexity: bigger, higher fan-out, more depth, multiple inheritance Preparation for Experiment Randomly removed correct mappings from standard and added incorrect ones To identify: 13 correct, 3 incorrect 7 missing.

8. PARTICIPANTS Volunteer undergrad/grad students Computer science, Biomedical Biochemistry Mechanical Electrical Software engineering. All novice users of semantic technologies and new to ontologies and ontolgy mapping.

9. PROTOCOL One-on-one sessions 1.Online tutorial on ontologies and ontology mapping 2.Instructions on "evaluate set of mappings between pair of ontologies“ 3.First task: A combination of visualization type and dataset; Video tutorial on visualization type before start 4.Second task: Left-over combination of visualization type and dataset; Video tutorial on visualization type before start Between one-on-one sessions: Vary ordering of visualization type and dataset. Ensures participants do not become overly familiar with a visualization type nor the domain of interest over time. Thus, visualization type only independent variable

10. QUANTITATIVE ANALYSIS (MEASURED) Effectiveness Task success rate: identification/creation/overall e.g. between O1 and O2: n1 correct, n2 incorrect and n3 missing mappings. Identification success: x + y / (n1 + n2) where x and y are the successfully identificed correct and incorrect mappings. Efficiency Time to complete task

11. QUANTITATIVE ANALYSIS (PRECEIVED) Computerized surveys NASA task load index (NASA-TLX) System Usability Scale (SUS) Usefulness Satisfaction and Ease of use (USE) 7 point likert scales for all Workload “the cost of accomplishing mission requirements for the human operator”. NASA-TLX measures 6 dimensions, each measured through a question mental, physical, temporal demand, effort performance and frustration level. Averaged 6 dimension scores (0 – 100)

12. QUANTITATIVE ANALYSIS (PRECEIVED) System Usability Scale Agreement on 5 positively 5 negatively worded statements “I thought the visualization was easy to use” “I found the visualization unnecessarily complex” For positive: scale position minus 1. For a negatively: n minus the scale position. Aggregated usability score (0 – 100) Usefulness Satisfaction and Ease of use Dimensions: Usefulness Ease of use Ease of learning Satisfaction User agreement to 30 statements. Mean rating (0 – 6)

13. QUALITATIVE ANALYSIS Reaction cards: 118 catchphrases "engaging“ "powerfull“ "rigid“ "dated“ etc. Pick top 5 cards

14. FINDINGS: EFFECTIVENESS

15. Conference Task: Graph-assisted slightly higher mean overall success score No statistically significant difference Biomed Task: Indented tree with higher mean, median and lower standard deviation in identification and overall success scores Also lower and less dispersed error rates Statistically significant result

16. FINDINGS: EFFICIENCY

17. Users assisted by indented tree had faster completion times Not statistically significant

18. FINDINGS: WORKLOAD (NASA-TLX)

19. Users assisted by graph found task more demanding than those assisted by indented graph Not statistically significant

20. FINDINGS: USABILITY (SUS)

21. In both tasks, indented tree was found more usable Not statistically signficiant

22. FINDINGS: USABILITY (USE)

23. Conference Task: Indented tree slightly higher rating in all dimensions Not statistically signficiant BioMed Task: Decrease in all ratings for both visualization techniques Higher values still found in indented tree Statisticaly significant difference for “usefulness” Suggests: As evaluation task becomes more complex visualization support appears to be less helpful regardless of the technique. Overall all, USE dimensions very comparable between the techniques

24. FINDINGS: USABILITY (REACTION CARDS)

25. Conference Task: Both visualization techniques were found easy to use BioMed Task More diverse reaction cards used Increase in number of negative cards present for both techniques Participants consistently used “Simplistic” to describe indented tree “Easy to use” to describe the graph Participants particularly liked how multiple inheritance is visualized in graphs

26. DISCUSSION: CORRELATION TESTS Overall, mostly weak or non-existent associations between variables, e.g. task success is correlated with usability scores. R-values indicate that visualization usability did not impact task success.

27. DISCUSSION: CORRELATION TESTS Error rates are correlated with task completion time. Stronger correlation in the BioMed task found, suggesting that if more time is spent to complete a task, users using graphs are likely to make fewer mistakes.

28. DISCUSSION: CORRELATION TESTS Error rates are correlated with task completion time. Stronger correlation in the BioMed task found, suggesting that if more time is spent to complete a task, users using graphs are likely to make fewer mistakes.

29. SUMMARY OF FINDSINGS Smaller ontologies: Likely to achieve same level of success regardless of visualization technique Complex ontologies: Indented tree is more effective Specifically: Indented tree more successful concerning evaluation of existing mappings Specifically: Graph more successful concerning creation of new mappings Efficiency results: Task completion time more likely to be a result of domain familiarity Worload results: Similar to efficiency Visualization that can seamlessly incorporate multiple inheritance is essential to users, thus graphs preferred in these scenarios Duplication required in indented trees which requires additional effort and can cause confusion for the users

30. SUMMARY OF FINDSINGS Screen space Indented tree Not always possible to view entire tree structure. Difficult for user to preserve a mental model of the ontological hierarchy. Graph More customizable, e.g. users can move explored nodes aside Users report: Flexibility allows to hold their attention Disadvantage: Can quickly get busy with large number of nodes Overall, the advantage of indented tree is familiarity and predictability Given the different strengths and weaknesses associated with graphs and indented trees Applications should be determined upon specific ontology characteristics, visualization needs and user goals Tool designers should consider combining multiple ontology visualization techniques that can engage users from different viewpoints yet are complementary to one another

31. LIMITATIONS & FUTURE DIRECTIONS Results dependent on indented tree and graph implementation, e.g. protégé and D3 Future Conduct studies with larger participant group to possibly achieve statistically significant results Could explore other visualization technique It can be even more informative to include ontology / mapping experts into the participant group Results based on two limited datasets. Larger datasets may discover other scalability issues.

32. SOMETHING RELATED Visualization techniques for ontologies used in existing tools: Indented tree: protege, webprotege, OBO-edit, structOntology, VectorBase, BioPortal, OntoLink Treemaps: [11] Graphs: Protege viz plugins, e.g. OwlViz, NavigOWL, TGVizTab, OWLPropViz. Others FlexViz, BioMixer, OLSVis. 3D-Techniques: Ontosphere Dynamically rescaling graphs to utilize screen space: SpaceTree Reducing nr. of classes shown (importance score): [15] Parent-child and sibling relationships simultaneously: CropCircle Several argue for the benefit of multiple viz techniques to adapt to user pref. and style: [16,17,18]