1. Indented Tree or Graph?
A Usability Study of Ontology Visualization Techniques in the Context of Class Mapping Evaluation
本体可视化技术在类型匹配评估中的可用性研究
Qingxia Liu
2019/2/19

2. Authors Bo Fu Natalya F. Noy Margaret-Anne Storey
a Post-Doctoral Research of
University of Victoria
visualizations of semantic data, semantic technologies in use and interoperability on the multilingual semantic web.
Natalya F. Noy
Senior research Scientist
at Stanford Center for Biomedical
Informatics Research (BMIR)
Semantic web; collaboration and social mechanisms in ontology design, evaluation, management, and mapping; evaluation of knowledge-based systems; semantic integration; ontology evolution; knowledge acquisition; semantic technologies in biomedicine.
Margaret-Anne Storey
professor of computer science of University of Victoria, Canada
software engineering, human-computer interaction, information visualization, social informatics and knowledge management.

3. Visualization Techniques
Intended tree
Ontology editors
Protégé, WebProtégé, OBOEdit, structOntology
Ontology browsers
VectorBase
Ontology Libraries
BioPortal
Ontology mapping tools
OntoLink
Graph (node-link diagram)
Protégé plugins
OwlViz, NavigOWL, TGVizTab, OWLPropViz
Others
Tree map
3D techniques
Intended tree: 任意两点间有且只有一条路径，缩进表示子关系
Graph： 有向箭头（指向子节点）
Goal:
To investigate the effectiveness and efficiency of the support that the two visualization techniques provide.

4. Prepare Datasets Participants: 36 Protocol: random Feedback:
The Ontology Alignment Evaluation Initiative (OAEI) 2012 conference
The BioMed tracks
Participants: 36
undergraduate and graduate students, novice users
Protocol: random
Feedback:
The NASA-task load index (NASA-TLX)
Workload: the cost of accomplishing mission requirements for the human operator
The System Usability Scale (SUS)
The Usefulness, Satisfaction and Ease of Use (USE) questionnaire
Reaction cards: adj.

5. Usability Study Overview
Figure 1. Sample Task Screen. In this example, graphs are used to visualize two biomedical
ontologies. Mappings to be evaluated are presented in a spreadsheet. Interacting with the visualizations,
participants must use drop-down lists containing either “yes” or “no” responses to
evaluate the correctness of existing mappings (in part 1) and add missing mappings by typing
class names (into part 2 of the spreadsheet).

6. Experiment Tasks Metrics Identification activities: 13+3
Creation activities: 7
Metrics
Success scores [0,1]
Success score = successNum / totalNum
Overall success = identification success + creation success
Error rates [0,1]
The lower, the fewer mistakes

7. Findings Effectiveness Overall success T-test (alpha=0.05)
Null hypothesis: there is no difference between the two user groups
Conference task: no significant difference
BioMed task: intended trees more effective
Figure 3. Visualization Effectiveness. The vertical axis illustrates mean overall success and the
horizontal axis represents the user groups using different visualizations. Error bars show 95%
confidence intervals, i.e., how far from the reported value the true (error free) value might be.
0.05 alpha
Once a t value is determined, a p-value can be found using a table of values from Student's t-distribution. If the calculated p-value is below the threshold chosen for statistical significance (usually the 0.10, the 0.05, or 0.01 level), then the null hypothesis is rejected in favor of the alternative hypothesis. --wiki “t-test”
零假设

8. Findings Efficiency no significant difference
Figure 4. Visualization Efficiency. The vertical axis represents mean time-on-task, and the
horizontal axis illustrates the user group. Error bars show 95% confidence intervals.

9. Findings
Workload
no significant difference

10. Findings Usability and Qualitative Feedback SUS USE
no significant difference
USE
Only significant on usefulness
SUS
USE
可用性和定性反馈

11. Findings Usability and Qualitative Feedback Reaction Card
Indented tree
Organized, straightforward, simplistic
Dull, boring, busy
Graph
Easy to use
Approachable, controllable —— Conference
Annoying, complex —— BioMed
Both
Distracting, frustrating, confusing
SUS
USE
可用性和定性反馈

12. Discussion Correlation Tests Findings
Weak or non-existent associations between variables
More time => graph make fewer mistakes
Findings
Small, simple ontologies(conference)——no different
Complex ontologies(BioMed)——intended tree

13. Findings Intended tree graph List-checking actives
Familiar and predictable
Disadvantages:
Duplication for multiple inheritance add to confusion
Fixed screen space:
Depth, number of descendants per node
Overviews
Flexibility--hold attention
Disadvantages:
In effective given a large number of nodes

14. Conclusion Combining multiple ontology visualization techniques
Should provide customizable visualizations:
Manageable
adaptive to diverse personal presences and styles
可定制、易于控制

15. Thank You !
Q & A

16. Main Point
Two Ontology visualization techniques in the state of the art:
Intended tree
Graph visualization
Goal:
To investigate the effectiveness and efficiency of the support that the two visualization techniques provide.