Indented Tree or Graph? A Usability Study of Ontology Visualization Techniques in the Context of Class Mapping Evaluation 本体可视化技术在类型匹配评估中的可用性研究 Qingxia Liu 2019/2/19 qxliu.nju@gmail.com

Authors Bo Fu Natalya F. Noy Margaret-Anne Storey a Post-Doctoral Research of University of Victoria visualizations of semantic data, semantic technologies in use and interoperability on the multilingual semantic web. Natalya F. Noy Senior research Scientist at Stanford Center for Biomedical Informatics Research (BMIR) Semantic web; collaboration and social mechanisms in ontology design, evaluation, management, and mapping; evaluation of knowledge-based systems; semantic integration; ontology evolution; knowledge acquisition; semantic technologies in biomedicine. Margaret-Anne Storey professor of computer science of University of Victoria, Canada software engineering, human-computer interaction, information visualization, social informatics and knowledge management.

Visualization Techniques Intended tree Ontology editors Protégé, WebProtégé, OBOEdit, structOntology Ontology browsers VectorBase Ontology Libraries BioPortal Ontology mapping tools OntoLink Graph (node-link diagram) Protégé plugins OwlViz, NavigOWL, TGVizTab, OWLPropViz Others Tree map 3D techniques Intended tree: 任意两点间有且只有一条路径，缩进表示子关系 Graph： 有向箭头（指向子节点）http://webhome.csc.uvic.ca/~bofu/study/o1.html Goal: To investigate the effectiveness and efficiency of the support that the two visualization techniques provide.

Prepare Datasets Participants: 36 Protocol: random Feedback: The Ontology Alignment Evaluation Initiative (OAEI) 2012 conference The BioMed tracks Participants: 36 undergraduate and graduate students, novice users Protocol: random Feedback: The NASA-task load index (NASA-TLX) Workload: the cost of accomplishing mission requirements for the human operator The System Usability Scale (SUS) The Usefulness, Satisfaction and Ease of Use (USE) questionnaire Reaction cards: 118 adj.

Usability Study Overview Figure 1. Sample Task Screen. In this example, graphs are used to visualize two biomedical ontologies. Mappings to be evaluated are presented in a spreadsheet. Interacting with the visualizations, participants must use drop-down lists containing either “yes” or “no” responses to evaluate the correctness of existing mappings (in part 1) and add missing mappings by typing class names (into part 2 of the spreadsheet).

Experiment Tasks Metrics Identification activities: 13+3 Creation activities: 7 Metrics Success scores [0,1] Success score = successNum / totalNum Overall success = identification success + creation success Error rates [0,1] The lower, the fewer mistakes

Findings Effectiveness Overall success T-test (alpha=0.05) Null hypothesis: there is no difference between the two user groups Conference task: no significant difference BioMed task: intended trees more effective Figure 3. Visualization Effectiveness. The vertical axis illustrates mean overall success and the horizontal axis represents the user groups using different visualizations. Error bars show 95% confidence intervals, i.e., how far from the reported value the true (error free) value might be. 0.05 alpha Once a t value is determined, a p-value can be found using a table of values from Student's t-distribution. If the calculated p-value is below the threshold chosen for statistical significance (usually the 0.10, the 0.05, or 0.01 level), then the null hypothesis is rejected in favor of the alternative hypothesis. --wiki “t-test” 零假设

Findings Efficiency no significant difference Figure 4. Visualization Efficiency. The vertical axis represents mean time-on-task, and the horizontal axis illustrates the user group. Error bars show 95% confidence intervals.

Findings Workload no significant difference

Findings Usability and Qualitative Feedback SUS USE no significant difference USE Only significant on usefulness SUS USE 可用性和定性反馈

Findings Usability and Qualitative Feedback Reaction Card Indented tree Organized, straightforward, simplistic Dull, boring, busy Graph Easy to use Approachable, controllable —— Conference Annoying, complex —— BioMed Both Distracting, frustrating, confusing SUS USE 可用性和定性反馈

Discussion Correlation Tests Findings Weak or non-existent associations between variables More time => graph make fewer mistakes Findings Small, simple ontologies(conference)——no different Complex ontologies(BioMed)——intended tree

Findings Intended tree graph List-checking actives Familiar and predictable Disadvantages: Duplication for multiple inheritance add to confusion Fixed screen space: Depth, number of descendants per node Overviews Flexibility--hold attention Disadvantages: In effective given a large number of nodes

Conclusion Combining multiple ontology visualization techniques Should provide customizable visualizations: Manageable adaptive to diverse personal presences and styles 可定制、易于控制

Thank You ! Q & A

Main Point Two Ontology visualization techniques in the state of the art: Intended tree Graph visualization Goal: To investigate the effectiveness and efficiency of the support that the two visualization techniques provide.