User validation in ontology alignment

Slides:



Advertisements
Similar presentations
Project Proposal.
Advertisements

Chapter 6: Design of Expert Systems
The Process of Interaction Design. Overview What is Interaction Design? —Four basic activities —Three key characteristics Some practical issues —Who are.
The Process of Interaction Design. What is Interaction Design? It is a process: — a goal-directed problem solving activity informed by intended use, target.
1 CS 430 / INFO 430 Information Retrieval Lecture 24 Usability 2.
4. Interaction Design Overview 4.1. Ergonomics 4.2. Designing complex interactive systems Situated design Collaborative design: a multidisciplinary.
Chapter 7 design rules.
FLCC knows a lot about assessment – J will send examples
Data Mining Techniques
Semantic Interoperability Jérôme Euzenat INRIA & LIG France Natasha Noy Stanford University USA.
S/W Project Management
An Integrated Approach to Extracting Ontological Structures from Folksonomies Huairen Lin, Joseph Davis, Ying Zhou ESWC 2009 Hyewon Lim October 9 th, 2009.
Chapter 8 Architecture Analysis. 8 – Architecture Analysis 8.1 Analysis Techniques 8.2 Quantitative Analysis  Performance Views  Performance.
Purpose of study A high-quality computing education equips pupils to use computational thinking and creativity to understand and change the world. Computing.
Automatic Lexical Annotation Applied to the SCARLET Ontology Matcher Laura Po and Sonia Bergamaschi DII, University of Modena and Reggio Emilia, Italy.
Design Science Method By Temtim Assefa.
11 C H A P T E R Artificial Intelligence and Expert Systems.
CSCD 487/587 Human Computer Interface Winter 2013 Lecture 3 HCI and Interactive Design.
Object-Oriented Software Engineering Practical Software Development using UML and Java Chapter 7: Focusing on Users and Their Tasks.
Of 33 lecture 10: ontology – evolution. of 33 ece 720, winter ‘122 ontology evolution introduction - ontologies enable knowledge to be made explicit and.
Midterm Stats Min: 16/38 (42%) Max: 36.5/38 (96%) Average: 29.5/36 (78%)
Topic (iii): Macro Editing Methods Paula Mason and Maria Garcia (USA) UNECE Work Session on Statistical Data Editing Ljubljana, Slovenia, 9-11 May 2011.
27/3/2008 1/16 A FRAMEWORK FOR REQUIREMENTS ENGINEERING PROCESS DEVELOPMENT (FRERE) Dr. Li Jiang School of Computer Science The.
Automatic Video Tagging using Content Redundancy Stefan Siersdorfer 1, Jose San Pedro 2, Mark Sanderson 2 1 L3S Research Center, Germany 2 University of.
Achieving Semantic Interoperability at the World Bank Designing the Information Architecture and Programmatically Processing Information Denise Bedford.
Quality Assurance in the Presence of Variability Kim Lauenroth, Andreas Metzger, Klaus Pohl Institute for Computer Science and Business Information Systems.
Identifying “Best Bet” Web Search Results by Mining Past User Behavior Author: Eugene Agichtein, Zijian Zheng (Microsoft Research) Source: KDD2006 Reporter:
Introduction to OOAD and UML
A session-based approach for aligning large ontologies Patrick Lambrix, Rajaram Kaliyaperumal Linköping University.
1 The XMSF Profile Overlay to the FEDEP Dr. Katherine L. Morse, SAIC Mr. Robert Lutz, JHU APL
Stages of Research and Development
Design rules.
WP4 Models and Contents Quality Assessment
Human Computer Interaction Lecture 21 User Support
Visualization for Ontology Evolution
Software Project Configuration Management
Towards a framework for architectural design decision support
Decision Support Systems
Learning in Neural Networks
Programme Board 6th Meeting May 2017 Craig Larlee
Preface to the special issue on context-aware recommender systems
Chapter 6: Design of Expert Systems
HI 5354 – Cognitive Engineering
Datamining : Refers to extracting or mining knowledge from large amounts of data Applications : Market Analysis Fraud Detection Customer Retention Production.
HCI in the software process
The design process Software engineering and the design process for interactive systems Standards and guidelines as design rules Usability engineering.
The design process Software engineering and the design process for interactive systems Standards and guidelines as design rules Usability engineering.
Presentation 王睿.
CSc4730/6730 Scientific Visualization
CSc4730/6730 Scientific Visualization
Extracting Semantic Concept Relations
Human Resources Management
Tomás Murillo-Morales and Klaus Miesenberger
ece 627 intelligent web: ontology and beyond
Usability Techniques Lecture 13.
HCI in the software process
Indented Tree or Graph? A Usability Study of Ontology Visualization Techniques in the Context of Class Mapping Evaluation 本体可视化技术在类型匹配评估中的可用性研究 Qingxia.
Magnet & /facet Zheng Liang
HCI in the software process
CHAPTER 7: Information Visualization
Chapter 7 design rules.
Chapter 7 design rules.
Chapter 7 design rules.
Chapter 10 Stakeholder Management
Chapter 10 Stakeholder Management
SECURITY AS NON-FUNCTIONAL REQUIREMENT IN SOFTWARE ENGINEERING
Retrieval Performance Evaluation - Measures
Template for methodological application
Chapter 7 design rules.
Comp 15 - Usability & Human Factors
Presentation transcript:

User validation in ontology alignment Zlatan Dragisic1, Valentina Ivanova1, Patrick Lambrix1, Daniel Faria2, Ernesto Jiménez-Ruiz3 and Catia Pesquita4 1 Linköping University 2 Gulbenkian Science Institute, Portugal 3 University of Oxford, UK 4 Universidade de Lisboa, Portugal

Motivation Many automatic ontology alignment systems Limits to the performance of automated systems Users involvement in the alignment process enables the detection and removal of erroneous mappings the addition of alternative and potentially new mappings the adjustment of system settings Even when the user makes error there is benefit from the interaction (selection of the most suitable alignment algorithms, and the incorporation of user knowledge) The growth of the ontology alignment area led to the development of

Motivation cont’d User involvement in ontology alignment is directly related to a number of the challenges facing the community [SE13] Quality and effectiveness of user intervention OAEI – Interactive matching track (2013) Users can/will make mistakes 2015 – Interactive matching extended to cover erroneous input explanation of matching results to users fostering user involvement in the matching process social and collaborative matching Adopting more advanced alignment techniques has brought diminishing returns [40,21]. This is likely due to the complexity and intricacy of the ontology alignment process, with each task having its particularities, dictated by both the domain and the design of the ontologies. Thus, automatic generation of mappings should be viewed only as a first step towards a final alignment, with validation by one or more users being essential to ensure alignment quality , experiments have shown that user validation is still beneficial up to an error rate of 20% [26], although the exact error threshold will depend on the alignment system and how it makes use of the user input. [SE13] Ontology Matching: State of the Art and Future Challenges, Shvaiko P, and Euzenat J, IEEE Transactions on Knowledge and Data Engineering

Contributions Identification of issues related to user validation in ontology alignment User profile Interface System services Qualitative evaluation evaluating how state-of-the- art systems deal with the issues Experiments evaluating how systems deal with erroneous input from the user While there has been work on user involvement, here the focus is on one aspect user validation First extensive qualitative evaluation A broad study of user validation in ontology alignment First experiments in the field Now we look at the user profile

Issues regarding user alignment validation User profile System services User interface

Identified issues regarding user profile Domain expertise of the user Technical expertise of the user Expertise with the alignment system Users can be expected to make mistakes Domain expertise - concerns his knowledge about the domain of the aligned ontologies, and therefore his ability to assess the correctness of a mapping conceptually (e.g., whether two ontology classes mapped as equivalent actually represent the same concept in the domain). Technical expertise - pertains to his knowledge about ontologies themselves, and his experience in knowledge engineering and modeling, and therefore his ability to assess the correctness of a mapping formally (i.e., whether a mapping is logically sound given the constraints of the two ontologies). Knowledge engineer, domain expert pictures

Identified issues regarding system services Agreement Maker AlViz AML CogZ Prompt COMA LogMap SAMBO RepOSE System services Stage of involvement before, after, during, iterative a+i a d b,a+i Suggestions selection Threshold/ advanced filtering ✓ ✓- - Feedback Propagation Recomputation Conflict detection/blocking/revalidation ✓- - ADD SOME EXAMPLES Say how current systems are doing First, is when the user is involved Similarity value above/below threshold filtering with respect to some principles (e.g., consistency, locality, and conservativity) [25] or quality checks [3] selecting only “problematic” mappings where different alignment algorithms disagree [8] and using a similarity propagation graph to select the most informative questions to ask the user [44] Propagating mapping confidence from validated mappings to those in their neighborhood, be that neighborhood defined from the structure of the ontologies [31,37,44] or from the pattern of similarity scores from the various alignment algorithms ( 2, B ) ( 3, F ) ( 6, D ) ( 4, C ) ( 5, C ) ( 5, E ) …… sim suggest th discard

Identified issues regarding system services Agreement Maker AlViz AML CogZ Prompt COMA LogMap SAMBO RepOSE System services Stage of involvement before, after, during, iterative a+i a d b,a+i Suggestions selection Threshold/ advanced filtering ✓ ✓- - Feedback Propagation Recomputation Conflict detection/blocking/revalidation ✓- - ADD SOME EXAMPLES Say how current systems are doing First, is when the user is involved Similarity value above/below threshold filtering with respect to some principles (e.g., consistency, locality, and conservativity) [25] or quality checks [3] selecting only “problematic” mappings where different alignment algorithms disagree [8] and using a similarity propagation graph to select the most informative questions to ask the user [44] Propagating mapping confidence from validated mappings to those in their neighborhood, be that neighborhood defined from the structure of the ontologies [31,37,44] or from the pattern of similarity scores from the various alignment algorithms (2,C) (5,F) (6,F) 0.5

Identified issues regarding user interface Agreement Maker AlViz AML CogZ Prompt COMA LogMap SAMBO RepOSE Alignment presentation Visual information seeking tasks ✓ ✓- ✓- - - Visual analytics Alternative views Grouping Validated/candidate mappings Metadata & context Ranking/ recommendations ✓ - - Mapping explanation Provenance & justification Impact/ consequences Overview Zoom Filter Details-on-demand Relate History Extract Visual Information Seeking Mantras – overview and defines seven low-level tasks to be supported by information visualization interfaces in order to enable enhanced data exploration and retrieval overview usually supported and filter, history and relate rarely supported Visual analytics – uses data mining uses interactive vis techniques provides helps you getting insights from the data. Providing enhanced information while addressing the working memory limits - data mining and interactive visualization techniques an overview of the mappings obtained between all the entities in the source ontology and in the target ontology, as presented in the Matcher Output Grid View; (2) the behavior of the entities of the source and target ontologies with respect to various statistics, as provided by the Entity Mapping Characteristics Scatter Plot View; (3) the mappings between entities in the source and target ontologies, which uses the interactive Ontology Tree View; (4) the results for all the matchers alongside the reference alignment (when available) for comparative analysis, as enabled by the Parallel Coordinate View Alternative views – graphs for information perception, while trees for searching (condensing information will overwhelm the user) TEXTUAL AND VISUAL JUSTIFICATION OF MAPPINGS

Identified issues regarding user interface Agreement Maker AlViz AML CogZ Prompt COMA LogMap SAMBO RepOSE Alignment presentation Visual information seeking tasks ✓ ✓- ✓- - - Visual analytics Alternative views Grouping Validated/candidate mappings Metadata & context Ranking/ recommendations ✓ - - Mapping explanation Provenance & justification Impact/ consequences Overview Zoom Filter Details-on-demand Relate History Extract Provide sufficient information to support decision making while not overwhelming the user Visual Information Seeking Mantras - defines seven low-level tasks to be supported by information visualization interfaces in order to enable enhanced data exploration and retrieval overview usually supported and filter, history and relate rarely supported

Identified issues regarding user interface Agreement Maker AlViz AML CogZ Prompt COMA LogMap SAMBO RepOSE Alignment presentation Visual information seeking tasks ✓ ✓- ✓- - - Visual analytics Alternative views Grouping Validated/candidate mappings Metadata & context Ranking/ recommendations ✓ - - Mapping explanation Provenance & justification Impact/ consequences Visual analytics – uses data mining uses interactive vis techniques provides helps you getting insights from the data. Providing enhanced information while addressing the working memory limits - data mining and interactive visualization techniques

Identified issues regarding user interface Agreement Maker AlViz AML CogZ Prompt COMA LogMap SAMBO RepOSE Alignment presentation Visual information seeking tasks ✓ ✓- ✓- - - Visual analytics Alternative views Grouping Validated/candidate mappings Metadata & context Ranking/ recommendations ✓ - - Mapping explanation Provenance & justification Impact/ consequences Alternative views – graphs for information perception, while trees for searching (condensing information will overwhelm the user)

Identified issues regarding user interface Agreement Maker AlViz AML CogZ Prompt COMA LogMap SAMBO RepOSE Alignment presentation Visual information seeking tasks ✓ ✓- ✓- - - Visual analytics Alternative views Grouping Validated/candidate mappings Metadata & context Ranking/ recommendations ✓ - - Mapping explanation Provenance & justification Impact/ consequences SHOW TASKS FOR vism Visual Information Seeking Mantras – overview and defines seven low-level tasks to be supported by information visualization interfaces in order to enable enhanced data exploration and retrieval overview usually supported and filter, history and relate rarely supported Visual analytics – uses data mining uses interactive vis techniques provides helps you getting insights from the data. Providing enhanced information while addressing the working memory limits - data mining and interactive visualization techniques an overview of the mappings obtained between all the entities in the source ontology and in the target ontology, as presented in the Matcher Output Grid View; (2) the behavior of the entities of the source and target ontologies with respect to various statistics, as provided by the Entity Mapping Characteristics Scatter Plot View; (3) the mappings between entities in the source and target ontologies, which uses the interactive Ontology Tree View; (4) the results for all the matchers alongside the reference alignment (when available) for comparative analysis, as enabled by the Parallel Coordinate View Alternative views – graphs for information perception, while trees for searching (condensing information will overwhelm the user) TEXTUAL AND VISUAL JUSTIFICATION OF MAPPINGS Frame names or synonyms are similar

Identified issues regarding user interface Agreement Maker AlViz AML CogZ Prompt COMA LogMap SAMBO RepOSE Alignment Interaction Accept/reject ✓ ✓- Create/refine - Search User annotation Session Temporary mappings

System services Stages of involvement Suggestions selection Before During After Iterative Suggestions selection Threshold Advance filtering Feedback propagation filtering with respect to some principles (e.g., consistency, locality, and conservativity) [25] or quality checks [3] selecting only “problematic” mappings where different alignment algorithms disagree [8] and using a similarity propagation graph to select the most informative questions to ask the user [44] Propagating mapping confidence from validated mappings to those in their neighborhood, be that neighborhood defined from the structure of the ontologies [31,37,44] or from the pattern of similarity scores from the various alignment algorithms Feedback propagataion, Recomputation Conflict detection/blocking/revalidation

User interface Alignment presentation Visual Information Seeking Mantra – (overview, zoom, filter, details-on-demand, relate, history and extract ) Visual analytics Alternative views Grouping Validated/candidate mappings Ranking/recommendation Explanation of mapping suggestions Metadata & context Provenance & justification Impact of decisions Ontologies are large and many mappings Information overload, memory load Visual Information Seeking Mantras – overview and directs defines seven low-level tasks to be supported by information visualization interfaces in order to enable enhanced data exploration and retrieval Visual analytics – helps you getting insights from the data. Providing enhanced information while addressing the working memory limits - data mining and interactive visualization techniques Alternative views – graphs for information perception, while trees for searcuing (condensing information will overwhelm the user)

User interface Alignment interaction Accept/reject Create/refine Search Annotations Sessions Temporary mappings

Qualitative evaluation State of the art systems which incorporate user validation and have a mature user interface: AgreementMaker AlViz AML CogZ/Prompt COMA LogMap RepOSE SAMBO Screenshots

Qualitative study - summary Majority of systems ask for validation after the alignment process Most systems use simple thresholds for candidate selection Feedback propagation mostly limited to conflict detection Some systems employ propagation strategies Visualization - mostly trees/graphs, mappings as links between nodes, or as lists/table Most systems support grouping of mappings Limited support for explanation and ranking/recommendation of mappings Alignment interaction Rejected mappings are rarely shown No distinction between candidate and validated mappings Overview and filter supported (to a certain extent), history and relate rarely supported Sessions supported directly or indirectly Example row from the study

Experiments Purpose Interaction simulated via reference alignment Simulate users with different expertise levels Show how systems service are affected by and cope with errors Interaction simulated via reference alignment Varying degrees of errors: 0%, 10%, 20%, 30% Anatomy (mid-size ontologies) and Conference track (small ontologies) Measures: Precision and recall Number of interactions Systems: AML JarvisOM LogMap ServOMBI Check how the interaction/error is computed AML – post-alignment interaction JarvisOM – interaction during the alignment LogMap – post-alignment interaction ServOMBI – post-alignment interaction

Experiments - results All tools improve with the all-knowing expert – highest Jarvis When increasing the error, performance deteriorates – most affected Jarvis ADD TEXTUAL POINTS

Experiments - results More improvement w.r.t. recall: AML More improvement w.r.t. precision: LogMap, ServOMBI ADD TEXTUAL POINTS

Experiments - results Errors impact precision: AML, JarvisOM Errors impact recall: ServOMBI ADD TEXTUAL POINTS

Experiments - results Redundant requests Extrapolation from user feedback Balance in TP/TN

Experiments - results

Experiments - results All tools improve with the all-knowing expert Highlighted differences

Experiments - results AML: Improves more in terms of recall At 20% drops below the non-interactive version Linearly affected by errors Does not extrapolate from the user feedback

Experiments - results JarvisOM: Highest improvement with user interaction Least requests to the oracle Affected severely by the user errors Results better than the non-interactive version even with 30% error

Experiments - results LogMap: Improves more in terms of precision Balanced positive/negative requests Error affects both precision and recall Increase in the number of requests

Experiments - results ServOMBI: Least improvment with interaction Improves more in terms of precision Most oracle requests (redundant as well) Strongly affected by the errors (recall) Increase in the number of requests

Conclusions A step towards guidelines and best practices for good user interface design User profiles need to be supported (errors) Systems need to prioritize which mappings to present to the user Extrapolate knowledge by feedback propagation (double-edged sword) The presented mappings need to be explained Consequences Justification

Future work Usability studies with real users Choice between conflicting mappings Non-binary classification