Danica Damljanović University of Sheffield

Outline Background: What are Ontologies? What are Natural Language Interfaces (NLIs)? What are Usability Enhancement Methods? Objective Improve NLIs to Ontologies with usability enhancement methods Our approach Two NLI systems for querying ontologies: QuestIO FREyA Two usability studies to test the usability enhancement methods Findings Demo Conclusion

Mary works for University of Sheffield, which is located in Sheffield. Sheffield is located in the United Kingdom. Mary lives in Sheffield. MARY PERSON UNIVERSITY OF SHEFFIELD ORGANISATION MARY UNIVERSITY OF SHEFFIELD SHEFFIELD CITY UNIVERSITY OF SHEFFIELD SHEFFIELD UNITED KINGDOM COUNTRY SHEFFIELD UNITED KINGDOM MARY SHEFFIELD SELECT ?country WHERE { ?person ?city ?city ?country FILTER ?person = MARY }

4 In which country does Mary live?

What are Usability Enhancement Methods? Who are the users? application developers end users

Customisation Ontology editing (e.g. using Protege) Domain lexicon NLI for querying … Domain knowledge WordNet Domain expert Ontology engineer NLI for Ontology authoring

The Objective Increase usability of Natural Language Interfaces to ontologies For end users: increase precision and recall For application developers: decrease the time for customisation

Our Approach ScopeRankingLexiconUsability methods Grammar analysis Supported language QuestIOone ontologyontology structure, string similarity ontology lexicalisaions none (automatic) Shallow (morphologica l analysis) grammatically correct/ question fragments FREyAa set of ontologies/ repository string similarity, synonym detection, user ontology lexicalisations, Wordnet, user vocabulary feedback, clarification dialogs (user interaction) Deeper (parsing) grammatically correct/ question fragments

Portability vs. Performance 9 moderate precision lowest precision highest precision high precision large datasets (several domains) simple factual/CNL questions complex/free-text questions small datasets (narrow domain) Damljanovic, D., Bontcheva, K.: Towards Enhanced Usability of Natural Language Interfaces to Knowledge Bases. In Devedzic V. and Gasevic D. (Eds.), Special issue on Semantic Web and Web 2.0, Annals of Information systems, Springer-Verlag, NLP Reduce, QuestIO ORAKEL, Querix, PANTO, AquaLog PowerAqua FREyA

QuestIO compare

QuestIO prototype

QuestIO: User Evaluation Usability testing: effectiveness: could the tasks could be finished using QuestIO efficiency: how quickly? user satisfaction System Usability Scale (SUS) subjective (was it easy to formulate a query?, etc.) Experimental setup: a complete counterbalanced repeated measures, task-based evaluation design Baseline (search engines) vs. QuestIO 12 subjects familiar with the domain (GATE software) four tasks: three defined, e.g....find parameters of Cebuano gazetteer... one undefined task,...find anything you want about GATE software...

QuestIO User Evaluation: Results Effectiveness: the scale from 0 (easy) to 2 (impossible) for QuestIO in comparison to for baseline, p = Efficiency: the subjects significantly slower when using baseline (157s) in comparison to QuestIO (107s), p=0.001 User satisfaction: SUS score satisfactory (69.38) Tasks: defined tasks: user satisfaction reaching 90% undefined tasks: user satisfaction low (~44%)

QuestIO: weaknesses Lexical failures: Tokenizer vs. Tokeniser Conceptual failures: missing concepts, relations, or both The users not being aware of why the failures happened Can this be improved with usability enhancement methods such as feedback and clarification dialogs?

FREyA - Feedback, Refinement, Extended Vocabulary Aggregator Feedback: showing the user system interpretation of the query Refinement: resolving ambiguity: generating dialog whenever one term refers to more than one concept in the ontology (precision) Extended Vocabulary: expressiveness: generating dialog whenever an “unknown” term appears in the question (recall) portability: no need for customisation from application developers The dialog: generated by combining the syntactic parsing and ontology-based lookup the system learns from the user’s selections 15

Feedback: answer is found

Feedback: No answer is found

Feedback: User Evaluation Usability testing: effectiveness efficiency user satisfaction System Usability Scale (SUS) subjective (was it easy to formulate a query?, etc.) Experimental setup: 30 subjects outside Sheffield, two domains (GATE software and US geography) four tasks: three defined: two repeated from the previous study one where the answer was not available, e.g....find states bordering hawaii... one undefined task,...find anything you want about GATE software or rivers, cities,... in the United States...

Does the feedback make any difference? Effectiveness: yes, p=0.01, 0.67 for QuestIO, 0.13 for FREyA Efficiency: no, although the overall result differs (180.5 seconds for QuestIO, seconds for FREyA), 2-tailed independent t-test reveals that this difference is not significant (p=0.852) Query Formulation: for the defined tasks there is no difference in the perception of the difficulty of the supported language (F=5.255, p=0.071), but for the undefined tasks the users believed that the language supported by FREyA is easier! (F=8.016, p=0.015) Showing that the system knows about certain concepts, but cannot find any relation between them was not clear. Interactive features were well accepted.

FREyA Workflow

Demo 03 June 2010ESWC

Evaluation: correctness 22  Mooney GeoQuery dataset, 250 questions  34 no dialog, 14 failed to be answered  Precision=recall=94.4%

Evaluation: Learning 23  10-fold cross- validation  202 Mooney GeoQuery questions that could be correctly mapped into SPARQL and required dialog  improvement from 0.25 to 0.48  Errors: ambiguity and sparseness

Evaluation: Ranking Mean Reciprocal Rank: 0.76

Learning the Correct Ranking  Randomly selected 103 dialogs from 202 questions (343 dialogs)  MRR increased for 6% from 0.72 to 0.78

Evaluation: Answer Type 26

Conclusion Combining syntactic parsing with ontology-based lookup in an interactive process of feedback and query refinement can increase the precision and recall of NLIs to ontologies, while reducing the time for customisation by shifting some tasks from application developers to end users. 27

Thank You!

More information... D. Damljanovic, M. Agatonovic, H. Cunningham: FREyA: an Interactive Way of Querying Linked Data, 1 st Workshop on Question-Answering over Linked Data, in conjunction with ESWC’11, (to appear) D. Damljanovic, M. Agatonovic, H. Cunningham: Natural Language Interfaces to Ontologies: Combining Syntactic Analysis and Ontology-based Lookup through the User Interaction. In Proceedings of the 7th Extended Semantic Web Conference (ESWC 2010), Springer Verlag, Heraklion, Greece, May 31-June 3, PDFPDF D. Damljanovic, M. Agatonovic, H. Cunningham: Identification of the Question Focus: Combining Syntactic Analysis and Ontology-based Lookup through the User Interaction. In Proceedings of the 7th Language Resources and Evaluation Conference (LREC 2010), ELRA 2010, La Valletta, Malta, May 17-23, PDF D. Damljanovic. Towards portable controlled natural languages for querying ontologies. In Rosner, M., Fuchs, N., eds.: Proceedings of the 2nd Workshop on Controlled Natural Language. Lecture Notes in Computer Science. Springer Berlin/Heidelberg, Marettimo Island, Sicily (September 2010)PDF