1. 8/25/2014Danica Damljanović1 Natural Language Interfaces to Ontologies: usability and performance (Transfer report) Student: Danica Damljanović Supervisor: Prof. Dr Hamish Cunningham

2. University of Sheffield NLP Introduction Semantic Web Ontologies and knowledge bases  Semantic search Semantic Web interfaces

3. University of Sheffield NLP Danica Damljanović Too complex? select c0, p1, c2, p3, c4, p5, i6 from {c0} rdf:type { }, {c2} p1 {c0}, {c2} rdf:type { }, {c4} p3 {c2}, {c4} rdf:type { }, {i6} p5 {c4}, {i6} rdf:type { } where p1=http://gate.ac.uk/ns/gate- ontology#parameterHasType and p3=http://gate.ac.uk/ns/gate- ontology#hasRunTimeParameter and p5=http://gate.ac.uk/ns/gate-ontology#containsResource and i6=

4. University of Sheffield NLP 4 8/25/2014 Danica Damljanović Semantic Search Interfaces Protégé (Noy et al., 2001) KIM (Kiryakov et al., 2004) TAP (Guha et al., 2003)  GetData  Reflection  TAPSearch SemSearch (Lei et al., 2006) NLIs to KBs

5. University of Sheffield NLP Danica Damljanović User preferences  (Kaufmann and Bernstein, 2007) Natural Language Interfaces preferred to keywords, menu- guided, and graphical interfaces  (Linckels, 2007): keywords preferred to NL interfaces

6. University of Sheffield NLP 6 8/25/2014 Danica Damljanović NLIs to KBs: challenges Robustness Portability What to show? Understanding information need Habitability Usability

7. University of Sheffield NLP 7 8/25/2014 Danica Damljanović Natural Language Interfaces Usability  Effectiveness  Efficiency  User satisfaction  Who uses NLIs? Application developers: customisation End users: search

8. University of Sheffield NLP 8 8/25/2014 Danica Damljanović Customisation and Retrieval Performance

9. University of Sheffield NLP Customisation of NLIs to KBs Ontology editing (e.g. using Protege) Domain lexicon NLI for querying … Domain knowledge WordNet Domain expert Ontology engineer NLI for Ontology authoring

10. University of Sheffield NLP 10 8/25/2014 Danica Damljanović End users point of view Can the User and the System speak the same language?

11. University of Sheffield NLP 11 8/25/2014 Danica Damljanović Design recommendations System Vocabulary>>User Vocabulary:  Feedback  Guided interfaces  Personalised vocabulary User Vocabulary>> System Vocabulary:  Clarification dialogs  Query refinement  Controlling the relevance  Ranking suggestions  Defining similarity User profiles

12. University of Sheffield NLP 12 8/25/2014 Danica Damljanović First experiments: Question-based Interface to Ontologies

13. University of Sheffield NLP Danica Damljanović QuestIO component diagram

14. University of Sheffield NLP 14 8/25/2014 Danica Damljanović NL --> SeRQL query

15. University of Sheffield NLP An Example 1.15 1.19 compare

16. University of Sheffield NLP Scoring relations We combine three types of scores: similarity score - using Levenshtein similarity metrics we compare input string from the user with the relevant ontology resource specificity score is based on the subproperty relation in the ontology definition. 0 1

17. University of Sheffield NLP Scoring relations (II) ‏ distance score is inferring an implicit specificity of a property based on the level of the classes that are used as its domain and range.

18. University of Sheffield NLP Danica Damljanović Evaluation: Initial results We evaluated: coverage and correctness scalability and portability

19. University of Sheffield NLP Evaluation on coverage and correctness 36 questions extracted from GATE list 22 out of 36 questions were answerable (the answer was in the knowledge base):  12 correctly answered (54.5%)‏  6 with partially corrected answer (27.3%)‏  system failed to create a SeRQL query or created a wrong one for 4 questions (18.2%)‏ Total score:  68% correctly answered  32% did not answer at all or did not answer correctly

20. University of Sheffield NLP 20 8/25/2014 Danica Damljanović Comparison with Aqualog We removed 6 questions that we knew were not supported by Aqualog 1 conjunction query “What are the run parameters of POS Tagger and Sentence splitter?” 1 query with brackets “Does GATE have a coreference resolution component (PR)?” 1 query starting with “How many... ” 3 queries not in a form of a full-blown question, for example “I cannot get Wordnet plugin to work“.

21. University of Sheffield NLP Evaluation on scalability and portability Sizes of the knowledge bases created based on: GATE ontology: http://gate.ac.uk/ns/gate-ontology Travel ontology: http://goodoldai.org.yu/ns/tgproton.owl Ontologies have not been customised or changed prior using with QuestIO!

22. University of Sheffield NLP Evaluation on scalability and portability

23. University of Sheffield NLP Danica Damljanović Evaluation on scalability

24. University of Sheffield NLP 24 8/25/2014 Danica Damljanović Expected contributions and research plan Addressing challenges without compromising retrieval performance:  How existing NLIs to KBs can increase the performance without a significant additional cost for customisation?  How methods for assisting the user in formulating the query can influence the retrieval performance? Richer navigation and semantic search interface which extend the concept of faceted search  Improving QuestIO

25. University of Sheffield NLP Expected contributions and research plan Evaluation with golden standard dataset which would be one of the contributions of this PhD:  Knowledge base  Set of queries  Set of relevant SPARQL queries Exploring methods for usability and their impact on retrieval performance

26. University of Sheffield NLP Related publications Journals: Danica Damljanovic, Kalina Bontcheva: Towards Enhanced Usability of Natural Language Interfaces. Annals of IS: Special Issue on “Semantic Web & Web 2.0”. Springer. 2009. (to appear) Workshops: Damljanovic, D.,Bontcheva, K.. Enhanced Semantic Access to Software Artefacts. In Workshop on Semantic Web Enabled Software Engineering (SWESE) held in conjunction with ISWC'08, Karlsruhe, Germany, October 2008. Conferences: Tablan, V., Damljanovic, D., Bontcheva, K.. A natural language query interface to structured information. In Proceedings of the 5h European Semantic Web Conference (ESWC 2008), Tenerife, Spain, June, 2008. Damljanovic, D., Tablan, V., Bontcheva, K.: A text-based query interface to owl ontologies. In: 6th Language Resources and Evaluation Conference (LREC), Mar- rakech, Morocco, ELRA (May 2008). Posters: Natural Language Queries for Enhanced Knowledge Access presented at Summer School on Multimedia Semantics Analysis, Annotation, Retrieval and Applications (SSMS07), Glasgow, UK, July 15-21, 2007.

27. University of Sheffield NLP Thanks …you for your attention! Hamish Cunningham and Kalina Bontcheva for continuous support Valentin Tablan for the help with QuestIO Vanesa Lopez from Open University for allowing me to play with AquaLog Professor Abraham Bernstein and Esther Kaufmann from the University of Zurich for sharing with me the Mooney dataset in owl format. Milan Agatonovic for continuous support

28. University of Sheffield NLP References (Noy et al., 2001)N. Noy, M. Sintek, S. Decker, M. Crubezy, R. Fergerson, and M. Musen. Creating Semantic Web Contents with Protege-2000. IEEE Intelligent Systems, 16(2):60-71, 2001. (Kiryakov et al., 2004) A. Kiryakov, B. Popov, D. Ognyano, D. Manov, A. Kirilov, and M. Goranov. Semantic annotation, indexing and retrieval. Journal of Web Semantics, ISWC 2003 Special Issue, 1(2):671-680, 2004. (Guha et al., 2003) R. Guha, R. McCool, and E. Miller. Semantic search. In WWW '03: Proceedings of the 12th international conference on World Wide Web, pages 700-709, New York, NY, USA, 2003. ACM. (Lei et al., 2006) Y. Lei, V. Uren, and E. Motta. Semsearch: a search engine for the semantic web. In Managing Knowledge in a World of Networks, pages 238{245. Springer Berlin /Heidelberg, 2006. (Kaufmann & Bernstein, 2007) E. Kaufmann and A. Bernstein. How useful are natural language interfaces to the semantic web for casual end-users? In Proceedings of the Forth European Semantic Web Conference (ESWC 2007), Innsbruck, Austria, June 2007. (Serge Linckels, 2007) C. M. Serge Linckels. Semantic interpretation of natural language user input to improve search in multimedia knowledge base. it - Information Technologies, 49(1):40-48, 2007.