1. 02/04/09Danica Damljanović1 Natural Language Interfaces to conceptual models: usability and performance Danica Damljanović danica@dcs.shef.ac.uk

2. University of Sheffield NLP Outline NLIs to KBs and their usability QuestIO – Question-based Interface to Ontologies  Demo and evaluation Towards better usability using FREYA Conclusion 2 02/04/09 Danica Damljanović

3. University of Sheffield NLP Motivation Querying ontologies with existing query languages (e.g., SPARQL) is not straightforward:  complex syntax: not easy to learn,  writing queries is error-prone task,  requires understanding of Semantic Web technologies.

4. University of Sheffield NLP Danica Damljanović Too complex? select c0, p1, c2, p3, c4, p5, i6 from {c0} rdf:type { }, {c2} p1 {c0}, {c2} rdf:type { }, {c4} p3 {c2}, {c4} rdf:type { }, {i6} p5 {c4}, {i6} rdf:type { } where p1=http://gate.ac.uk/ns/gate- ontology#parameterHasType and p3=http://gate.ac.uk/ns/gate- ontology#hasRunTimeParameter and p5=http://gate.ac.uk/ns/gate-ontology#containsResource and i6=

5. University of Sheffield NLP 5 Danica Damljanović Semantic Search Interfaces Form-based, graphical:  Protégé (Noy et al., 2001)‏  KIM (Kiryakov et al., 2004)‏ Keyword based:  TAP Search (Guha et al., 2003)‏  SemSearch (Lei et al., 2006)‏ NLIs to KBs

6. University of Sheffield NLP Natural Language Interfaces Allow users to interact with a system using written or spoken language to perform tasks which require knowledge of a formal query language  NLIs to structured data: NLIs to DBs: TEAM, PRECISE NLIs to KBs  NLIs to semi-structured data Open-domain question answering systems  Other: E.g. NLC: replacement for a programming language Dialog and tutoring systems

7. University of Sheffield NLP Danica Damljanović NLIs to knowledge bases  (Kaufmann and Bernstein, 2007)‏ Natural Language Interfaces preferred to keywords, menu- guided, and graphical interfaces  (Linckels, 2007): keywords preferred to NL interfaces

8. University of Sheffield NLP 8 Danica Damljanović NLIs to KBs

9. University of Sheffield NLP 9 Danica Damljanović Usability of NLIs Who uses NLIs?  Application developers: customisation  End users: search Usability  Effectiveness  Efficiency  User satisfaction

10. University of Sheffield NLP 10 Danica Damljanović Usable NLIs to KBs: challenges Robustness Portability What to show? Understanding information need Habitability

11. University of Sheffield NLP Customisation of NLIs to KBs Ontology editing (e.g. using Protege)‏ Domain lexicon NLI for querying … Domain knowledge WordNet Domain expert Ontology engineer NLI for Ontology authoring

12. University of Sheffield NLP 12 Danica Damljanović Habitability Can the User and the System speak the same language?

13. University of Sheffield NLP 13 Danica Damljanović Design recommendations System Vocabulary>>User Vocabulary:  Feedback  Guided interfaces  Personalised vocabulary User Vocabulary>> System Vocabulary:  Clarification dialogs  Query refinement  Controlling the relevance  Ranking suggestions  Defining similarity User profiles

14. University of Sheffield NLP 14 Danica Damljanović Question-based Interface to Ontologies

15. University of Sheffield NLP Danica Damljanović QuestIO component diagram

16. University of Sheffield NLP 16 Danica Damljanović NL --> SPARQL query Filtering concepts Ranking concepts Query Creator Query Execution

17. University of Sheffield NLP An Example 1.15 1.19 compare

18. University of Sheffield NLP Scoring relations We combine three types of scores: similarity score - using Levenshtein similarity metrics we compare input string from the user with the relevant ontology resource specificity score is based on the subproperty relation in the ontology definition. 0 1

19. University of Sheffield NLP Scoring relations (II) ‏ distance score is inferring an implicit specificity of a property based on the level of the classes that are used as its domain and range.

20. University of Sheffield NLP Demo http://www.tao- project.eu/researchanddevelopment/demos anddownloads/movies/questio/questio.htmlhttp://www.tao- project.eu/researchanddevelopment/demos anddownloads/movies/questio/questio.html http://www.tao- project.eu/researchanddevelopment/demos anddownloads/movies/prototype- tutorial/prototype-tutorial.htmlhttp://www.tao- project.eu/researchanddevelopment/demos anddownloads/movies/prototype- tutorial/prototype-tutorial.html 20 Danica Damljanović

21. University of Sheffield NLP Danica Damljanović Evaluation coverage and correctness scalability and portability

22. University of Sheffield NLP Evaluation on coverage and correctness 36 questions extracted from GATE list 22 out of 36 questions were answerable (the answer was in the knowledge base):  12 correctly answered (54.5%)‏  6 with partially corrected answer (27.3%)‏  system failed to create a SeRQL query or created a wrong one for 4 questions (18.2%)‏ Total score:  68% correctly answered  32% did not answer at all or did not answer correctly

23. University of Sheffield NLP 23 Danica Damljanović Comparison with AquaLog We removed 6 questions that we knew were not supported by Aqualog 1 conjunction query “What are the run parameters of POS Tagger and Sentence splitter?” 1 query with brackets “Does GATE have a coreference resolution component (PR)?” 1 query starting with “How many... ” 3 queries not in a form of a full-blown question, for example “I cannot get Wordnet plugin to work“.

24. University of Sheffield NLP Evaluation on scalability and portability

25. University of Sheffield NLP Danica Damljanović Evaluation on scalability

26. University of Sheffield NLP User-centric task-based evaluation Training: using video tutorials 12 participants, 4 tasks:  3 defined, e.g. “Find runtime parameters of Cebuano gazetteer.”  1 free task: “Think of any task that you would like to perform using this prototype.” Measured:  Efficiency, Effectiveness and User satisfaction 26 Danica Damljanović

27. University of Sheffield NLP Efficiency: average time per task

28. University of Sheffield NLP Effectiveness: how successfully the tasks were finished?

29. University of Sheffield NLP Post –task survey: question 1

30. University of Sheffield NLP Post –task survey: question 2

31. University of Sheffield NLP Post –task survey: question 3

32. University of Sheffield NLP Post –task survey: question 4

33. University of Sheffield NLP Post –task survey: question 5

34. University of Sheffield NLP Evaluation: conclusion Browsing ontology mostly helpful – but only for users who are familiar with ontologies Refinement pane: primitive, still favourable for defined tasks Tasks not finished:  in cases when the user information need was not precisely expressed (undefined tasks)‏  When the answer was not in the ontology 34 Danica Damljanović

35. University of Sheffield NLP “find out which are the runtime parameters of Cebuano Gazetteer” “cebuano gazetter parameters” “ What are the runtime parameters of cebuano gazetteer?“ “what are the parameters of cebuano gazetteer?” “Cebuano gazetteer runtime parameters“ “Runtime parameters of cebuano gazetteer“ “Cebuano runtime parameters“ “Cebuano gazeteer“>>“Cebuano gazetteer“  Find parameters by browsing the ontology 35 Danica Damljanović

36. University of Sheffield NLP “runtime parameters of Cebuano Gazetteer” “Cebuano runtime parameters“ GATE plugin Resource Parameter Processing Resource ContainsResource hasInitTimeParameter hasRuntimeParameter

37. University of Sheffield NLP Undefined tasks “Developer of Tokeniser” ”Projects about GATE ”, ”GATE web site” ” Tokenizer ” vs. ” Tokeniser ” ” Gazetteer ” vs. ”Gazeteer ” ”horacio saggion publications” vs. ”horacio saggion articles” ”Author of morphological analyser ” >> ”Developer of morphological analyser ” >> ”Developer and morphological analyser ” 37 04/06/09 Danica Damljanović

38. University of Sheffield NLP Next steps: FREyA 38 Danica Damljanović FREyA (Feedback, Refinement, Extended vocabulary, Agregation)‏

39. University of Sheffield NLP Example the user's query:  Cities in Europe FREyA:  Cities >>locatedIn >>Countries >> locatedIn >> Europe  Cities >>locatedIn >> Countries >> partOf >> Europe Attempto:  which cities are located in countries that are part of Europe? 39 04/06/09 Danica Damljanović

40. University of Sheffield NLP Challenges Return answers in real time:  which rivers flow through Germany >> which rivers flow through (cities in) Germany Refinement model - an ontology?  with concepts such as query (hasAnswer) answer, refinement, feedback, subject of query (main subject), domain, context.  find out what the user wants and instantiate the refinement ontology. 40 Danica Damljanović

41. University of Sheffield NLP 41 Danica Damljanović Expected contribution Usability improvement of NLIs:  Eliminate training  Improve the performance by implementing user-system interaction using FREYA: help the user to easily familiarize himself with the system capabilities express his need more precisely, in a way which is understandable by the system

42. University of Sheffield NLP 42 Danica Damljanović Thank you Questions?

43. University of Sheffield NLP References (Noy et al., 2001)N. Noy, M. Sintek, S. Decker, M. Crubezy, R. Fergerson, and M. Musen. Creating Semantic Web Contents with Protege-2000. IEEE Intelligent Systems, 16(2):60-71, 2001. (Kiryakov et al., 2004) A. Kiryakov, B. Popov, D. Ognyano, D. Manov, A. Kirilov, and M. Goranov. Semantic annotation, indexing and retrieval. Journal of Web Semantics, ISWC 2003 Special Issue, 1(2):671-680, 2004. (Guha et al., 2003) R. Guha, R. McCool, and E. Miller. Semantic search. In WWW '03: Proceedings of the 12th international conference on World Wide Web, pages 700-709, New York, NY, USA, 2003. ACM. (Lei et al., 2006) Y. Lei, V. Uren, and E. Motta. Semsearch: a search engine for the semantic web. In Managing Knowledge in a World of Networks, pages 238{245. Springer Berlin /Heidelberg, 2006. (Kaufmann & Bernstein, 2007) E. Kaufmann and A. Bernstein. How useful are natural language interfaces to the semantic web for casual end-users? In Proceedings of the Forth European Semantic Web Conference (ESWC 2007), Innsbruck, Austria, June 2007. (Serge Linckels, 2007) C. M. Serge Linckels. Semantic interpretation of natural language user input to improve search in multimedia knowledge base. it - Information Technologies, 49(1):40-48, 2007.