Query Processing and Reasoning How Useful are Natural Language Interfaces to the Semantic Web for Casual End-users? Esther Kaufmann and Abraham Bernstein Presented By Stephen Lynn

Query Processing and Reasoning Overview  Natural Language Interfaces  Goals/Objectives  Introduce 4 Interfaces  Experiment  Evaluation Results  Future Work

Query Processing and Reasoning Natural Language Interfaces  Plain text queries  Phrases  Full Sentences  Challenges  Linguistic Variability (ambiguous meaning)  Domain Independence  Retrieval Performance (linked to portability)  Usefulness of NLIs

Query Processing and Reasoning Goals/Objectives Usability of NLIs Usefulness of NLIs

Query Processing and Reasoning Evaluation Interfaces  Portable  Domain-Independent  Good Performance  4 Interfaces  Least to Most Restrictive

Query Processing and Reasoning NLP-Reduce  Free-form text query  Remove Stop Words/Puncuation  Word Stemming  Identify Triple Structures (no details)  Enhanced Triple Store (WordNet)  Generate SPARQL  Return Results

Query Processing and Reasoning NLP-Reduce

Query Processing and Reasoning Querix  Parse Query  Extract Query Skeleton from Syntax Tree  Identifies Triple Patterns  Match Triples to Knowledge Base Resources  Generate SPARQL  Enhanced with WordNet Synonyms  Return Results

Query Processing and Reasoning Querix

Query Processing and Reasoning Querix – Ambiguity Resolution  What is the biggest state in the US?

Query Processing and Reasoning Ginseng  UI based on a grammar  Built dynamically from target knowledgebases  Incremental Parser  Offer possible completions (code completion)  Only accepts entries in list  No invalid queries  Convert to SPARQL  Return Results

Query Processing and Reasoning Ginseng

Query Processing and Reasoning Symantic Crystal  Graphical Display of Ontology  Select Elements in Ontology  No Invalid Queries  Specify Constraints  Incrementally Build Query  Generate SPARQL  Return Results

Query Processing and Reasoning Semantic Crystal

Query Processing and Reasoning Usability Study  How usable and useful are NLI applications?  Setup  48 subjects  4 interfaces  Same 4 questions for each interface (minor changes)  Area of Alaska?  Number of lakes in Florida?  States that have city named Springfield?  Rivers run through state that has largest city in US?  Change sequence of interfaces

Query Processing and Reasoning Experiment 1.Read Introduction Notes 2.Instructions on Interface #1 3.Answer 4 questions with interface 4.Fill out Usability survey about Interface 5.Repeat 2-4 for other Interfaces 6.Fill out Comparison Questionnaire

Query Processing and Reasoning Evaluation Results

Query Processing and Reasoning Evaluation Results

Query Processing and Reasoning Strengths  Good General Points  Automation is good (not Sematic Crystal)  Result format affects user trust  Balance between freedom and restriction  User Evaluation  Analysis

Query Processing and Reasoning Weaknesses  Completion time not a deciding factor in satisfaction  Still pushing Semantic Crystal  Personal Attachment  Unclear distinction between QL and Interface

Query Processing and Reasoning Future Work  Compare with more NLIs  Multiple Domains  Single Infrastructure w/Different Uis  Evaluate Usability/Usefulness