1. Evaluating Semantic Search Query Approaches with Expert and Casual Users
Khadija Elbedweihy, Stuart N. Wrigley, and Fabio Ciravegna
Reported by Xiangqian Lee @ Websoft

2. Contents Introduction Usability Study Results Conclusion
Experiment Setup
Results
Results dependent on user types
Results independent on user types
Conclusion

3. Introduction
Traditional metrics to evaluate an IR system: precision & recall
More and more focus on Interactive IR(IIR).But still more focus on performance, less on the user side.
Divide users to two groups:
Casual users(for SW)
Experts(for SW)

4. Introduction 4 kinds of semantic search tools: Graph-based tools
Affective Graphs1
Semantic-Crystal4
Form-based tools
K-search
Controlled Natural Language Tools(with guidance)
Ginseng3
Free Natural Language Tools
NLP-Reduce2 1 2 3 4

5. Usability Study/Experiment Setup
Dataset: geography dataset within the Mooney NL Learning Data
5 questions for users to solve:
Give me all the capitals of the USA ?
What are the cities in states through which the Mississippi runs ?
Which states have a city named Columbia with a city population over 50,000?
Which lakes are in the state with the highest point ?
Tell me which rivers do not traverse the state with the capital Nashville?

6. Usability Study/Experiment Setup
Details
20 subjects: half for experts & half for casual users.
Use tools in random order & solve the questions in random order.
Focus on:
1)input time;
2)num of attempts
3)Answer found rate
More: System Usability Scale(SUS) questionnaire
Extended by:
1) Tool Rank
2) Query Interface Rank,
3) Results Content Rank,
4) Results Presentation Rank.

7. Results/User Dependence Results
Experts:

8. Results/User Dependence Results
Experts:
Prefer Graph- & Form- based approaches
Both view-based
Can form more complex queries(visualize search space & the connections between concepts)
Experts prefer Affective Graphs to Semantic-Crystals:
Intuitive & pleasant to use
Difference:
presents all ontologies
Shows ontologies & relationships only selected by users

9. Results/User Dependence Results
Experts
Frustrated by controlled NL
Restricting the vocabulary was annoying
Longest input time
Feedbacks:
Cannot construct queries in the way expert want.
Need to know the vocabulary in advance.
Guidance helps not much to reduce frustration.

10. Results/User Dependence Results
Casual Users

11. Results/User Dependence Results
Casual Users
Graph-based tools more complex if entire ontology not shown.
Prefer Semantic-Crystal to Affective Graphs.
Tool Interface is important.
Like animated,modern and visually-appealing design.
Casual Users prefer form-based tools
Less complicated than graph-based ones.
Allowing more complex queries than NL-based ones.
Presence of inverse relations are viewed as redundancy.
Casual users like controlled-NL support
Prefer to be controlled by vabularies to having more expressiveness.

12. Results independent of user type
View-based Tools
NL-based Tools
Graph-based
Tools
Form-based
Tools
ControlledNL
Tools
Free NL
Tools
Ability to construct more complex queries
Simple and Natural

13. Results independent of user type
Form-based faster but more tedious than graph-based
Hard to find what I was looking for once
Free-NL: simplest, most natural
Feedbacks: have to guess right
words
Have most attempt times
Results Content & presentations
are important
more related info with results
provide similar searches

14. Results independent of user type
Experts like the ability to see the formal representations of the query.
Experts plan query formulation more. Casual users more like try-and-error.

15. Thanks!