Interactive Probabilistic Search for GikiCLEF Ray R Larson School of Information University of California, Berkeley Ray R Larson School of Information University of California, Berkeley

September 21, 2007CLEF Corfu, Greece GikiCLEF Task  Perform QA style retrieval for complex questions with geographic elements  Task specifications were not really clear (to me at least) since only Wikipedia article titles that WERE answers were acceptable - and not articles that CONTAINED answers if the article “type” was wrong  Perform QA style retrieval for complex questions with geographic elements  Task specifications were not really clear (to me at least) since only Wikipedia article titles that WERE answers were acceptable - and not articles that CONTAINED answers if the article “type” was wrong

September 21, 2007CLEF Corfu, Greece GikiCLEF Task  Questions were VERY complex, such as:  Which countries have the white, green and red colors in their national flag?  Which authors were born in and write about the Bohemian Forest?  What Belgians won the Ronde van Vlaanderen exactly twice?  List the left side tributaries of the Po river  Questions were VERY complex, such as:  Which countries have the white, green and red colors in their national flag?  Which authors were born in and write about the Bohemian Forest?  What Belgians won the Ronde van Vlaanderen exactly twice?  List the left side tributaries of the Po river

September 21, 2007CLEF Corfu, Greece Approach to GikiCLEF  We had no idea about how to handle many of these questions  So, we decided to devote our participation to exploring approaches via an interactive interface to the Cheshire II system  We wanted to see what techniques would be effective (and which not) in suggesting documents with relevant content  At least until we realized that relevant content was not a relevant answer  We had no idea about how to handle many of these questions  So, we decided to devote our participation to exploring approaches via an interactive interface to the Cheshire II system  We wanted to see what techniques would be effective (and which not) in suggesting documents with relevant content  At least until we realized that relevant content was not a relevant answer

September 21, 2007CLEF Corfu, Greece Adapting Cheshire II for GikiCLEF  For this task we created an interactive version of the database which included:  Multiple indexes and re-implementation of links in the wikipedia corpus as title searches to both retrieve identical and similar titles  Cross-searching between the different language corpora  Some parts of some queries were better searched in specific languages  Relied on semi-intelligent translations (me) and occasionally Babelfish  For this task we created an interactive version of the database which included:  Multiple indexes and re-implementation of links in the wikipedia corpus as title searches to both retrieve identical and similar titles  Cross-searching between the different language corpora  Some parts of some queries were better searched in specific languages  Relied on semi-intelligent translations (me) and occasionally Babelfish

September 21, 2007CLEF Corfu, Greece

September 21, 2007CLEF Corfu, Greece

September 21, 2007CLEF Corfu, Greece

September 21, 2007CLEF Corfu, Greece Not all topics are so simple  Some topics require multiple background searches to help determine possible answers  E.g. Searching for the football teams of all South American countries requires that you know all South American Countries  Often Wikipedia List pages are available for these kinds of questions -- but are not themselves considered relevant in this task  For example…  Some topics require multiple background searches to help determine possible answers  E.g. Searching for the football teams of all South American countries requires that you know all South American Countries  Often Wikipedia List pages are available for these kinds of questions -- but are not themselves considered relevant in this task  For example…

September 21, 2007CLEF Corfu, Greece

September 21, 2007CLEF Corfu, Greece

September 21, 2007CLEF Corfu, Greece NOT RELEVANT ANSWER?

September 21, 2007CLEF Corfu, Greece RELEVANT - But no way to verify without the flag page or images and computer vision analysis

September 21, 2007CLEF Corfu, Greece Multilingual Search  Conducted English search first and identified what I thought were relevant items  Then did the same with each other language using the English results as a guide, but open to new relevant items not in the English collections  Usually relied on translation approximations and cognates  E.g. Bulgarian had enough similarities to Russian to be able to get a sense of the meaning  Conducted English search first and identified what I thought were relevant items  Then did the same with each other language using the English results as a guide, but open to new relevant items not in the English collections  Usually relied on translation approximations and cognates  E.g. Bulgarian had enough similarities to Russian to be able to get a sense of the meaning

September 21, 2007CLEF Corfu, Greece Search Methods  Basic ranked search was not very effective on its own  Using ranked search with all terms required (boolean constraint) was more effective  Sometimes the best approach was a simple Boolean exact match on names  But when going across languages may need ranked approximate searches too  Definitely need a type classification index for pages that can be used to constrain results  Probably need to use link indexes more often too (to find pages of the correct type that link to the relevant page)  Basic ranked search was not very effective on its own  Using ranked search with all terms required (boolean constraint) was more effective  Sometimes the best approach was a simple Boolean exact match on names  But when going across languages may need ranked approximate searches too  Definitely need a type classification index for pages that can be used to constrain results  Probably need to use link indexes more often too (to find pages of the correct type that link to the relevant page)

September 21, 2007CLEF Corfu, Greece Limitations  Interactive search was a very slow process  Often took several hours per question  The short test period fell during a family vacation  It is more fun to go out to dinner at a nice restaurant than to try to translate Bulgarian  Interactive search was a very slow process  Often took several hours per question  The short test period fell during a family vacation  It is more fun to go out to dinner at a nice restaurant than to try to translate Bulgarian

September 21, 2007CLEF Corfu, Greece Limitations  As a result of the preceding constraints I was only able to complete 22 of 50 topics  But each included all languages whenever possible  In spite of this (and since scoring penalizes wrong answers as well as rewarding correct ones…) manage to score pretty well  As a result of the preceding constraints I was only able to complete 22 of 50 topics  But each included all languages whenever possible  In spite of this (and since scoring penalizes wrong answers as well as rewarding correct ones…) manage to score pretty well

September 21, 2007CLEF Corfu, Greece Results from GikiCLEF Site

September 21, 2007CLEF Corfu, Greece Conclusions  The interactive approach showed some search strategies than might be exploited automatically  Automatic approaches that rely only on conventional IR techniques will probably continue to lag the “knowledge-based” approaches used for this task  The trick for the future will be trying to effective combine the two  The interactive approach showed some search strategies than might be exploited automatically  Automatic approaches that rely only on conventional IR techniques will probably continue to lag the “knowledge-based” approaches used for this task  The trick for the future will be trying to effective combine the two