1. CLEF 2008 Multilingual Question Answering Track
UNED
Anselmo Peñas
Valentín Sama
Álvaro Rodrigo
CELCT
Danilo Giampiccolo
Pamela Forner

2. QA 2008 Task and Exercises QA Main task (6th edition)
Pilot: QA WSD, English newswire collections with Word Sense Disambiguation
Answer Validation Exercise – AVE (3rd edition)
QA on Speech Transcripts – QAST (2nd edition)

3. Main Task QA 2008 Organizing Committee
CELCT (D. Giampiccolo, P. Forner): Italian
UNED (A. Peñas): Spanish
U. Groeningen (G. Bosma): Dutch
U. Limerick (R. Sutcliff): English
DFKI (B. Sacalenau): German
ELDA/ELRA (N. Moreau): French
Linguateca (P. Rocha): Portuguese
Bulgarian Academy of Sciences (P. Osenova): Bulgarian
IASI (C. Forascu): Romanian
U. Basque Country (I. Alegria): Basque
ILSP (P.Prokopidis): Greek

4. Evolution of the Track 2003 2004 2005 2006 2007 2008 Target languages9 10 11
Collections
News 1994
+News 1995
+Wikipedia Nov. 2006
Type of questions
200 Factoid
+ Temporal restrictions
+ Definitions
- Type of question
+ Lists
+ Linked questions
+ Closed lists
Supporting information
Doc.
Snippet
Pilots and Exercises
Temporal restrictions
Lists
AVE
Real Time
WiQA
QAST
WSDQA

5. 200 questions FACTOID DEFINITION CLOSED LIST LINKED QUESTIONS
(loc, mea, org, per, tim, cnt, obj , oth)
DEFINITION
(per, org, obj, oth)
CLOSED LIST
Who were the components of The Beatles?
Who were the last three presidents of Italy?
LINKED QUESTIONS
Who was called the “Iron-Chancellor”?
When was he born?
Who was his first wife?
Temporal restrictions by date, by period, by event
NIL questions (without known answer in the collection)

6. SOURCE LANGUAGES (questions)
43 Activated Language Combinations (at least one registered participant)
TARGET  LANGUAGES  (corpus and answers) BG DE EL EN ES EU FR IT NL PT RO
SOURCE LANGUAGES (questions)

7. Activated Tasks MONOLINGUAL CROSS-LINGUAL TOTAL CLEF 2003 3 5 87
Activated Tasks
MONOLINGUAL
CROSS-LINGUAL
TOTAL
CLEF 2003 3 5 8
CLEF 2004 6 13 19
CLEF 2005 15 23
CLEF 2006 7 17 24
CLEF 2007 29 37
CLEF 2008 10 33 43

8. Submitted runs Submitted runs Monolingual Cross-lingual CLEF 2003 17 6
Submitted runs
Monolingual
Cross-lingual
CLEF 2003 17 6 11
CLEF 2004
48 (+182%) 20 28
CLEF 2005
67 (+40%) 43 24
CLEF 2006
77 (+15%) 42 35
CLEF 2007
37 (-52%)
CLEF 2008
51 (+38%) 31 8

9. Participant groups CLEF 2003 - 8 CLEF 2004 13 5 18 (+125%) 22
Newcomers
Veterans
TOTAL
Registered
CLEF 2003 - 8
CLEF 2004 13 5 18
(+125%) 22
CLEF 2005 9 15 24
(+33%) 27
CLEF 2006 10 20 30
(+25%) 36
CLEF 2007 14
(-26%) 29
CLEF 2008 21 33

10. List of Participants (random order)
Bulgaria

11. Groups per year and target collection
Natural selection?
Task Change
Above 20 groups

12. Groups per target collection

13. 2008 participation: Comparative evaluation?
Language
Runs
Different groups
Portuguese 9 6
Spanish 10 4
English 5
German 11 3
Romanian 2
Dutch 1
Basque
French
Bulgarian
Italian
Greek
Lack from evaluation perspective:
4 languages without comparison between different groups
Breakout session

14. Results: Best and Average scores

15. Best scores by language

16. Best scores by participant

17. Results depend on type of questions
Definitions
Almost solved for several systems 80%-95%
Factoids
50%-65% for several systems
Temporal restrictions
Same level of difficulty as factoids for some systems
Closed lists
Still very difficult
Linked questions
Now wikipedia provides more answers

18. Conclusion Same task as 2007
Same level of participation (slightly better)
11 target languages (9 with participation)
43 activated subtasks
21 participants
51 runs
Same results (slightly better)

19. Future direction Less participants per language
Poor comparison
Change methodology: one task for all
Critics to QA over wikipedia
Easier to find questions with IR
No user model
Change collection
QA proposal for 2009
SC and breakout