1. Cross-lingual Information Retrieval (CLIR) Johns Hopkins University
Classification
MATERIAL: QuickSTIR
Cross-lingual Information Retrieval (CLIR)
Kevin Duh
Johns Hopkins University
Classification

2. Outline Lessons learned: What worked Lessons learned: Negative results
Multi-view Document Representations
Score Function Tuning
Lessons learned: Negative results
Query expansion
Dynamic thresholding
Future challenges:
Reducing False Alarm
Exploiting crawled data for CLIR

3. Multi-view Document Representation
Robust handling of potential errors in 1-best transcription/translation

4. N-best Representation (details): Efficiency vs Accuracy
M-best
All MxN
MQWV
Somali
Sampled
Representation

5. Bag-of-Phrase (BOP) Representation
Extract all possible phrases from Phrase-based MT (PBMT) decoder

6. Experiment settings for the results I am about to report…
ASR: Semi-supervised Kaldi; MT: Moses w/ crawled bitext
5-best lists
Language
Query Set
Document Collection
ASR
WER
PBMT BLEU
Somali
QUERY1+2
(700 queries)
ANALYSIS1+2
(838 documents)
48.4
18.3
Swahili
QUERY2+3
(1000 queries)
(813 documents)
32.9
28.7
Tagalog
(844 documents)
40.3
33.0
MQWV (beta=40)  optimistic, but more stable comparison
System similar to official submission in 1S Evaluation:
CLIR AQWV - JHU: ; Other Teams: , ,

7. MQWV improvements due to Multi-View Representations

8. MQWV improvements due to Multi-View Representations

9. MQWV improvements due to Multi-View Representations

10. MQWV improvements due to Multi-View Representations

11. Multi-view Representation with many MT Systems (adding CyberTrans)

12. Multi-view Representation with many MT Systems (adding CyberTrans)

13. Multi-view Representation with many MT Systems (adding CyberTrans)

14. Scoring Function Tuning
Re-visit assumptions of classic BM25 IR score
Diverse document representations (varying length, output type) require re-tuning:
Inverse Document
Frequency of
query term
Frequency of
query term in document
Document
length ratio
Document
Query:
[q1,q2,…]
Tunable Hyperparameters

15. Saturation hyperparameter: k1
N-best+BoP have many repetitions,
so desire an early saturation

16. Outline Lessons learned: What worked Lessons learned: Negative results
Multi-view Document Representations
Score Function Tuning
Lessons learned: Negative results
Query expansion
Dynamic thresholding
Future challenges:
Reducing False Alarm
Exploiting crawled data for CLIR

17. Query Expansion for handling the rich structure of MATERIAL Query Language
Conceptual expansion:
1A:Query4474: Nakuru,travel+
1B:Query1133: cat, “population control”+
1B:Query2058: “food warnings”+, shrimp
1B:Query1120: caffeine, EXAMPLE_OF(vegetable)
Disambiguation:
1A:Query1619: hawk[hyp:animal]
1B:Query1657: fire[syn:burn], incident

18. Query Expansion for handling the rich structure of MATERIAL Query Language
Nukuru,travel+
travel
travelers, trips
tourism, tourist activities
personal expense, ...

19. Dynamic Thresholding to reduce gap with Oracle QWV
Query1
Doc23
Doc42
Doc02
Doc11
Query2
Doc03
Doc01
Doc02
Doc42
Query3
Doc12
Doc03
Doc04
Doc06
Single Threshold
(manually tuned)
Query1
Doc23
Doc42
Doc02
Doc11
Query2
Doc03
Doc01
Query3
Doc12
Doc04
Doc06
Dynamic Threshold
(predict per query)

20. Dynamic Thresholding to reduce gap with Oracle QWV

21. Outline Lessons learned: What worked Lessons learned: Negative results
Multi-view Document Representations
Score Function Tuning
Lessons learned: Negative results
Query expansion
Dynamic thresholding
Future challenges:
Reducing False Alarm
Exploiting crawled data for CLIR

22. Challenging Queries
We have a handle on Pmiss thanks to multi-view representation
Easily plug-in any and however many MT engines
But queries with high PFalseAlarm are difficult
Insufficient training data to learn intricacies in the query language

23. Ideal CLIR Architecture (my opinion)
Multi-view Representation
PBMT
NMT
RBMT
2nd stage CLIR
Learning-to-Rank
Doc06
Doc12
Doc03
Doc04
Reduce PFalseAlarm
WebCrawl CLIR data
1st stage CLIR
ElasticSearch
Doc12
Doc03
Doc04
Doc06
Query
Reduce Pmiss

24. Status: WikiCLIR silver training data in 25 languages
Available at:
English query
Relevant foreign
document

25. Can learning-to-rank models optimize MQWV? Probably yes!
High Correlation between MQWV and Mean Average Precision (MAP) on multiple CLIR systems and languages
MAP
There’s large literature on direct optimization of common IR metrics, e.g.
Yue (SIGIR2007) A Support Vector method for optimizing Average Precision
MQWV

26. Summary (1/2) Lessons learned: What worked
Multi-view Document Representations
Score Function Tuning

27. Summary (2/2) a Lessons learned: Negative results Future challenges:
Query expansion, Dynamic thresholding
Future challenges:
Reducing False Alarms
Exploiting crawled data for CLIR
PBMT
NMT
RBMT
2nd stage CLIR
Learning-to-Rank
Doc06
Doc12
Doc03
Doc04
Reduce PFalseAlarm
WebCrawl CLIR data
1st stage CLIR
ElasticSearch
Doc12
Doc03
Doc04
Doc06
Query
Reduce Pmiss