1. LDMT MURI Data Collection and Linguistic Annotations November 4, 2011 Jason Baldridge, UT Austin Ulf Hermjakob, USC/ISI

2. Purpose Collect and build data Monolingual text Bilingual text Linguistic annotations to support work on machine translations for Kinyarwanda-English Malagasy-English

3. Overview Source, type and size of data Language consultants Kinyarwanda data Malagasy data Annotation An idea Accomplishments, challenges, future releases

4. Text sources Bible (highly multilingual parallel corpus) Dictionaries, phrasebooks Interview transcripts Newspapers

5. Pbook (0.9k)Pbook (0.7k) GWord (8b) BILINGUAL (16k) ENGLISH monolingual (huge) KINYARWANDA monolingual (7m) ENG treebank ENG text KIN text KIN treebank PTB (1m) word align Kinyarwanda Data Resources 1.0 Release 2.0 Release News (7m) KGMC (5.8k)KGMC (4.8k) Dict (9k)Dict (8k) NONE word counts

6. KGMC (270k)KGMC (225k) Pbook (0.9k)Pbook (0.7k) GWord (8b) BILINGUAL (285k) ENGLISH monolingual (huge) KINYARWANDA monolingual (7m) ENG treebank ENG text KIN text KIN treebank PTB (1m) word align Kinyarwanda Data Resources 1.0 Release 2.0 Release News (7m) KGMC (5.8k)KGMC (4.8k) BBC (0.3k) IGT (0.1k)IGT (0.06k) Dict (9k)Dict (8k) NONE KGMC (2.9k)KGMC (3.8k) BBC (0.3k) IGT (0.06k) IGT (0.1k) NOTE: no gold morph-split text word counts

7. Bible (730k)Bible (725k) Gword (8b) BILINGUAL (730k) ENGLISH monolingual (huge) MALAGASY monolingual (zero) ENG treebank ENG text MLG text MLG treebank PTB (1m) word align Malagasy Data Resources 1.0 Release 2.0 Release NONE none

8. Bible (730k)Bible (725k) News (2.1k)News (2.3k) Gword (8b) BILINGUAL (732k) ENGLISH monolingual (huge) MALAGASY monolingual (zero) ENG treebank ENG text MLG text MLG treebank PTB (1m) word align Malagasy Data Resources 1.0 Release 2.0 Release NONE none News (2.1k)News (2.3k) NOTE: no gold morph-split text

9. Quality of Original Texts Perfectly clean: English Bible Reasonably edited: Newspapers (kin/mlg) Uneven editing: Genocide protocols – Spelling errors – missing/sloppy punctuation – untranslated text (missing or still in source language) Kinyarwanda word ikaragiro (which means dairy) repeatedly translated as diary. “... over there, the houses that belong to the diary.”

10. Native speaker consultants UT reached out to speakers of both languages Kinyarwanda – Several speakers near Austin – Most would like some payment – One has helped with translation and consultation Malagasy speakers – Many speakers from around US and Canada – Most would like some payment – Two have helped with translations

11. Native speaker consultants At this point, UT does need to have access to paid informants. – Need texts from other genres translated – Need to ask questions about meanings of some sentences for linguistic analysis The CMU-Rwanda initiative may provide us with a further avenue for obtaining consultants for Kinyarwanda. – Also a potential source of data

12. Overview Source, type and size of data Language consultants Kinyarwanda data Malagasy data Annotation An idea Accomplishments, challenges, future releases

13. KGMC Transcripts Collaboration between Kigali Genocide Memorial Center and the Human Rights Documentation Initiative at UT Austin Library – http://www.lib.utexas.edu/hrdi/ http://www.lib.utexas.edu/hrdi/ – http://www.kigalimemorialcentre.org http://www.kigalimemorialcentre.org Transcriptions of survivor testimonies filmed for the Genocide Archive Rwanda http://www.genocidearchiverwanda.org.rw/index.php/Welcome_to_Genocide_Archive_Rwanda http://www.genocidearchiverwanda.org.rw/index.php/Kmc00005-sub2-eng-glifos

14. KGMC Data 48 translated transcripts – all translated into English – 33 into French 41 untranslated transcripts (only Kinyarwanda)

15. KGMC Data Original format: Microsoft Word, in tables

16. KGMC Data normalization Converted to XML using a semi-automatic process Each language represented side-by-side Script to process the MS Word format – Iteratively modified based on output and error detection – Needed to handle missing data and misalignments between time spans across translations Final manual verification and correction of each file.

17. Example XML

18. Overview Source, type and size of data Language consultants Kinyarwanda data Malagasy data Annotation An idea Accomplishments, challenges, future releases

19. Malagasy Bible Online version of 1865 Malagasy Bible – http://www.madapourchrist.org/ http://www.madapourchrist.org/ Preparation: – Convert HTML to text – Align with the NET Bible (New English Translation) using verses – Currently have 686 chapters aligned Obvious problem: 150 year-old Malagasy text

20. Malagasy Dictionary Online dictionary of Malagasy – http://malagasyworld.org http://malagasyworld.org 63k words – English definitions for 8000 words – French definitions for 10,000 words Includes parts-of-speech, mostly coarse- grained (noun, verb, adjective, etc.)

21. Malagasy Dictionary Scraped and processed to produce clean XML

22. Malagasy texts Texts from six webpages – 3 from Lakroa: http://www.lakroa.mg/http://www.lakroa.mg/ – 3 from Lagazette: http://www.lagazette-dgi.com/http://www.lagazette-dgi.com/ Translated by native speakers to English to create small parallel corpus for initial analysis and annotation.

23. Overview Source, type and size of data Language consultants Kinyarwanda data Malagasy data Annotation An idea Accomplishments, challenges, future releases

24. Morphological analysis UT Austin obtained and adapted XFST analyzer created by Dalrymple, Liakata and Mackie 2006. Applied it to the Malagasy website texts from Lakroa and Lagazette, hand-selecting the correct analysis for each word. These need to be integrated with the standard tokenization and data organization. Kinyarwanda morph analyzer in development.

25. Syntactic annotations Did initial pilot annotations with example sentences from the linguistics literature. Annotated KGMC (kin) and Lagazette and Lakroa (mlg) texts with phrase structures. – Used a fairly standard set of labels and structures – Trees created for both the source language sentences and their English translations

26. Example KGMC tree

28. Syntactic annotations Phrase structures were created before standardizing the tokenization; had to be grafted back onto correct tokens. Current trees are still pilot annotations! Need to do many things, including: – reconsider the choice of node labels – add head markers (enable easy conversion to dependency analyses) – review and incorporate feedback from others – graft some existing trees to standard tokenization

29. Overview Source, type and size of data Language consultants Kinyarwanda data Malagasy data Annotation An idea: data-driven dictionary development Accomplishments, challenges, future releases

30. Data-driven Dictionary Development Current dictionary size is moderate – 6,632 entries with 3,890 distinct Kin. words/phrases – many relatively common words not covered Idea: increase dictionary size using translators – based on data analysis of monolingual corpora – using NLP techniques to leverage process Goals – Additional bitext for direct use in MT training – Improved resource for morphological analyzers

31. Data-driven Dict. Dev. (Example) Monolingual Kinyarwanda corpus contains – ikinini (43 occ.), ibinini (96 occ.); not in dictionary Automatically predict lexical form(s), POS – ikinini (noun, plural: ibinini) Elicit English translation: pill, tablet – providing examples from corpus in context Generate dictionary entry as well as MT bitext – ikinini=pill, ikinini=tablet, ibinini=pills, ibinini=tablets

32. Overview Source, type and size of data Language consultants Kinyarwanda data Malagasy data Annotation An idea: data-driven dictionary development Accomplishments, challenges, future releases

33. Accomplishments Released monolingual, bilingual, and tree-banked data for Kinyarwanda and Malagasy – Data release v1.0 in February 2011 – Data release v2.0 in October 2011 Tools that can be shared – Tokenizer for Kinyarwanda and Malagasy – Diagnostic tools to check encoding, character sets, tokenization, tree well-formedness etc.

34. Challenges Need for more and better annotation tools to annotate faster and assure consistency – sentence segmentation, treebanking,... Need guidelines, workflow for data acquisition and annotation process Need reliable language experts for Kinyarwanda and Malagasy Need more data Wikipedia, LDS, mlg/fre

35. Data release v2.1 (target: Dec. 2011) Full sentence-level segmentation on Kinyarwanda-English text Release tokenizers, morph analyzers, diagnostic tools

36. Data release v3.0 (target: May 2012) Highest priority In-domain bilingual test sets – 500 sentences (300 newswire, 200 conversation) – Naturally occurring, source texts on both sides – Multiple translation if possible Large, modern Malagasy monolingual corpora Head markings (syntactic) Word alignment

37. Data release v3.0 (target: May 2012) Next priority Increase size of Kinyarwanda-English dictionary More Malagasy-English news bitext Typo correction Bible in Kinyarwanda (?) Malagasy-English dictionary Morphological gold standard