Overview of corpora and other language resources The Árni Magnússon Institute for Icelandic Studies Overview of corpora and other language resources The Árni Magnússon Institute for Icelandic Studies In recent years, starting about 10 years ago, we have been building various language resources for Icelandic, to be made available electronically. Much of this work has been at or in collaboration with The Árni Magnússon Institute. We maintain a website (malfong.is) which lists all available language resources, not just our resources but everything else we know about too. I will introduce the corpora and language resources that have been built in recent years and made available by The Árni Magnússon Institute for Icelandic Studies. I will briefly describe the resources and their usage licenses. I may also mention resources made available by others. Some of our data sets have quite severe restrictions of use but others can be used for almost anything in almost every way imaginable. Some licenses were specially made for the data set in question and some are CC licenses. Some of the datasets I talk about have “recently” been made available using “open” licenses, even though they were collected years before. And that is what we want to do, make our data as open as possible so as many as possible can and will use them. March 9th, 2015

Overview Text Corpora Speech Corpora Language descriptions and Dictionaries Language Tools

Text Corpora Proprietary license Proprietary license CC BY 3.0 license Others: Icelandic Parsed Historical Corpus; Wortschatz The Icelandic Frequency Dictionary was the first big project where a considerable amount of contemporary Icelandic texts were collected, analyzed and reported on. When we started training taggers, i.e. for tagging the Tagged Icelandic Corpus, this was the obvious data to use and about 70% of the original data has been available to everyone for a few years now. Not all, because when the data was collected, mostly from books (up to 50% is fiction), the copyright holders only accepted the use for the purposes of that research, so we had to get them to accept another license when we published it electronically 20 years later. This license is quite complicated but essentially only allows use for research. The tagged Icelandic Corpus is more balanced, so to say. It includes texts from printed books, fiction and non-fiction, newspapers, periodicals, websites (blogs, educational, government, etc.) speeches made at the Parliament, student essays, radio and tv scripts, e-mail lists, etc. Some of that data is freely available for everyone for free use, but most of it is not and therefore the whole corpus has a license similar to that of the IFD. A 1 million word subset of MIM (tagged icelandic corpus) was created, MIM-GOLD. It was automatically tagged (using IFD as training data) and the tags were then manually corrected. This work is in its final stages. The planned correction process was finished last year and we have estimated the tagging accuracy using this data. We found a few flaws that will not be too much of a hassle to fix to get the accuracy of the tags to the same level as the IFD. The current version is available for download now. It has the same license as MIM. Only for research. The Saga Corpus is a corpus of old Icelandic sagas, 41 texts in all. Most of the texts were published in this form between 1985 and 1991, the texts were normalized to Modern Icelandic spelling and several inflectional endings were also changed to modern icelandic form. It was tagged iteratively, first using a method developed for modern Icelandic. The tagging accuracy was measured in random samples (88%, compared to 90.4% for IFD texts). Some texts were then selected for manual correction. They were added to the IFD data and a new model created, finally reaching accuracy of 92.7%. The saga corpus is distributed with a CC BY 3.0 license, which makes it pretty close to public domain. Icelandic Parsed Historical Corpus – is a diachronic corpus with samples of written Icelandic from the 12th century to modern times. 1 million words and is mostly comparable to the corpora of historical English, developed at Upenn. Wortschatz is a text corpora of more than 500 million running words, mostly from the National Library's web scraping archives. (2005 + 2010) Developed at the Univeristy of Leipzig.

Speech Corpora Parliament Speech Corpus Hjal Corpus Málrómur 20 hours of speech CC BY 3.0 Hjal Corpus Collected in 2003 for speech recognition Málrómur Currently 44 hours of clean speech Collected in cooperation with Google for Speech Recognition CC BY 4.0 ISLEX Recordings Recordings of all the Icelandic words (48.500) in the ISLEX dictionary and roughly 700 phrases. CC BY NC ND 3.0 Others: Jensson Corpus, Thor Corpus, RÚV discussions. The Parliament Speech corpus contains 20 hours of speech (180.000 running words) in synchronized text- and sound files. Recordings from 2004-2005 with detailed transcriptions in text files. Information about speakers (age, gender) are provided as well. This data is intended to reflect natural spoken Icelandic under formal conditions. The discussion periods were chosen as they primarily consist of unprepared speeches that are unlikely to have been written in advance and read out loud. The transcriptions and processing of the material was mostly carried out by students. The Hjal Corpus was collected in 2003 for training a speech recognition system. It contains over 90.000 sound files, each containing an utterance of one or more words recorded over phone. Most of them have only a single word (but there are numbers, place names, etc.) It is hard to estimate the total duration because the sound files contain lots of silences before and after the utterances. Málrómur is the most recent speech corpus. It was collected 3 years ago by Reykjavik University and The Árni Magnússon institute in collaboration with Google. Google used this data as a basis for training their recognizer for Icelandic, but the recordings are open source, that is available to all. We recorded around 130 thousand utterances and include information on speakers (age group, gender). We are in the process of cleaning the data, that is cutting of long silences before and after the utterances and making sure the spoken text is the same as the prompts given. We have published 57 thousand utterances, in total around 44 hours of clean speech. This data was recorded using Samsung phones, but not through the phone line. ISLEX Recordings. Recorded in a studio. Read by one woman (50 years old). The three corpora mentioned at the bottom contain in total between 6 and 7 hours of speech, with multiple speakers under good recording conditions.

Language Descriptions and Dictionaries Pronunciation dictionary Over 50.000 phonetically written word forms CC BY 3.0 BÍN – Database of modern icelandic inflection 276.512 paradigms Proprietary license The Icelandic Terminology Bank 42 termbases CC BY SA 3.0 ISLEX – Icelandic – Scandinavian dictionary 50.000 words CC BY NC ND 3.0 IceWordNet The pronunciation dictionary was built as a part of the Hjal-project, discussed earlier. This is a list of phonetically transcribed words read by the participants in the Hjal-project. The Database of modern icelandic inflection is a collection of paradigms. The project started in 2002 and the work is still ongoing. It currently contains more than 276.000 paradigms. The data is available for download and can be used with certain restrictions, such as the user is not allowed to distribute the database to others. This database has proved very useful and is used in a wide variety of projects. Everything for web search, spellcheckers/grammar checkers to computer games. The Icelandic Terminology Bank is a syndicate of termbases, which have been collected by specialists in their fields. The Árni Magnússon Institute has provided the infrastructure for keeping records of the terms and publishing them online. The terminology bank contains around 60 searchable termbases, and 42 of these can be downloaded and used under an open license. The termbases vary greatly in size and details, with the smallest containing a few hundred terms but the biggest tens of thousands of terms. ISLEX is an online multilingual dictionary with modern Icelandic as a source language and Danish, Norwegian and Swedish as target languages, with Faeroish being opened this month and Finnish is also being worked on. The online dictionary access is free of charge and the data is available for researches under a non-commercial, non-derivative license. IceWordNet is based on Princeton WordNet. It consists of nearly 5000 Icelandic translations of the words from the core list from Princeton, along with the Icelandic synonyms of the words

Language Tools Older tools: CombiTagger, IceNLP, Lemmald New tools: Skrambi, Nefnir, Kvistur A few tools have been developed for working with language resources, these include CombiTagger, Lemmald and IceNLP for tagging, lemmatizing, tokenizing, parsing and recognizing named entities. These older tools can use some updating as their accuracy is not always optimal, to say the least. Tomorrow we will hear about some of the new tools being developed. These include a spellchecker and lemmatizer.