Presentation is loading. Please wait.

Presentation is loading. Please wait.

The FIDA & MULTEXT-East language resources Tomaž Erjavec Department of Knowledge Technologies Jožef Stefan Institute, Ljubljana

Published by Modified over 9 years ago

Similar presentations

Presentation on theme: "The FIDA & MULTEXT-East language resources Tomaž Erjavec Department of Knowledge Technologies Jožef Stefan Institute, Ljubljana"— Presentation transcript:

1 The FIDA & MULTEXT-East language resources Tomaž Erjavec Department of Knowledge Technologies Jožef Stefan Institute, Ljubljana tomaz.erjavec@ijs.sitomaz.erjavec@ijs.si, http://nl.ijs.si/et/ http://nl.ijs.si/et/ tomaz.erjavec@ijs.sihttp://nl.ijs.si/et/ Gralis 2006 Gralis 2006 Institut für Slawistik der Universität Graz 2006-05-09

2 Gralis 2006-05-09 Tomaž Erjavec Dept. of Knowledge Technologies, Jožef Stefan Institute Overview 1. Background 2. FIDA: a reference corpus of Slovene 3. MULTEXT-East: morphosyntactic resources for Central and East- European languages 4. Other language resources for Slovene

3 Gralis 2006-05-09 Tomaž Erjavec Dept. of Knowledge Technologies, Jožef Stefan Institute Language Resources LR comprise three layers of data: LR comprise three layers of data: –corpora: mono- or multilingual, reference or specialised, … /variously annotated/ –lexica: vocabularies, morphosyntactic, syntactic, semantic, (ontologies) –standards: linguistic and technical encoding LRs, esp. corpora are used for empirical language research: LRs, esp. corpora are used for empirical language research: –linguistic studies: (annotated) corpus + (sophisticated) search engine –human language technology R&D: testing and training dataset

4 Gralis 2006-05-09 Tomaž Erjavec Dept. of Knowledge Technologies, Jožef Stefan Institute Part I. The FIDA corpus Slovene reference corpus for linguistic studies Slovene reference corpus for linguistic studies

5 Gralis 2006-05-09 Tomaž Erjavec Dept. of Knowledge Technologies, Jožef Stefan Institute FIDA http://www.fida.net/ Joint project (1997-2000) of Filozofska fakulteta Vojko Gorjanc, Marko Stabej, Špela Vintar Filozofska fakulteta Vojko Gorjanc, Marko Stabej, Špela Vintar Institut Jožef Stefan Tomaž Erjavec Institut Jožef Stefan Tomaž Erjavec DZS Simon Krek DZS Simon Krek Amebis Peter Holozan, Miro Romih Amebis Peter Holozan, Miro Romih Financed by industry partnerns

6 Gralis 2006-05-09 Tomaž Erjavec Dept. of Knowledge Technologies, Jožef Stefan Institute Characteristics of FIDA monolingual monolingual synchronous synchronous written language written language reference reference –representative –balanced annotated annotated

7 Gralis 2006-05-09 Tomaž Erjavec Dept. of Knowledge Technologies, Jožef Stefan Institute Sizes Total 103,513,072 words 29,177 texts Avg. text length 3,548 words Largest texts: Leksikon DZS: 508,370 words 69 texts > 100.000 Smallest texts: 2.648 < 100 words 2 x rezgrtshdrghgth4 2 x rezgrtshdrghgth4

8 Gralis 2006-05-09 Tomaž Erjavec Dept. of Knowledge Technologies, Jožef Stefan Institute Time Composition Oldest/most recent text: 1989/2000 Oldest/most recent text: 1989/2000 Average date 1997-02 Average date 1997-02 Texts/Words with unknown date: 3.94%/8.28% Texts/Words with unknown date: 3.94%/8.28%

9 Gralis 2006-05-09 Tomaž Erjavec Dept. of Knowledge Technologies, Jožef Stefan Institute FIDA taxonomoy: publication types … Ft.P.P.O (published) 95.72% Ft.P.P.O.K (books) 22.71% Ft.P.P.O.P (periodicals) 70.50% Ft.P.P.O.P.C (newspaper) 46.59% Ft.P.P.O.P.C.D (daily) 32.67% Ft.P.P.O.P.C.T (weekly) 66.18% Ft.P.P.O.P.C.V (multi-weekly)17.74% …

10 Gralis 2006-05-09 Tomaž Erjavec Dept. of Knowledge Technologies, Jožef Stefan Institute FIDA taxonomoy: text types Ft.Z (text type) 99.47% Ft.Z.N (non-ficiton) 93.57% Ft.Z.N.N (non-professional)75.14% Ft.Z.N.S (professional) 18.37% Ft.Z.N.S.H (hum. & soc. sci.)10.57% Ft.Z.N.S.N (nat. & tech. sci.) 6.04% Ft.Z.U (fiction) 5.90% Ft.Z.U.D (drama) 0.10% Ft.Z.U.P (poetry) 0.17% Ft.Z.U.R (prose) 5.12%

11 Gralis 2006-05-09 Tomaž Erjavec Dept. of Knowledge Technologies, Jožef Stefan Institute Markup of FIDA corpus elements annotated with meta- data (bibliographic, taxonomy) corpus elements annotated with meta- data (bibliographic, taxonomy) text linguistically annotated text linguistically annotated encoded according to international standards and recommendations encoded according to international standards and recommendations –technical: SGML, TEI P3 –linguistic: MULTEXT-East (MULTEXT, EAGLES)

12 Gralis 2006-05-09 Tomaž Erjavec Dept. of Knowledge Technologies, Jožef Stefan Institute Linguistic annotation

13 Gralis 2006-05-09 Tomaž Erjavec Dept. of Knowledge Technologies, Jožef Stefan Institute Accesibility Exploitation by partners: –DZS: new dictionaries –Amebis: development of HLT –Arts faculty: teaching –IJS: research on HLT Availability to the public: –access via concordance engine by Amebis –free access, but displays only few hits –possibility of academic licences FIDA (web site) no longer maintained!

14 Gralis 2006-05-09 Tomaž Erjavec Dept. of Knowledge Technologies, Jožef Stefan Institute FIDA+ http://www.fidaplus.net/ http://www.fidaplus.net/ FIDA Plus project: FIDA Plus project: –Filozofska fakulteta, Fakulteta za družbene vede, Institut Jožef Stefan –DZS, Amebis Financed by the ministry + ind. partners Financed by the ministry + ind. partners Extend the corpus with Extend the corpus with –Web materials –spoken component Better linguistic markup Better linguistic markup Free concordances: up to 100 lines Free concordances: up to 100 lines Also possibility of licences Also possibility of licences

15 Gralis 2006-05-09 Tomaž Erjavec Dept. of Knowledge Technologies, Jožef Stefan Institute Concordancer

16 Gralis 2006-05-09 Tomaž Erjavec Dept. of Knowledge Technologies, Jožef Stefan Institute Output

17 Gralis 2006-05-09 Tomaž Erjavec Dept. of Knowledge Technologies, Jožef Stefan Institute Extended searches

18 Gralis 2006-05-09 Tomaž Erjavec Dept. of Knowledge Technologies, Jožef Stefan Institute Corpus “Nova Beseda” http://bos.zrc-sazu.si/ being developed at Institute for Slovene language, ZRC SAZU (Primož Jakopin) being developed at Institute for Slovene language, ZRC SAZU (Primož Jakopin) Web concordancer with no hit limit Web concordancer with no hit limit now larger than FIDA now larger than FIDA but much less varied: fiction, Delo, DZ but much less varied: fiction, Delo, DZ not linguistically annotated not linguistically annotated

19 Gralis 2006-05-09 Tomaž Erjavec Dept. of Knowledge Technologies, Jožef Stefan Institute Part II. MULTEXT-East multilingual morphosyntactic resources for HLT development multilingual morphosyntactic resources for HLT development

20 Gralis 2006-05-09 Tomaž Erjavec Dept. of Knowledge Technologies, Jožef Stefan Institute MULTEXT-East resources MULTEXT-East: Copernicus Joint Project COP 106 (1995-1997) Multilingual Texts and Corpora for Eastern and Central European Languages MULTEXT-East: Copernicus Joint Project COP 106 (1995-1997) Multilingual Texts and Corpora for Eastern and Central European Languages MULTEXT-East Based on the results of EU MULTEXT (~West) Based on the results of EU MULTEXT (~West) To produce a harmonised BLARK for six languages: To produce a harmonised BLARK for six languages:BLARK –corpus encoding standardisation (TEI / CES) –multilingual parallel, comparable, speech corpora –morphosyntactic specifications (EAGLES / MULTEXT) –(inflectional) lexicon –annotated corpus –language processing tools

21 Gralis 2006-05-09 Tomaž Erjavec Dept. of Knowledge Technologies, Jožef Stefan Institute History of MULTEXT-East resources First release 1998 on TELRI CD-ROM Vol II: already extended with new languages First release 1998 on TELRI CD-ROM Vol II: already extended with new languages Resources since 1998 available on the Web: http://nl.ijs.si/ME/ Resources since 1998 available on the Web: http://nl.ijs.si/ME/http://nl.ijs.si/ME/ Second release 2002 in scope of EU CONCEDE: re-encoding in XML/TEI, harmonisation Second release 2002 in scope of EU CONCEDE: re-encoding in XML/TEI, harmonisation Third release 2004: merge of first two releases, further languages Third release 2004: merge of first two releases, further languages Work (indirectly) supported by: TELRI, CONCEDE, NSF grant, bi-lateral projects Work (indirectly) supported by: TELRI, CONCEDE, NSF grant, bi-lateral projects

22 Gralis 2006-05-09 Tomaž Erjavec Dept. of Knowledge Technologies, Jožef Stefan Institute The Languages of MULTEXT-East Germanic: English Germanic: EnglishEnglish Romance: Romanian Romance: RomanianRomanian Baltic: Baltic: –Latvian Latvian –Lithuanian Lithuanian Finno-Ugric: Finno-Ugric: –Estonian Estonian –Hungarian Hungarian Slavic: Russian (East Slavic) Russian (East Slavic) Russian Czech (West Slavic) Czech (West Slavic) Czech Slovene (South West Slavic) Slovene (South West Slavic) Slovene Resian (Slovene dialect) Resian (Slovene dialect) Resian Croatian (South West Slavic) Croatian (South West Slavic) Croatian Serbian (South West Slavic) Serbian (South West Slavic) Serbian Bulgarian (South East Slavic) Bulgarian (South East Slavic) Bulgarian In progress: Macedonian Macedonian Persian Persian

23 Gralis 2006-05-09 Tomaž Erjavec Dept. of Knowledge Technologies, Jožef Stefan Institute Version 3 Available on http://nl.ijs.si/ME/V3/ Available on http://nl.ijs.si/ME/V3/http://nl.ijs.si/ME/V3/ Some parts completely free, others free for research  Web licence Some parts completely free, others free for research  Web licence Web pages gives: Web pages gives: –extensive documentation –bibliography list –web licence form –resource download

24 Gralis 2006-05-09 Tomaž Erjavec Dept. of Knowledge Technologies, Jožef Stefan Institute The MULTEXT morphosyntactic trinity 1. MULTEXT-East morphosyntactic specifications 2. MULTEXT-East morphosyntactic lexica 3. MULTEXT-East morphosyntactically annotated "1984" corpus

25 Gralis 2006-05-09 Tomaž Erjavec Dept. of Knowledge Technologies, Jožef Stefan Institute 1. Morphosyntactic specifications Based on EAGLES / MULTEXT Based on EAGLES / MULTEXT Define PoS, their attributes and values Define PoS, their attributes and values The specs are a document containing: The specs are a document containing: –introduction –common tables –language particular sections Written in LaTeX  PDF & HTML Written in LaTeX  PDF & HTMLPDFHTMLPDFHTML Derived XML/TEI encoding as feature structures Derived XML/TEI encoding as feature structuresXML/TEI

26 Gralis 2006-05-09 Tomaž Erjavec Dept. of Knowledge Technologies, Jožef Stefan Institute Example common table

27 Gralis 2006-05-09 Tomaž Erjavec Dept. of Knowledge Technologies, Jožef Stefan Institute Example language specific section table (shows only categories actually used) table (shows only categories actually used) notes notes combinations combinations lexicon lexicon for Slovene (FIDA): localisation of category names for Slovene (FIDA): localisation of category names

28 Gralis 2006-05-09 Tomaž Erjavec Dept. of Knowledge Technologies, Jožef Stefan Institute Morphosyntactic Complexity

29 Gralis 2006-05-09 Tomaž Erjavec Dept. of Knowledge Technologies, Jožef Stefan Institute 2. The lexica Medium size morphosyntactic lexica Medium size morphosyntactic lexica Languages: English, Romanian, Slovene, Czech, Bulgarian, Estonian, Hungarian, Serbian. Languages: English, Romanian, Slovene, Czech, Bulgarian, Estonian, Hungarian, Serbian. ~ all word-forms of cca 15.000 lemmas ~ all word-forms of cca 15.000 lemmas Lexical entry is composed of three fields: Lexical entry is composed of three fields: –the word-form: the inflected form of the word –the lemma: the base-form of the word –the morphosyntactic description (MSD)

30 Gralis 2006-05-09 Tomaž Erjavec Dept. of Knowledge Technologies, Jožef Stefan Institute Example: Slovene lexicon abeced abeceda Ncfdg abeced abeceda Ncfpg abeceda = Ncfsn abecedah abeceda Ncfdl abecedah abeceda Ncfpl abecedam abeceda Ncfpd abecedama abeceda Ncfdd abecedama abeceda Ncfdi abecedami abeceda Ncfpi abecede abeceda Ncfpa abecede abeceda Ncfpn abecede abeceda Ncfsg abecedi abeceda Ncfda abecedi abeceda Ncfdn …

31 Gralis 2006-05-09 Tomaž Erjavec Dept. of Knowledge Technologies, Jožef Stefan Institute Lexicon sizes

32 Gralis 2006-05-09 Tomaž Erjavec Dept. of Knowledge Technologies, Jožef Stefan Institute 3. The “1984” corpus Languages: En, Ro, Sl, Cs, Et, Hu, Sr, (Bg, Ru, (Mk, Hr, Tr,…)) Languages: En, Ro, Sl, Cs, Et, Hu, Sr, (Bg, Ru, (Mk, Hr, Tr,…)) Structuraly annotated Structuraly annotated Sentence aligned with English Sentence aligned with English Words annotated with lemma and MSD Words annotated with lemma and MSD Encoded in TEI P4 (XML) Encoded in TEI P4 (XML)

33 Gralis 2006-05-09 Tomaž Erjavec Dept. of Knowledge Technologies, Jožef Stefan Institute Example linguistic encoding Bil Bil je je jasen jasen,, mrzel mrzel aprilski aprilski dan dan in in ure ure so so bile bile trinajst trinajst.. … Sentence alignment & Context disambiguated lemmas and MSDs

34 Gralis 2006-05-09 Tomaž Erjavec Dept. of Knowledge Technologies, Jožef Stefan Institute Quantifying the corpus

35 Gralis 2006-05-09 Tomaž Erjavec Dept. of Knowledge Technologies, Jožef Stefan Institute Utility of MULTEXT-East LRs Specifications became, for some, the “national” standard Specifications became, for some, the “national” standard Training/testing dataset for HLT development: PoS taggers, lemmatizers, lexicon extractors, ILP Training/testing dataset for HLT development: PoS taggers, lemmatizers, lexicon extractors, ILP A base dataset for further annotation and experiments: A base dataset for further annotation and experiments: –Word-sense disambiguation –WordNet development and evaluation –Syntactic parser induction Teaching aid in HLT courses Teaching aid in HLT courses ~ 100 registered users ~ 100 registered users As a BLARK “best practice” for new languages: Resian, Croatian, Macedonian, Persian As a BLARK “best practice” for new languages: Resian, Croatian, Macedonian, Persian

36 Gralis 2006-05-09 Tomaž Erjavec Dept. of Knowledge Technologies, Jožef Stefan Institute LRs @ JSI http://nl.ijs.si/nl.html#Resource Also ours: VAYNA, GORE, sloWNet Contributors to: FIDA, DSI, FDV, JRC-ACQUIS

37 Gralis 2006-05-09 Tomaž Erjavec Dept. of Knowledge Technologies, Jožef Stefan Institute Overview of Slovene LRs and services @ Slovenian Language Technologies Society http://nl.ijs.si/sdjt/

38 Gralis 2006-05-09 Tomaž Erjavec Dept. of Knowledge Technologies, Jožef Stefan Institute Thank you!

Download ppt "The FIDA & MULTEXT-East language resources Tomaž Erjavec Department of Knowledge Technologies Jožef Stefan Institute, Ljubljana"

Similar presentations

Using OLIF, The Open Lexicon Interchange Format Susan McCormick OLIF2 Consortium October 1, 2004.

Using OLIF, The Open Lexicon Interchange Format Susan McCormick OLIF2 Consortium October 1, 2004.
Part Two: Using Xaira to explore corpora Richard Xiao

Part Two: Using Xaira to explore corpora Richard Xiao
Disambiguation of homographic adjective and adverb forms in Croatian texts Danijela Merkler*, Daša Berović*, Željko Agić** * Department of Linguistics.

Disambiguation of homographic adjective and adverb forms in Croatian texts Danijela Merkler*, Daša Berović*, Željko Agić** * Department of Linguistics.
MULTEXT-East Version 4: multilingual morphosyntactic specifications for lots of languages Tomaž Erjavec Department of Knowledge Technologies.

MULTEXT-East Version 4: multilingual morphosyntactic specifications for lots of languages Tomaž Erjavec Department of Knowledge Technologies.
Introduction to BLaRKs Helmer Strik Dept. of Linguistics Centre for Language and Speech Technology (CLST) Radboud University Nijmegen, the Netherlands.

Introduction to BLaRKs Helmer Strik Dept. of Linguistics Centre for Language and Speech Technology (CLST) Radboud University Nijmegen, the Netherlands.
MULTEXT-East Version 4: Multilingual Morphosyntactic Specifications, Lexicons and Corpora Tomaž Erjavec Department of Knowledge Technologies Jožef.

MULTEXT-East Version 4: Multilingual Morphosyntactic Specifications, Lexicons and Corpora Tomaž Erjavec Department of Knowledge Technologies Jožef.
MIG-KOMM-EU Multilingual intercultural business communication in Europe University of Bucharest Faculty of Foreign Languages and Literatures German Studies.

MIG-KOMM-EU Multilingual intercultural business communication in Europe University of Bucharest Faculty of Foreign Languages and Literatures German Studies.
Annotating language data Tomaž Erjavec Institut für Informationsverarbeitung Geisteswissenschaftliche Fakultät Karl-Franzens-Universität Graz Tomaž Erjavec.

Annotating language data Tomaž Erjavec Institut für Informationsverarbeitung Geisteswissenschaftliche Fakultät Karl-Franzens-Universität Graz Tomaž Erjavec.
Jing-Shin Chang National Chi Nan University, IJCNLP-2013, Nagoya 2013/10/15 ACLCLP – Activities (2011-2013) & Text Corpora.

Jing-Shin Chang National Chi Nan University, IJCNLP-2013, Nagoya 2013/10/15 ACLCLP – Activities ( ) & Text Corpora.
Totale Multilingual Tokenisation, Tagging and Lemmatisation Tomaž Erjavec Dept. of Knowledge Technologies, Jožef Stefan Institute Ljubljana, Slovenia JRC.

Totale Multilingual Tokenisation, Tagging and Lemmatisation Tomaž Erjavec Dept. of Knowledge Technologies, Jožef Stefan Institute Ljubljana, Slovenia JRC.
WG3: Innovative e-dictionaries Simon Krek „Jožef Stefan“ Institute, Ljubljana, Slovenia Carole Tiberius Institute of Dutch Lexicology, Leiden, the Netherlands.

WG3: Innovative e-dictionaries Simon Krek „Jožef Stefan“ Institute, Ljubljana, Slovenia Carole Tiberius Institute of Dutch Lexicology, Leiden, the Netherlands.
What is a national corpus. Primary objective of a national corpus is to provide linguists with a tool to investigate a language in the diversity of types.

What is a national corpus. Primary objective of a national corpus is to provide linguists with a tool to investigate a language in the diversity of types.
The University of Wisconsin-Madison Universal Morphological Analysis using Structured Nearest Neighbor Prediction Young-Bum Kim, João V. Graça, and Benjamin.

The University of Wisconsin-Madison Universal Morphological Analysis using Structured Nearest Neighbor Prediction Young-Bum Kim, João V. Graça, and Benjamin.
Multilingual eLearning in LANGuage Engineering. Project Overview  Project span: Oct 2004 – Oct 2007  Kick-off meeting Oct 9-10 2004  Project goals:

Multilingual eLearning in LANGuage Engineering. Project Overview  Project span: Oct 2004 – Oct 2007  Kick-off meeting Oct  Project goals:
Prim(j)ena MULTEXT-East standarda i normi TEI u izradi paralelnih korpusa Applikation des MULTEXT-East und der TEI-Normen bei der Erstellung von Parallelkorpora.

Prim(j)ena MULTEXT-East standarda i normi TEI u izradi paralelnih korpusa Applikation des MULTEXT-East und der TEI-Normen bei der Erstellung von Parallelkorpora.
The MULTEXT-East multilingual language resources Tomaž Erjavec Department of Knowledge Technologies Jožef Stefan Institute, Ljubljana

The MULTEXT-East multilingual language resources Tomaž Erjavec Department of Knowledge Technologies Jožef Stefan Institute, Ljubljana
Eleni Galiotou, Dept. of Informatics

Eleni Galiotou, Dept. of Informatics
New Slovene corpora within the »Communication in Slovene« project Nataša Logar BergincSimon Krek University of LjubljanaAmebis, Kamnik Faculty of Social.

New Slovene corpora within the »Communication in Slovene« project Nataša Logar BergincSimon Krek University of LjubljanaAmebis, Kamnik Faculty of Social.
Michal Křen Institute of the Czech National Corpus Charles University, Prague SLAVICORP Warszawa, 22 November 2010 Accessing the.

Michal Křen Institute of the Czech National Corpus Charles University, Prague SLAVICORP Warszawa, 22 November 2010 Accessing the.
Resources Primary resources – Lexicons, structured vocabularies – Grammars (in widest sense) – Corpora – Treebanks Secondary resources – Designed for a.

Resources Primary resources – Lexicons, structured vocabularies – Grammars (in widest sense) – Corpora – Treebanks Secondary resources – Designed for a.

Similar presentations

Ads by Google