CLARIN-PL CLARIN-PL – Research User-driven Language Technology Infrastructure Maciej Piasecki Wrocław University of Technology G4.19 Research Group

Basic Notions  Language Technology (LT)  language resources and tools  robust in terms of quality and coverage  multipurpose  component based  Language Technology Infrastructure  a software framework (architecture or platform)  for combining language tools with language resources into processing chains (or pipelines)  the defined processing chains are next applied to language data sources  interoperability, also with the external systems Humanistyka Cyfrowa Warszawa CLARIN-PL

LT in Humanities and Social Sciences: Barriers  Physical – language tools and resources are not accessible in Internet  Informational – descriptions are not available or there is no means for searching  Technological – lack of commonly accepted standards for LT, lack of a common platform, varieties of technological solutions, insufficient users’ computers  Related to knowledge – the use of LT requires programming skills or knowledge from the area of natural language engineering  Legal – licences for language resources and tools (LRTs) limit their applications Humanistyka Cyfrowa Warszawa CLARIN-PL

CLARIN Support for Humanities & Social Sciences  CLARIN is ERIC type consortium of  11 countries (Austria, Bulgaria, Czech Republic, Denmark, Estonia, Germany, Lithuania, The Netherlands, Poland, Portugal, Sweden) and The Dutch Language Union  1 observer: Norway  Focus area:  Supporting research in Humanities and Social Sciences  Users: researchers, PhD students, students and scientific institutions  CLARIN Mission  To significantly lower the barriers for the use of Language Technology in Humanities & Social Sciences (H&SS)  To facilitate or enable research methods based on automated analysis of text and speech resources Humanistyka Cyfrowa Warszawa CLARIN-PL

CLARIN Offer  Integration of different LT components into one interoperable system  Common, flexible meta-data standard (CMDI)  Central searching for resources (Virtual Language Observatory)  One sign on and one login into the distributed infrastructure  Decreased Physical and Informational Barriers  Common standards: promoting, co-ordinating, harmonising  Web Services for Language Tools and Resources  Decreased Technological Barrier  Installation-free, access via Web Applications  Decreased Knowledge Barrier  Common licences and promotion of the open access  Decreased Legal Barrier Humanistyka Cyfrowa Warszawa CLARIN-PL

CLARIN: Portal Humanistyka Cyfrowa Warszawa CLARIN-PL

CLARIN: Virtual Language Observatory Humanistyka Cyfrowa Warszawa CLARIN-PL

CLARIN: Federated Content Search – Searching Corpora Humanistyka Cyfrowa Warszawa CLARIN-PL

LTI Development Paradigms  Bottom-up  a collected offer approach  based on linking together the already existing Language Resources and Tools  focused on accessibility, technical interoperability and processing chains  Top-down  following on user-centred design paradigm  research applications for H&SS are a starting point  Bi-directional  linking of Language Resources and Tools  combined with the development of research applications Humanistyka Cyfrowa Warszawa CLARIN-PL

Bi-directional LTI Development  Idea  development of the necessary elements  a distributed network infrastructure  basic LT processing chain  combined with user-centred approach to the development of research applications  Top-down part  close co-operation with key users from the H&SS domain  a metaphor of the Agile-like light weight software designing method with emphasis to prototyping  amendments to the shape of the technical basis: LRTs, standards,  inspirations, identification of the further user needs, next iterations Humanistyka Cyfrowa Warszawa CLARIN-PL

CLARIN-PL: the Consortium  Polish scientific consortium  Wrocław University of Technology, G4.19 Research Group  Institute of Computer Science, Polish Academy of Science  Polish-Japanese Institute of Information Technology, Chair of Multimedia  University of Łódź, PELCRA group at Chair of English Language and Applied Linguistics  Institute of Slavic Studies, Polish Academy of Science  Wrocław University  Goal: implementation of the Polish part of the CLARIN ERIC LTI  Follows the bi-directional approach to LTI development Humanistyka Cyfrowa Warszawa CLARIN-PL

CLARIN-PL: Mission  Starting point  Several publicly available language resources and tools for Polish,  But still many were lacking  Deeper technological barrier: restricted applications  CLARIN-PL Pillars:  CLARIN-PL Language Technology Centre  the Polish node of the CLARIN distributed infrastructure  Complete set of the basic Language Resources & Tools for Polish  Research applications for H&SS  first set for key users and selected H&SS sub-domains. Humanistyka Cyfrowa Warszawa CLARIN-PL

CLARIN-PL Language Technology Centre  Location in Wrocław University of Technology  based on modified D-Space system from Lindat (Czech CLARIN)  One sign-on, one login (a member of the Pioneer.id Federation)  Advanced repository system for language resources  Persistent Identifiers for resources and tools  Rich CMDI meta-data – CLARIN wide visibility in the central search  Interface for Federated Content Search  depositing service for researchers from H&SS  application for the Data Seal of Approval  Adherence to all CLARIN specifications about standards and protocols  Web Services for LRTs:  the basic processing chain of Polish  Prototype system for flexible composition of the natural language processing chains  support for developers SOAP & REST interfaces  Web Applications for LRTs  Knowledge Sharing: expertise and support for the users Humanistyka Cyfrowa Warszawa CLARIN-PL

CLARIN-PL: Language Resources 1.Polish Morphological Dictionary 2.Polish Speech Corpora 3.Annotated Polish Corpora 4.Bilingual Corpora 5.Polish Historical Corpus 6.Semantic lexicon  Wordnet for Polish  formal description of lexical meanings 7.Dictionary of Multiword Expressions 8.Bilingual semantic lexicon 9.Lexicon of Proper Names 10.Syntactic-semantic Valency Dictionary 11.Robust syntactic-semantic grammar Humanistyka Cyfrowa Warszawa CLARIN-PL

CLARIN-PL: Language Resources 1.Polish Morphological Dictionary 2.Polish Speech Corpora 3.Annotated Polish Corpora 4.Bilingual Corpora 5.Polish Historical Corpus 6.Semantic lexicon  plWordNet 3.0  formal description of lexical meanings 7.Dictionary of Multiword Expressions 8.Bilingual semantic lexicon 9.Lexicon of Proper Names 10.Syntactic-semantic Valency Dictionary: 11.Robust syntactic-semantic grammar Humanistyka Cyfrowa Warszawa CLARIN-PL

CLARIN-PL: Language Resources  Starting point – a set of large resources  a huge National Corpus of Polish (1 billion tokens)  plWordNet 2.1 – a very large wordnet for Polish  Korpus Politechniki Wrocławskiej – an open Polish corpus with rich annotation  Expanded resources  plWordNet 3.0 – a huge semantic lexicon of Polish  a comprehensive description of the Polish lexico-semantic system (~ lemmas, ~ senses)  fully mapped to English Princeton WordNet  described formally by mapping to an ontology  Dictionary of multiword expressions described syntactically  NELexicon 2.0 – a huge lexicon of Polish Proper Names (2.5 mln) Humanistyka Cyfrowa Warszawa CLARIN-PL

CLARIN-PL: Language Resources for Polish  Expanded resources  Conversational corpus (following PELCRA and NKJP)  A large semantic valency lexicon for Polish predicative lexical units  Newly built resources  Transcribed training-testing Polish speech corpus  Bi-lingual corpora:  Polish-English, Polish-Bulgarian-Russian, Polish-Lithuanian  Polish historical corpus (for the years )  Corpora annotated for: meta-data, anaphora, time expressions, spatial expressions, semantic relations and situations Humanistyka Cyfrowa Warszawa CLARIN-PL

plWordNet 2.2 in CLARIN-PL Humanistyka Cyfrowa Warszawa CLARIN-PL

plWordNet 2.2 in CLARIN-PL Humanistyka Cyfrowa Warszawa CLARIN-PL

CLARIN-PL: Language Tools for Polish  Systems for searching corpora, especially Polish corpora  Spokes for conversational and bilingual corpora  Poliqarp 2.0 for richly annotated  Historical corpora [New]  Text mining (information extraction)  Recognition and classification of Proper Names  Recognition of anaphoric links  Recognition and classification of time expressions and spatial expressions [New]  Situation recognition [New]  Extraction of multiword expressions (collocations)  A generic set of morpho-syntactic tools for Polish that can be adapted to a domain specified by the user [New] Humanistyka Cyfrowa Warszawa CLARIN-PL

CLARIN-PL: Language Tools for Polish  Word Sense Disambiguation based on plWordNet  Shallow semantic parser [New]  Deep syntactic-semantic parser [New]  Tools for the extraction of the semantic-pragmatic information from documents and collections of documents, e.g.  keywords [New],  semantic relations between text fragments  and text summaries Humanistyka Cyfrowa Warszawa CLARIN-PL

Basic Language Tools for Polish 1.Segmentation into tokens and sentences 2.Morphological analysis 3.Morphological guessing of unknown words (both without context and context sensitive) 4.Morpho-syntactic tagging 5.Word Sense Disambiguation 6.Chunker and shallow syntactic parser 7.Named Entity Recognition and disambiguation 8.Co-reference and anaphora resolution 9.Temporal expression recognition 10.Semantic relation recognition 11.Event recognition 12.Shallow semantic parser 13.Deep syntactic parser with disambiguated output: dependency and constituent 14.Deep semantic parser Humanistyka Cyfrowa Warszawa CLARIN-PL

Basic Language Tools for Polish 1.Segmentation into tokens and sentences 2.Morphological analysis 3.Morphological guessing of unknown words (both without context and context sensitive) 4.Morpho-syntactic tagging 5.Word Sense Disambiguation 6.Chunker and shallow syntactic parser 7.Named Entity Recognition and disambiguation 8.Co-reference and anaphora resolution 9.Temporal expression recognition 10.Semantic relation recognition 11.Event recognition 12.Shallow semantic parser 13.Deep syntactic parser with disambiguated output: dependency and constituent 14.Deep semantic parser Humanistyka Cyfrowa Warszawa CLARIN-PL

Basic Language Tools for Polish 1.Segmentation into tokens and sentences 2.Morphological analysis 3.Morphological guessing of unknown words (both without context and context sensitive) 4.Morpho-syntactic tagging 5.Word Sense Disambiguation 6.Chunker and shallow syntactic parser 7.Named Entity Recognition and disambiguation 8.Co-reference and anaphora resolution 9.Temporal expression recognition 10.Semantic relation recognition 11.Event recognition 12.Shallow semantic parser 13.Deep syntactic parser with disambiguated output: dependency and constituent 14.Deep semantic parser Humanistyka Cyfrowa Warszawa CLARIN-PL

CLARIN-PL: Processing Chain for Polish Humanistyka Cyfrowa Warszawa CLARIN-PL

CLARIN-PL: Recognition and classification of Proper Names Humanistyka Cyfrowa Warszawa CLARIN-PL

Bi-directional - Top-down Part: First Applications  Approaching users  already active, interested, working on large textual and speech resources, …  covering a maximal variety of research areas, e.g. linguistics, literary studies, psychology, political studies and sociology  matching the available language tools for Polish  the first set of several prototype application illustrating possibilities and facilitating identification of the needs  First applications  Spokes – searching corpora of conversational data  A system for collecting Polish text corpora from the Web  A open textometric and stylometric system focused on Polish  Semantic text classification for sociology  Literary Map Humanistyka Cyfrowa Warszawa CLARIN-PL

Spokes (University of Łódź) Humanistyka Cyfrowa Warszawa CLARIN-PL

System for Collecting Polish Text Corpora from the Web  Requests from the users revealed gaps in the available technology  existing corpus building systems were too sensitive to text encoding errors found in the web  not designed for informal corpora like blogs  A system for collecting Polish text corpora from the Web had to be constructed:  based on tools from the Masaryk University in Brno  to detect texts including larger number of errors (by morphological analysis)  supports semi-automated extraction of texts from blogs, posts on forums, etc.  integrated with tools for processing Humanistyka Cyfrowa Warszawa CLARIN-PL

Open Textometric and Stylometric System  System designed for characteristic features of Polish  like rich inflection, weakly constrained word order  Based on several existing components including Stylo (Eder & Rybicki)  Enabling the use of features defined on any level of the linguistic structure:  from the level of word forms  up to the level of the semantic-pragmatic structures.  Available as Web Application and a Web Service  Stylometric techniques appear to be applicable in many tasks of H&SS  sociology (characteristic features that are for different subgroups), political studies (similarity and differences between political parties), literary studies … Humanistyka Cyfrowa Warszawa CLARIN-PL

Semantic Text Classification for Sociology  Users: Collegium Civitas, Warsaw  Goal  Support for large scale analysis of the source materials  Automatically annotate documents and text fragments with pre-defined semantic categories  Definition of categories by examples  Automated semantic grouping of documents and text fragments  Support for  Corpus building  Manual annotation of the learning sub-corpus  Automated annotation process  Statistical analysis of the results Humanistyka Cyfrowa Warszawa CLARIN-PL

GeTClasS – Generalised Text Classification for Sociology Humanistyka Cyfrowa Warszawa CLARIN-PL

Literary Map  Users: Digital Humanities Centre of The Institute of Literary Research (Polish Academy of Sciences)  Goal  Support for using maps in the literary criticism  Tool for the identification of all geographical names in the literary text (or a corpus) and mapping them onto a geographical map  Tasks 1.Identification and semantic classification of the referring language expressions 2.Disambiguation of the referents 3.Mapping the referents onto a map (geo-location) 4.Recognition of the semantic relations and statistical analysis Humanistyka Cyfrowa Warszawa CLARIN-PL

Literary Map Humanistyka Cyfrowa Warszawa CLARIN-PL

Conclusions  Application of LT to the research in Humanities & Social Sciences seem to be much more challenging than in commercial systems!  LT for Polish achieved a stage in which valuable support can be provided for research applications  Bi-directional approach combines  development of the basic, universal set of language tools and resources  with inspirations from the research applications Humanistyka Cyfrowa Warszawa CLARIN-PL

Thank you very much for your attention! Supported by the Polish Ministry of Science and Higher Education [CLARIN-PL]

Bi-directional: bottom-up part  LRTs and LRT chains can be useful …  if the required tools and resources exist,  and, they are robust!  What is the minimal set of LRTs?  What kind of LRTs can be called robust?  automated applications in H&SS seem to require high quality of language tools and mostly large coverage of resource  BLARK – The Basic Language Resource Kit  “the minimal set of language resources that is necessary to do any precompetitive research and education at all” (Krauwer, 2003) and also basic processing chains  possible reference point to compare LRTs for different languages PALC 2014 Łódź CLARIN-PL