CLARIN Common Language Resources and Technology Infrastructure Daan Broeder & Dieter van Uytvanck Max-Planck Institute for Psycholinguistics TF-EMC2 Meeting, Dec

What is CLARIN The CLARIN project is a large-scale pan-European collaborative effort to coordinate and make language resources and technology available and readily useable for Language & SSH (Social Sciences & Humanities) researchers. Resources: Lexica, text corpora, multi-media/multi- modal recordings, … Technology: applications & (web-)services as parsers, tokenizers, speech recognizers & segementators, … TF-EMC2 Meeting, Dec

The problem Existence and location of resources only known to insiders Archives mostly unconnected islands Every archive has its own standards for storage and access Normally need to download first when processing resources Social sciences and humanities researchers are not language or speech technologists They are often not aware of the potential benefits of using language and speech technology Available tools are hard to use for non-specialist TF-EMC2 Meeting, Dec

CLARIN is an EU Infrastructure project with 4.2 ME funding for a 3 year preparatory phase Additional funding from national governments (at this moment at least 7 ME + 9 ME) The CLARIN consortium has now 32 partners from 26 EU countries The CLARIN community has 146 member organizations in 32 countries (mostly from NLP organizations) CLARIN is based on earlier initiatives with many participants: LangWeb, EARL, TELRI, LIRICS and more recent DAM-LR CLARIN overview TF-EMC2 Meeting, Dec

CLARIN Organization WP1Management & Coordination, Steven Krauwer, OTS U. Utrecht NL WP2Technical Infrastructure, Peter Wittenburg, MPI Nijmegen NL WP3Humanities Overview, Tamas Varadi, Hung Ac. Sc., Hun DLODARIAH Liaison, Martin Wynne, Oxford Univ. UK, WP5Language Resources & Technology Erhard Hinrichs, U. Tuebingen, Ger WP6Dissemination, Dan Crista, U. Iasi Rom WP7IPR and Business Models, Kimmo Koskiennemie, Univ. Helsinki Fin WP8Organizational Agreements, Bente Maegaard, Univ. Copenhagen, Dk TF-EMC2 Meeting, Dec

Time plan Preparatory Phase –Limited set of federated centers (10+) –Showcases, demonstrators –WP8: Investigate embedding in national funding schemes for construction phase & maintenance Construction Phase –No important European funding –Depend on national project commitments 2020? - … Maintenance Phase TF-EMC2 Meeting, Dec

CLARIN “Holy Grail” User Scenario A researcher authenticates himself with his own organization and creates a “virtual” collection of resources from different repositories. He does this on the basis of browsing a catalogue, searching through metadata, or searching in resource content. He is then able to use a workflow specification tool and process this virtual collection with possibly a mix of home grown and remote service components. Resulting data can be added to the origin repositories with proper access rights and the “virtual” collection specification can be stored for future reference. For our domain this is very ambitious and challenging, but even a partial realization is worthwhile! TF-EMC2 Meeting, Dec

DAM-LR EU project ( ) Small EU project on archive integration of 4 partners corpus/computational linguistics and endangered language documentation Resource discovery: sharing a single metadata set for searching & browsing Authentication & Authorization: single user identity, single sign-on by using Shibboleth. Referencing and citing “archived resources” using a single persistent identifier system. TF-EMC2 Meeting, Dec

AAI & Federation issues Experiences from DAM-LR wrt. to AAI: –Standard eduP. attr. set is probably sufficient, (but CCs …) –Shibboleth is nice when using web applications, but applications need access too! –Shibboleth efficient when dealing with groups e.g. staff, student, … But our domain has also to deal with individuals -> store user IDs in authorization records –DAM-LR federation of both IdPs & SPs, CLARIN aims at a much larger potential user group whose home organizations do not want to run a CLARIN specific IdP -> use the national IDFs TF-EMC2 Meeting, Dec

CLARIN Federation Infrastructure I CLARIN wants to be a LR&T “service federation” simplified and unified rules for licensing, accessing agreements with national identity federations must make sure all necessary attributes are available cater also for AA of non-web applications and web services interaction with GRID AAI national Identity Federations eJournal Service Providers LRT Service Providers Trust Agreement Trust Agreements TF-EMC2 Meeting, Dec

Applications need Authentication too IdP Shib. apache userapplication User scenario: Copying resources from different repositories to the local machine archiveA The application speaks only HTTP with basic authentication It does not understand form based authentication employed by the Shib. IdP Shib. apache archiveB The application is also not able to profit from the SSO over archives IMDI copier TF-EMC2 Meeting, Dec Possible solution: Use certificates for authentication Obtained by SLCS But can auth. handshake be mimicked by sw

CHAT EAF Shoebox MPI Archive DB/SE Search service Parsers “normalize” the structural format The scenario of searching through the content of just one archive is no problem there is just one SP that needs to check the if the user has access to the annotations. Searching through annotations Auth DB IdP TF-EMC2 Meeting, Dec

CHAT EAF Shoebox MPI Archive Archive B DB/SE CHAT Search service Search service Specialized web portal Federative search scenario Parsers “normalize” the structural format Searching through annotations Auth DB IdP Auth DB TF-EMC2 Meeting, Dec The web portal app would like to act on behalf of the user and access the search services.

Licenses & Code of conducts 1 IdP SPa SPb user SP requires CC signed and takes care of this but only for its own domain This can break the SSO if the user is required to sign the same CC several times browser TF-EMC2 Meeting, Dec CC DB CLARIN will harmonize the CCs and licenses to a limited number

Licenses & Code of conducts 2 IdP SPa SPb user browser TF-EMC2 Meeting, Dec Store the CC DB info in the user attributes at the IdP But how does it get there? Special app? Not every IdP will/can run this CC DB

Licenses & Code of conducts 3 IdP SPa SPb user browser TF-EMC2 Meeting, Dec Create special CC service. This is part of the SPF independent of the IDFs CC DB CC service

The End Thank you for your attention More info: