EUDAT operations perspective and initial achievements Johannes Reetz EGI Technical Forum 2012 EGI/EUDAT/PRACE operations workshop Prague, 18 Sep 2012 eudat.eu.

Slides:



Advertisements
Similar presentations
EUDAT Towards a European Collaborative Data Infrastructure Alison Kennedy and Rob Baxter Jan 2012.
Advertisements

EUDAT Towards a European Collaborative Data Infrastructure Damien Lecarpentier – CSC, IT Center for Science, Finland ISC’11, Hamburg, 20 June 2011.
EUDAT Towards a pan-European Collaborative Data Infrastructure Ari Lukkarinen CSC-IT Center for Science, Finland APA Conference, November 6th, 2012.
AAF Middleware update February Presented by Terry Smith Technical Manager and Heath Marks Manager.
EGI-Engage EGI-Engage Engaging the EGI Community towards an Open Science Commons Project Overview 9/14/2015 EGI-Engage: a project.
Data Archiving and Networked Services DANS is an institute of KNAW en NWO Trusted Digital Archives and the Data Seal of Approval Peter Doorn Data Archiving.
Climate Sciences: Use Case and Vision Summary Philip Kershaw CEDA, RAL Space, STFC.
EGI: SA1 Operations John Gordon EGEE09 Barcelona September 2009.
Australian Access Federation and other Middleware Initiatives Presented at TF-EMC2, Prague 4 Sep 2007 Patty McMillan, The University of Queensland.
Federated Identity Management for HEP David Kelsey WLCG GDB 9 May 2012.
CLARIN Infrastructure Vision (and some real needs) Daan Broeder CLARIN EU/NL Max-Planck Institute for Psycholinguistics.
Towards a European network for digital preservation Ideas for a proposal Mariella Guercio, University of Urbino.
Customer Interface for wuw.com 1.Context. Customer Interface for wuw.com 2. Content Our web-site can be classified as an service-dominant website. 3.
EGI-InSPIRE RI EGI-InSPIRE EGI-InSPIRE RI EGI Towards H2020 Tiziana Ferrari/EGI.eu WLCG Collaboration Workshop.
RI EGI-InSPIRE RI EGI Future activities Peter Solagna – EGI.eu.
State of e-Authentication in Higher Education August 20, 2004.
Exploring ‘Workspaces’ Tom Visser, SARA compute and networking services, Amsterdam Garching Workshop 21 st September 2010.
Summary of AAAA Information David Kelsey Infrastructure Policy Group, Singapore, 15 Sep 2008.
EGI-InSPIRE RI EGI-InSPIRE EGI-InSPIRE RI (Present and) Future of the EGI Services for WLCG Peter Solagna – EGI.eu.
EGEE-III-INFSO-RI Enabling Grids for E-sciencE EGEE and gLite are registered trademarks EGEE-III All Activity Meeting Brussels,
Security Policy Update David Kelsey UK HEP Sysman, RAL 1 Jul 2011.
TCS-ICS interactions Kuvvet Atakan 1 and the WP6 and WP7 Teams 1 University of Bergen / Department of Earth Science.
EGEE-III INFSO-RI Enabling Grids for E-sciencE EGEE and gLite are registered trademarks Robin McConnell NA3 Activity Manager 28.
Security Policy: From EGEE to EGI David Kelsey (STFC-RAL) 21 Sep 2009 EGEE’09, Barcelona david.kelsey at stfc.ac.uk.
University of Washington Collaboration: Identity and Access Management Lori Stevens University of Washington October 2007.
Replicate Research Data Safely eudat.eu/b2safe B2SAFE How to replicate your data using EUDAT’s B2SAFE Version 3 November 2015 This work is.
EUDAT receives funding from the European Union's Horizon 2020 programme - DG CONNECT e-Infrastructures. Contract No B2ACCESS LSDMA.
EGI-InSPIRE RI EGI-InSPIRE EGI-InSPIRE RI Evolution of AAI for e- infrastructures Peter Solagna Senior Operations Manager.
ATOS in Period 2 (WP4 leader) Technical Review Period 2 Michel van Adrichem, Mick Symonds, Josep Martrat This document produced by Members of the Helix.
EMI is partially funded by the European Commission under Grant Agreement RI NA2 : Outreach and Training Emidio Giorgio, INFN NA2 Leader 2 nd EMI.
CLARIN EUDAT2020 uptake plan Dieter Van Uytvanck CLARIN ERIC EUDAT User Forum, Rome.
EUDAT receives funding from the European Union's Horizon 2020 programme - DG CONNECT e-Infrastructures. Contract No EPOS and EUDAT.
EGI-InSPIRE RI EGI-InSPIRE RI EGI-InSPIRE EGI-InSPIRE RI EGI-InSPIRE EGI-InSPIRE RI
Get Data to Computation eudat.eu/b2stage B2STAGE How to shift large amounts of data Version 4 February 2016 This work is licensed under the.
Store and exchange data with colleagues and team Synchronize multiple versions of data Ensure automatic desktop synchronization of large files B2DROP is.
EGI-InSPIRE RI EGI-InSPIRE EGI-InSPIRE RI EGI Services for Distributed e-Infrastructure Access Tiziana Ferrari on behalf.
EGI-InSPIRE RI EGI-InSPIRE EGI-InSPIRE RI A pan-European Research Infrastructure supporting the digital European Research.
EUDAT receives funding from the European Union's Horizon 2020 programme - DG CONNECT e-Infrastructures. Contract No EUDAT Services.
EGI-InSPIRE EGI-InSPIRE RI EGI strategy towards the Open Science Commons Tiziana Ferrari EGI-InSPIRE Director at EGI.eu.
Introduction to AAI Services
CESSDA SaW Training on Trust, Identifying Demand & Networking
Bob Jones EGEE Technical Director
Accessing the VI-SEEM infrastructure
PIDs in EUDAT Webinar, 15 Februari 2013
Towards a pan-European Collaborative Data Infrastructure
EUDAT Towards a European Collaborative Data Infrastructure
Next Steps.
GISELA & CHAIN Workshop Digital Cultural Heritage Network
AAI for a Collaborative Data Infrastructure
Certification of Trusted Repositories
Ian Bird GDB Meeting CERN 9 September 2003
Identity Management and Authorization
Federated Identity Management for Researchers (FIM4R)
NA3: User Community Support Team
EGI-Engage Engaging the EGI Community towards an Open Science Commons
Exploitation of ISS Scientific data - sustainability
Identity Management and Authorization
Mark van de Sanden Giovanni Morelli
Solutions for federated services management EGI
Thursday pilot session: 7-minutes
EOSCpilot All Hands Meeting 9 March 2018, Pisa
WP 5 Shared Data Access & Enrichment
BioMedBridges – Work Packages 2 & 12
Common Solutions to Common Problems
European Research Data Services, Expertise & Technology Solutions
EOSCpilot All Hands Meeting 9 March 2018, Pisa
GISELA & CHAIN Workshop Digital Cultural Heritage Network
EUDAT Site and Service Registry
Bird of Feather Session
EOSC-hub Contribution to the EOSC WGs
Presentation transcript:

EUDAT operations perspective and initial achievements Johannes Reetz EGI Technical Forum 2012 EGI/EUDAT/PRACE operations workshop Prague, 18 Sep 2012 eudat.eu

raw data description registration preservation citable publication analysis enrichment referable data temporary data increasingly automatic processes as well as crowd sourcing increasing volumes Knowledge Creation Cycle is changing 2

Lossless separation of content and carrier in the digital domain changes the world - a revolution comparable with the invention of book printing Data creators are not known personally to data users anymore - we need to solve the trust problem Science and research are one of the primary drivers for the data tides Data accessibility changes nature, pace and direction of research Diversity in many dimensions is the dominant feature of many scientific information and this is expected to increase due to the inherent innovation forces Technology allows to involve the citizens in different roles – not just as consumers, also as contributors of valuable information (crowd sourcing), increasing volume and complexity Increasing pressure towards open access Setting the Scene (1) 3

Repurposing data within and across disciplinary domains requires significant efforts on integration and interoperability (e.g. wrt meta data), as well as agreement on standards and sharing practices. It is not just the sheer volume of data, but its increasing complexity and diversity - even though the increasing amount of valuable data needs to be tackled too Data re-usage in an open data infrastructure implies that data creators, managers and users must rely on each other’s quality of work. Advanced mechanisms are necessary to establish trust between all stakeholders. This requires input and feedback from the communities. Setting the Scene (2) 4

5 Collaborative Data Infrastructure (HLEG)

The Case for Multi-Disciplinary Services 6

7 The EUDAT project

EUDAT Communities on board 8

9 Communities ↔ Common Data Centres

1. Capturing Communities Requirements (WP4)  Understand how data is organised + collect requirements starting with the EUDAT initial communities: EPOS, LIFEWATCH, CLARIN, ENES, VPH  Further interviews have been conducted and analysis documents are available about CESSDA, DARIAH, DiXa, ECRIN, EMSO, ICOS, PanData and WLCG 2. Building the (generic) services (WP5)  Technology appraisal  Development of services (via taskforces) 3. Deploying services and operating the collaborative infrastructure (WP6)  Co-designing the collaborative (federated) data infrastructure with interfaces for federated operations  Integrating and coordinating resource provision, service provision, operations and support Service Delivery Approach 10

Service Handover 11

Classification of Services 12 1.Collaborative Services wiki, task tracker, versioning system, webserver, user documentation produce and publish 2.Operational Services monitoring, site registry, service registry, resource registry, resource provisioning coordination 3.EUDAT Core Services rule-oriented data mngmt system, file transfer service, credential conversion EUDAT Coordinated Core Services persistent identifier service, credential delegation 4.Community Service Housing EUDAT site provides resource, community admin deploys/operate the service 5.External Community Service external site, e.g., community center provides resources and services 1.Collaborative Services wiki, task tracker, versioning system, webserver, user documentation produce and publish 2.Operational Services monitoring, site registry, service registry, resource registry, resource provisioning coordination 3.EUDAT Core Services rule-oriented data mngmt system, file transfer service, credential conversion EUDAT Coordinated Core Services persistent identifier service, credential delegation 4.Community Service Housing EUDAT site provides resource, community admin deploys/operate the service 5.External Community Service external site, e.g., community center provides resources and services

Classification of Services 13 1.Collaborative Services wiki, task tracker, versioning system, webserver, user documentation produce and publish 2.Operational Services monitoring, site registry, service registry, resource registry, resource provisioning coordination 3.EUDAT Core Services rule-oriented data mngmt system, file transfer service, credential conversion EUDAT Coordinated Core Services persistent identifier service, credential delegation 4.Community Service Housing EUDAT site provides resource, community admin deploys/operate the service 5.External Community Service external site, e.g., community center provides resources and services 1.Collaborative Services Confluence, Jira, SVN, Apache+DRUPAL, Zopyx Produce & Publisch Authoring Environment (?) 2.Operational Services Nagios+SVN, GOCDB, Resource Coordination Tool (under development), … 3.EUDAT Core Services iRODS, gridFTP, XSEDE file manager, Contrail federation software (?) EUDAT Coordinated Core Services EPIC PID service (MoU), Globus myproxy 4.Community Service Housing e.g. CLARIN WebLicht (service for the automatic annotation of text corpora) 5.External Community Service …

14 Dynamic replication to HPC workspace for processing

Attribute Provider AuthZ either community-managed or ( ) attributes provided by user’s home IdP are reused Identity credential conversion AtP 1 AtP 2 AtP 3 zoned credential conversion service unique user Ids, project-wise mapped to attribute based access control information Different types of Identity Providers AuthN * consolidated credentials IdP A IdP B IdP D IdP C eID shib OpenID x.509 *

Community center EUDAT center EPOS CLARIN ENES VPH Lifewatch 16 Towards Safe Replication

rule-oriented replication CLARIN 17

raw data description citable publication analysis enrichment EUDAT business domain - two generic data services 18 registration preservation temporary data referable data Data Staging Safe Replication

Generic Service Model Fundamental Core Services meet strongly overlapping service requirements Extended Core Services are mainly community-supported, community requirements are typically overlapping between some disciplines Collaboration between Teams Fundamental Core Services are operated and supported by an Operations Team which collaborates across the participating centres. Extended Core Services and other joint multi-disciplinary service must be community-supported, the requirements are overlapping between a specific subset of disciplines WP6 task breakdown structure 19

Operational approach 20 Coordinated provisioning of network, storage, and compute Resources Coordinated provisioning of Services on top of these resources, including a registry for all the services deployed and operated and service status monitoring Coordination of Operations via EUDAT teams, integration support for (new) resource/service providers, issue tracking, change management, user support with interfaces to community support teams Coordinated security and quality management CSIRT, security and quality management, certification of repositories and centres (Data Seal of Approval) Operations

21 Site A Services Compute Storage Deputy Security SUPPORT Site A Network Site A Services Compute Storage Deputy Security SUPPORT Site B Network Site A Services Compute Storage Deputy Security SUPPORT Site C Network Site A Services Compute Storage Deputy Security SUPPORT Site D Network Service Provisioning Resource Provisioning Qlty Assurance, Security Coordination & Support

22 EUDAT Operationsteam (1st Project Year) bi-weekly meetings

Organisation of EUDAT Operations 1st User Forum - Barcelona, 7/8. March Support Security ResourcesServices Operations Quality Assurance Operations

Coordination Quality Assurance Coordinated Service Provisioning 1st User Forum - Barcelona, 7/8. March Support Security Resources Core services AAI, PID, data management, … Community services e.g. workflow, metadata registries Operational services e.g.site/service reg., monitoring, TTS, Collaborative services e.g. wiki, task tracker, svn, webpage, Services Operations

Coordination Quality Assurance Coordinated Resource Provisioning 1st User Forum - Barcelona, 7/8. March Support Security Services Resources network service hosting platforms storage Operations

Coordinated Support 1st User Forum - Barcelona, 7/8. March Security ResourcesServices Operations Coordination Quality Assurance Operations Support community support (tbd) internal support, issue tracking (done) user documentation (tbd) data centres integration support (tbd) Support

Coordinated Security 1st User Forum - Barcelona, 7/8. March Support ResourcesServices Operations Coordination Quality Assurance Support Security risk analysis (started) security management plan security policies, guidelines security assessment operational security (started) (Computer Security Incident RT) Operations

Coordinated Quality Assurance 1st User Forum - Barcelona, 7/8. March Support Security ResourcesServices Operations Coordination Support Quality Trusted (certified, e.g. DSA) data management long-term preservation compliance with security policies, data privacy policies. Operations

1st User Forum - Barcelona, 7/8. March Operations Coordinated Operation Operations Team EUDAT centre and community deputies administration of site registry config./change management problem solving, escalation Operations Support Security ResourcesServices Quality Assurance

30 Nagios based monitoring

31 as site registry

32 GOCDB as service registry

Helpdesk TTS Aim of the EUDAT TTS Provide a contact point for support activity Provide a tool to manage: –Internal Request/Issues –External Request/Issues Integrated with the Resource/Service Provisioning Coordination Tool 33

TTS configuration Approach on a “per site” basis: –Three different types of ticket which results in 3 different generic queues: Support Resources Services Security? –A queue for each site to allow local support collaboration on a single request/issue 34

Access to the service Web: using the Crowd SSO credentialshttps://helpdesk.eudat.eu Mail: 35 Queue NameQueue Typ address to open ticket Mail address notified SupportRouting queuettseudat- Helpdesk responsible ResourcesRouting queuettseudat- juelich.de ServicesRouting queuettseudat- t

Achievements in Operations 36 Initial operations team is working, bi-weekly meetings Services for collaboration and communication available and accessible (currently using CROWD SSO for authN/authZ)  Confluence Wiki, Jira, SVN, VC, Web server, TTS (Request Tracker) Service Hosting Framework is provided via the initial centres  Coordinated provisioning of service hosting resources with storage  Resources are provided in a coordinated way, many are currently used for test environments EUDAT-EPIC MoU in preparation; EPIC PID service run by SARA will be provided as coordinated core service GOCDB has been tested using a separate instance. It was shown to be useful as a site and service registry, it is used by the central monitoring. Central monitoring service makes progress (central instance available) – demo in June 2012 Security plan available for EUDAT (site security contacts, initial CSIRT) Quality assessment document is available (based on the Data Seal of Approval), sites are now starting with the assessment process.

Plan for the next project year 37  TTS and Site Registry Transition from preproduction to production  User documentation: Produce and Publish Authoring Framework (tbd), Editorial Team  Guidelines for integrating additional centres and external services in the 2 nd year  Service Provisioning  advance the Nagios monitoring infrastructure and further integrate production services  promote existing services to communities via EUDAT website with WP3  get communities services integrated (community service housing) with WP4  Resource Provisioning (T6.3)  Complete the Resource coordination framework: DB with web frontend  collect further resource (storage) requirements from the communities  Security and QA  impove acceptable user policy (wrt to privileged external admins)  advance the security plan for EUDAT and its implementation  Centers proceed with the DSA assessment

Welcome to the 1st EUDAT Conference! October 2012, Barcelona International event with keynotes from Europe and US A forum to discuss the future of data infrastructures Project presentations and poster sessions 2nd EUDAT User Forum Training tutorials