Get Data to Computation eudat.eu/b2stage B2STAGE How to shift large amounts of data Version 4 February 2016 This work is licensed under the Creative Commons CC-BY 4.0 licence. Attribution: EUDAT –

eudat.eu/b2stage B2STAGE is… a reliable, efficient, light-weight and easy-to-use service to transfer research data sets between EUDAT storage resources and high- performance computing (HPC) workspaces 2

eudat.eu/b2stage A truly pan-European Infrastructure 3 EUDAT offers common data services to both research communities and individuals through a network of 35 European organisations. EUDAT wants to enable European researchers from any discipline to preserve, find, access, and process data in a trusted environment, as part of a Collaborative Data Infrastructure. European infrastructures Technology Providers Research Communities

eudat.eu/b2stage Community-Driven Solutions 4 PHYSICAL SCIENCES & ENGINEERING SOCIAL SCIENCES & HUMANITIES MATERIALS & ANALYTICAL FACILITIES ENVIRONMENTAL SCIENCES MAPPER BIOMEDICAL & MEDICAL SCIENCES EUDAT services are designed, built and implemented based on user community requirements.

eudat.eu/b2stage The EUDAT Service Suite 5

eudat.eu/b2stage move large amounts of data between data stores and high- performance compute resources re-ingest computational results back into EUDAT deposit large data sets into EUDAT resources for long-term preservation Facilitating communities to: Features: high-speed transfer reliable and light-weight manages permanent PIDs 6 B2STAGE Features

eudat.eu/b2stage Why use B2STAGE? 7 Research challenges are getting larger and more complex : E.g. full-Earth climate simulation, coupled simulations of multiple organs in the human body, seismic analyses of earthquakes at continental scale High level benefits Researcher data and compute demands are rising fast Efficient transfer of data to high performance computing (HPC) workspaces is essential especially in distributed computing, where resources are geographically dispersed

eudat.eu/b2stage Why use B2STAGE? 8 Facilitates transfer of large data collections from EUDAT storage resources to HPC facilities. Specific User Requirements Provides the means to re-ingest computational results back into the EUDAT infrastructure. Ingests data sets into EUDAT resources for long-term preservation. Offers reliable, efficient, easy-to-use tools to manage data transfers. The Data Staging Script is the only tool handling data transfer using PIDs.

eudat.eu/b2stage Who can use B2STAGE? Researchers can transfer large data collections from EUDAT storage resources to HPC facilities for processing. Community Managers can replicate community data through a lightweight service and ingest data sets to EUDAT storage resources for long term preservation. 9

eudat.eu/b2stage How can you use B2STAGE? EUDAT offers B2STAGE to all registered researchers and interested communities, enabling them to make use of the service to stage data out of EUDAT, and ingest computational results back. Access to remote HPC facilities should be negotiated and arranged by individual users in parallel. To help researchers use the B2STAGE service, EUDAT offers documentation, training material and a service helpdesk. 10 For more information please

eudat.eu/b2stage How can you use B2STAGE? 11

eudat.eu/b2stage How does B2STAGE work? 12 GridFTP server iRODS-DSI User desktop GridFTP client data control PID Registry PID control HPC GridFTP server

eudat.eu/b2stage User desktop How does B2STAGE work? 13 GridFTP client File system GridFTP server iRODS-DSI PID Registry PID data control

eudat.eu/b2stage B2STAGE User communities VPH Community ingesting data onto EUDAT resources Approximately 12TB will be ingested through this service VPH data also replicated between RZG and PSNC sites B2STAGE will foster the collaboration with EGI and PRACE to develop cross-infrastructure usage: B2STAGE will be the main service to enable the interoperability of these infrastructures. Numerous new communities to adopt it as part of the 2015 and 2016 Calls for Collaboration 14

eudat.eu/b2stage B2STAGE summary B2STAGE offers: data staging functionalities to easily and efficiently transfer data from EUDAT storage resources to HPC facilities a powerful mechanism to ingest data onto EUDAT resources a script to facilitate the staging, ingest and retrieval of PID information of transferred data B2STAGE is unique in handling PIDs for the data 15

eudat.eu/b2stage Future features The Data Staging Script will be replaced by a modular and extensible python library which will furnish the users with a programmable interface towards most of the EUDAT services. 16

eudat.eu/b2stage 17 For more info: User documentation: Thank you