European Grid Initiative Data Services and Solutions Part 2: Data in the cloud Enol Fernández Data Services and Solutions Webinar1

Outline Data in the cloud Use cases and technical details Block Storage Object Storage Native cloud solutions State of the art: status in EGI Examples Future Plans Data Services and Solutions Webinar2

What is the EGI Federated Cloud 3 EGI Federated Cloud is based on: Standards and validation: federation is based on common Open-Standards – OCCI, CDMI, OVF, GLUE, etc... Heterogeneous implementation: no mandate on the cloud technology, the only condition is to expose the chosen interfaces and services. The EGI Federated Cloud is federation of institutional private Clouds, offering Cloud Infrastructure as a Service to scientists in Europe and worldwide. Data Services and Solutions Webinar

FedCloud IaaS Capabilities Computing VM Management VM Marketplace Storage Block Storage Object Storage Data Services and Solutions Webinar4

Block Storage Persistent block level storage to use with VMs Use as any other block device from VMs Snapshotable Simple usage Consistent and low- latency performance SSDs (in some sites) High Performance From GB to TB Create and attach to VMs on demand Scale to your needs VM Data Services and Solutions Webinar5

Object Storage Data storage infrastructure for storing and retrieving data from anywhere at any time Simple REST APIs for managing and accessing data API Access Store as much data as needed. Get accounted only for the space used. Scalable Define ACLs on each object, share publicly your data Sharing Data Services and Solutions Webinar6

Block Storage vs Object Storage Block StorageObject Storage Access only from within a VM only at the same site the VM is located from any device connected to the internet. Sharingnot possible possible (data can be kept private or public) Accounting for the entire volume, regardless how much of it is actually used only for the data stored Integration easy with any application capable to write/read file from a local disk requires a client to be integrated within the application Data Services and Solutions Webinar7

Use Cases Block StorageObject Storage Application hosting Data Processing Database Large Data File Storage & Backup Static Content Media Serving & Sharing Big Data Data Services and Solutions Webinar8

Block Storage: Typical Use Store your data on volumes Data persists independently of VM Stripe volumes for better performance Share via network filesystem (e.g. NFS) or use as DB store VM NFS Data Services and Solutions Webinar9

Block Storage: OCCI OCCI (Open Cloud Computing Interface) is a OGF standard API to facilitate interoperable access to cloud resources Block storage in FedCloud is managed via OCCI: create/delete volumes Attach/detach (link/unlink in OCCI terms) to VMs Once attached, use as other disk in VM Data Services and Solutions Webinar10

Object Storage: CDMI FedCloud object storage is managed via CDMI (Cloud Data Management Interface) RESTful API for operations on storage objects Developed by SNIA, now ISO/IEC Very flexible API, based on capabilities: Object basic capabilities (create/get/delete/list) Object ACLs Import from external sources, export as Filesystems … Data Services and Solutions Webinar11

Native Cloud Solutions Cloud Management Frameworks (CMF) provide their own APIs for managing cloud storage Usually more features than OCCI/CDMI However, not (yet) fully integrated in EGI’s FedCloud OpenStackOpenNebulaSynnefo Block Storage Cinder APIOpenNebula API Cinder API Object StorageSwift APIN/A Pithos API + Swift API Data Services and Solutions Webinar12

State of the Art: Block Storage Block storage is supported on all FedCloud CMFs and sites OpenStackOpenNebulaSynnefo OCCI Basic Operations Yes OCCI advanced (resize, snapshot) No Native API Basic Operations Yes Native API advanced YesPartialYes Data Services and Solutions Webinar13

State of the Art: Object Storage CDMI support CDMI server framework by Synnefo On going effort to support OpenStack Basic client available Native APIs allow basic and advanced capabilities OpenStackSynnefoOneDataOpenNebula CDMI Basic Operations In ProgressYes N/A Native APIYes N/A Data Services and Solutions Webinar14

Example: Chipster Chipster is a graphical application for data analysis, with server backend Original Chipster VM included big collection of tools and data (~200GB) Deployment at FedCloud Separated VMs from tools and data with block storage NFS server for these volumes Chipster VMs mount the NFS exports on start-up NFS Server NFS Server Tools Volume Data Volume Chipster VM Chipster VM Chipster VM Chipster VM EGI FedCloud Resource Provider Data Services and Solutions Webinar15

Example: EISCAT-3D (I) EISCAT-3D is a 3D imaging radar to be located in the northernmost parts of Europe. Open Source Geospatial Catalogue (OSGC) Portal provides access to the data stored in Object Storage providers at FedCloud Planning extra services to further process the EISCAT-3D data and make it available in the portal Data Services and Solutions Webinar16

Example: EISCAT-3D (II) 17 Open Source Geospatial Catalogue (OSGC) CESNET site (CZ) Catalogue EISCAT archive Object Storage Juelich site (DE) OpenStack SWIFT CDMI with HTTP export EGI Federated Cloud Near Real Time tool to import data automatically from receiving stations Admin tools Scientific users Data administrators Web browser wget 5m files, ~1TB in total On-site Off-site Phase 1: In ENVRI Phase 2: In EGI-Engage Pre- processing service 1 Pre- processing service N... Processing / visualization service 1 Processing / visualization service N... Data Services and Solutions Webinar

Plans EGI-ENGAGE: Effort to further develop OCCI/CDMI interfaces in FedCloud OneData development Storage Testbed Other related projects: INDIGO will develop (data) cloud solutions Data Services and Solutions Webinar18

Distributed multi-provider storage Flexible access control Intra-federations scenarios for sharing data Works with Tokens or X.509 POSIX client for mounting user’s space Scalable from Single NAS to Large Datacentre Can be deployed on top of high-performance parallel storage solutions with very small overhead < 5%. Support for open data scenarios in preparation Onedata is currently supported by: PLGrid, EGI-Engage, INDIGO-DataCloud, ESPREX for ISS Data Services and Solutions Webinar19

Storage Testbed Testbed will allow to: Test tools and setups in a distributed and big enough collection of resources Pilot applications to be migrated to production Currently looking for Resource Providers Join as users/use cases to articulate requirements and preferences for this infrastructure Data Services and Solutions Webinar20

References EGI Federated Cloud resources Wiki site: User support: User support Federated Cloud Communities: Federated Cloud Storage HOWTO: orage orage Related Standards: OCCI: CDMI: Data Services and Solutions Webinar21