Bringing cloud technology to distributed data infrastructures EGI CF 2013 Martin Hellmich (presenter) Jedrzej Rybicki Maciej Brzeźniak Date :

A bit of context 2 Towards a pan-European Collaborative Data Infrastructure Production Services Safe Replication Data Staging Metadata AAI Research & Development Scalable Federation Architectures Data Preservation Data Access and Transfer Workflows

Three Projects Cloud storage integration –iRODS managing an OpenStack Swift backend –Extending DPM with S3 storage In-storage processing –Call Hadoop jobs from iRODS 3

My Goal Show the projects Find interest in the communities (we are interdisciplinary) Start discussion about cloud integration –Backend or frontend? –Outsource or restructure? –Where are limitations? 4

The Cloud Integration Projects iRODS-OpenStack Expose existing S3/OpenStack storage (managed otherwise) iRODS frontend protocols Local storage as cache 5 DPM-S3 Add new storage to DPM Expose HTTP only (but grid-aware, X509, VOMS) Outsource storage and network traffic

iRODS-OpenStack Swift Maciej Brzezniak Date :

Sidestep: iRODS compound resources 7 iRODS resources: Cache Archive Virtual iRODS compound resources: Virtual resource Maps from PUT/GET to POSIX Provides a cache

iRODS managing an S3 backend Ingredients: iRODS server S3 Driver (in C) iRODS-S3 Driver Glue Swift-to-S3 frontend 8 iRODS Site Disks OpenStack Swift/S3

Achievements Transparent cloud storage Cloud auth through central accounts Low Overhead through iRODS Speedups with caching Limitations: Filesize limit (2/5GB) Issue moving files inside the cloud 9 iRODS Site Disks S3/OpenStack

DPM-S3 Martin Hellmich Date :

DPM now uses dmlite 11 S3

Sidestep: the S3 protocol HTTP + custom headers Access ID + Secret Key + HTTP Cmd + Time => Signature Can be: Header: Authorization: AWS WSAccessKeyId:Signature In URL: ?AWSAccessKeyId=AKIAIOSFODNN7EXAMPLE&Signature= NpgCjnDzr%2BWFzoENXmpNDUsSn8%3D&Expires=

Extending DPM with S3 Storage 13 Site Disks S3 Signed URL redirect Ingredients: dmlite dmlite-plugins-s3 Amazon S3 OpenStack Swift S3 frontend Ceph/RadosGW

Achievements Only nameserver traffic local Cloud storage managed with central account Grid-enabled HTTP Standard HTTP clients Filesize limit (or S3 client) 14 Site Disks S3 Signed URL redirect

In-Storage Processing Jedrzej Rybicki & Benedikt von St. Vieth Date :

Motivation Example HPC workflow: 16 Site High Performance Computing Storage preprocessing Site High Performance Computing Storage + preprocessing

Sidestep: iRODS rules 17 Condition: $objPath like /x/y/z/* Or $rescName == demoResc8 Rule: printHello { print_hello; } Act freely on certain triggers At least C and Python

Benedikt von St. Vieth & Jedrzej Rybicki 18 In-Storage Processing

Achievements 19 Everything is a file Easy job specification in Apache Pig Caching of results Predefined scripts or custom jobs?

Summary 20 There are different ways to integrate cloud storage for different scenarios Storage-based computing can be made transparent

Thank you! OpenStack/iRODS –Maciej Brzezniak (PSNC) DPM-S3 –Martin Hellmich (CERN) In-storage processing on iRODS –Jedrzej Rybicki / Benedikt von St. Vieth (JSC) 21 Projects contacts Any Questions?