1. Virtual multidisciplinary EnviroNments USing Cloud infrastructures Data Management at VENUS-C Ilja Livenson KTH livenson@kth.se

2. VENUS-C Introduction EU FP7 infrastructure project aiming at – creating a platform that enables user applications to leverage on cloud computing principles; – creating a sustainable infrastructure with a valid business model. Led by Engineering, partners include MS/EMIC, BCS, KTH, OGF.eeig, UPV, CNR and many more. User scenarios from biomedicine, civil engineering, civil protection and emergencies, marine biodiversity and more. There will be a more specialized section on VENUS-C: Wed 13/04, 11:00.

3. Data Management (overview) Goals Architecture and functionality Deployment models + demo Status and plans

4. Goals Provide an integration layer for data management facilities – Cloud storage – DCI storage Not a mandatory component, but providing additional value – Ease of migration – More convenient than using cloud API directly

5. Data Access Strategies

6. Motivation Centralized control over resources Easier access to resources (no additional credentials) Optimization effect from optimizing data of multiple users can be higher than if optimized individually Easier release cycle. It is much easier to update a central CDMI-proxy service than a set of deployed libraries Easier exposure of local storage through RESTful API

7. CDMI SNIA’s Cloud Data Management Interface – http://www.snia.org/cloud http://www.snia.org/cloud – Standard (1.0.1h) + rising adoption by vendors CDMI provides an interface description for performing a set of operations on the data elements from the cloud CDMI objects: – Data – Queue – Container – Domain – Capability

8. Data Abstractions Required by VENUS-C Applications Blob – generic data item + metadata Message Queue – FIFO queue (Key-value database) – Aka NoSQL databases – Semantics greatly depend on implementation – Not part of a CDMI v1

9. CDMI-Proxy Structure Core CDMI FUSE HTTP FTP CIFS AuthZ/AuthN Generic BlobGeneric MQ Generic Document DB local Azure S3 CDMI local Azure SQS CDMI Azure CouchDB SimpleDB

10. Client Side Developing CDMI SDKs in.Net, Java and Python, also exporting as CLIs Community efforts – SARA – OCCI/CDMI demo from – More to come

11. Data Flow

12. Deployment Models Everything from the laptop – Client would need to have a business relationship with a cloud provider VENUS-C on-premises – E.g. VENUS-C services deployed at a research group VENUS-C in the cloud – E.g. a commercial offer to a company

13. Demo deployment

14. VCDM Added Value Extended namespace for 1-level cloud storages (AWS S3, Azure Blob) 3 rd party transfers with encryption of the content Enrichment of data items with (approximate) costs Basic accounting + interface to VENUS-C accounting and billing engine

15. Security

16. Status and plans Core functionality is getting more mature – Supported ADTs: Blobs and Message Queues Delivery of the first prototype is due in Autumn 2011 – Prerelease earlier CDMI frontend is compatible with at least one other CDMI service Will not expose document store via CDMI – Custom installations at DCIs with a shared security system

17. Technical Details CDMI-Proxy core – Twisted networking engine (Python) Backends – Metadata store: CouchDB, Azure Table, AWS SimpleDB – Blobs: localdisk, Azure Blob, AWS S3, CDMI – MQ: AMQP (RabbitMQ ), Azure Queue, AWS SQS, CDMI

18. Thank you!

19. Agency Scientist University Army-of-OneCommercial S/W support Application Support Customer Support Central Services Resources Included Optional Not provided