1. Occopus and its usage to build efficient data processing workflow infrastructures in clouds József Kovács, Péter Kacsuk, Ádám Novák, Ádám Visegrádi MTA SZTAKI

2. Motivations for Occopus – End user view There are many use cases where the goal is to set up a complete infrastructure or a set of services (called as virtual infrastructures) on demand in the cloud Goal is to enable end-user scientists to easily (ideally by 1 click) deploy such a VI in the cloud either temporarily or for a longer period of time Example: o biologist would like to run autodock in-silico simulation between one receptor molecule and 100.000 ligand molecules o The biologist takes the autodock VI from EGI AppDB and by 1 click deploys it in a target cloud and uses it

3. Vina Generator Vina Collector Step 2: Deploy Autodock VI by 1 click in EGI Cloud Example: Autodock virtual infrastructure in cloud EGI AppDB VI descriptor Step 1: Take VI descriptor of Autodock VI from EGI AppDB Inputs: + Vina config + Receptor molecule + Ligand set Vina config Ligand sets All docking resuts Output: 5 Best docking results Step 3: Run autodock appl in cloud with 100.000 ligands

4. Motivations for Occopus – VI developer view Enable virtual infrastructure developers to develop the required VIs and VI descriptors for various cloud types Requirements: o Easy and fast development of VIs without very deep cloud expertise o Easy specification/description of the required VI o Fast and automatic deployment of the VI specified by the VI developer o Fault-tolerant life-cycle management of deployed virtual machines o Cloud type independent solution

5. Cloud orchestrators: State of the art

6. Main features of Occopus Command line tools and REST API service Multi-cloud support Pluggable architecture -> e.g. connecting to a new cloud requires only a new plugin Error-detection (fatal/transient) and recovery Support for configuration management tools (like Chef) Garbage collection at cancellation Manual scaling Serial and parallel synchronization strategies among nodes of a VI

7. How does Occopus work? At deployment time the dependencies among nodes are managed Required communication among nodes are handled to enable their interaction and collaboration If Occopus works as a service it manages the whole life-cycle of the nodes A BC [ B, A ] [ C, A ] A B C parallel A BC [ B, A ] [ C, A ] [ B, C ] A B C parallel sequential VI dependency graph VI dependency descriptor Deployment order

8. Pluggable Occopus architecture Occopus EC2 Node definition resolver plugins Cloud handler plugins Service Composer plugins NOVA CloudBroker ChefChef+cloudinit CloudBroker Node definition resolver: interprets node definition (which is a kind of template) and fills it by actual values. Cloud handler: is able to talk to a cloud interface, to instantiate, shutdown and query virtual machines. Service Composer: is able to cooperate with third-party configuration management tools, like `chef`.

9. Supported single-cloud setups: EC2 EC2 Occopus EC2 A B C e.g. OpenNebula or Amazon Occopus can handle EC2 cloud interface based clouds: o OpenNebula based clouds in EGI FedCloud (tested on SZTAKI Cloud) o Amazon and other commercial clouds Nodes of deployable infrastructure are instantiated on the target cloud through EC2. EC2 plugin of Occopus utilises Cloud-init for contextualisation. A BC VI Descriptor Can run on your desktop or as a service

10. Supported single-cloud setups: NOVA NOVA Occopus Nova A B C OpenStack Occopus can handle NOVA cloud interface and hence Openstack based clouds in EGI FedCloud (tested on BIFI Cloud) Nodes of deployable infrastructure are instantiated on the target cloud through NOVA. NOVA plugin of Occopus utilises Cloud-init for contextualisation. A BC VI Descriptor

11. Support multi cloud setups in EGI FedCloud EC2 Occopus EC2 A NOVA Nova B EC2 C e.g. SZTAKI Cloud e.g. BIFI Cloud e.g. CESNET Cloud Occopus can utilise multiple interfaces within a cloud federation like EGI FedCloud Nodes of deployable VI are instantiated on the target clouds defined by the VI descriptor A BC VI Descriptor

12. Supported multi-cloud setup through CloudBroker Occopus A B CBP C e.g. CloudSigma e.g. OpenStack e.g. Amazon EC2 Nova Occopus can utilise CloudBroker multi-cloud support for commercial clouds CBP plugin of Occopus utilises file upload facilities for contextualisation A BC VI Descriptor

13. Example: Autodock workflow in multi-cloud Autodock infrastructure deployed by Occopus Data Collector Service 195.111.0.168:80 Data Collector Service 195.111.0.168:80 MTA SZTAKI OpenNebu la CBP Generato r BIFI OpenStack Vina Amazon AWS Collect or Inputs: + Vina config + Receptor molecule + Ligand set Vina config Ligand sets All docking resuts Best docking results Vina NOVA EC2 Outputs: + Best docking result

14. How to describe your infrastructure Occopus requires 3 descriptions: o Virtual infrastructure description: Specifies the nodes (services) to be deployed and all cloud- independent attributes e.g. input values for a service. Specifies the dependencies among the nodes, to decide the order of deployment Specifies scaling related attributes like min, max number of instances o Node definition: Defines how to construct the node on a target cloud. This contains all cloud dependent settings, e.g. image id, flavour, contextualisation o Cloud configuration: Specifies the type of the target cloud, its endpoints, credentials and all cloud specific attributes See detailed tutorials at the Occopus web page: o http://occopus.lpds.sztaki.hu/tutorials http://occopus.lpds.sztaki.hu/tutorials

15. Example: single cloud + dependencies nodes: - &R name: receiver type: receiver_node backend_id: my_ec2_cloud variables: message: „I am receiver” - &S name: sender type: sender_node backend_id: my_ec2_cloud variables: message: "I am sender" dependencies: - [ *S, *R ] 'node_def: receiver_node': - implementation_type: chef+cloudinit backend_id: my_ec2_cloud service_composer_id: dummy image_id: ami-00001363 instance_type: m1.small context_template: !text_import url: file://receiver-node.yaml 'node_def:sender_node': - implementation_type: chef+cloudinit backend_id: my_ec2_cloud service_composer_id: dummy image_id: ami-00001363 instance_type: m1.small context_template: !text_import url: file://sender-node.yaml Infrastructure description:Node definitions: [ *S, * R ] means: S depends on R, therefore R must be deployed before S sender_node.yaml points to a cloud_init configuration file: #cloud-config write_files: - content: "{{variables.message}}\n" path: /tmp/message.txt permissions: '0644'

16. TOSCA is a standard for cloud orchestrators and hence Occopus will be an implementation of TOSCA in the very near future Even now Occopus is very close to be a TOSCA implementation The core concept and main use-cases are very similar Support for TOSCA in Occopus

17. TOSCA and Occopus Topology templates: Defines the topology of a service as a directed graph Occopus equivalent: Virtual infrastructure description TOSCA description

18. TOSCA and Occopus Node types: Define the properties and interfaces of a component Have capability and requirement definitons Are defined separately (for reuse purposes) Occopus equivalent: Node definition TOSCA description

19. TOSCA and Occopus Relationship types: Define the semantics and any properties of a relationship Indicates the elements it connects and the direction of the relationship (source and target) Are defined separately (for reuse purposes) In Occopus, relationships with attributes are in the infrastructure description TOSCA description

20. Current state of Occopus Open-source (License: Apache v2) Release v0.2.1 Python 2.7 Base webpage: http://occopus.lpds.sztaki.huhttp://occopus.lpds.sztaki.hu Git: https://github.com/occopushttps://github.com/occopus Documentation: o Users’ Guide o Developers’ Guide o Tutorials Testing: Python nosetests Deployment: setuptools o Package repository: http://pip.lpds.sztaki.hu/packageshttp://pip.lpds.sztaki.hu/packages

21. Occopus webpage

22. Occopus on github

23. Roadmap for future work Tosca converter to read Tosca descriptions Utilising the rOCCI interface to further support EGI FedCloud Integration with the EGI AppDB: o Work has started to store Occopus VI descriptions in AppDB o Goal: automatically extract information about target Clouds Make a standard PaaS for EGI FedCloud Resource monitoring and auto scaling Integrate service discovery tools (e.g. consul) Web-based user interface