INDIGO – DataCloud PaaS activities @INFN RIA-653549 marica.antonacci@ba.infn.it INFN-Bari

Paradigm shifts From static partitioning to dynamic resource allocation Improve resource utilization Improve scalability and fault tolerance From Monolithic App to Microservices Easier to develop Easier to scale Fault isolation From VMs to Containers Run your application (even legacy) everywhere: on public and private clouds INDIGO-DataCloud RIA-653549

Docker ecosystem in Openstack Heat Solum Kubernetes Docker Magnum OpenShift CloudFoundry INDIGO-DataCloud RIA-653549

Heroku-inspired frameworks Deis builds upon Docker and CoreOS to provide a lightweight PaaS Build/Release/Run Similar solutions: flynn, tsuru INDIGO-DataCloud RIA-653549

Mesosphere Mesosphere offers a layer of software that organizes your machines, VMs, and cloud instances and lets applications draw from a single pool of intelligently- and dynamically-allocated resources, increasing efficiency and reducing operational complexity Key Concept: Data Center Operating System INDIGO-DataCloud RIA-653549

Mesos Two-Level Scheduling INDIGO-DataCloud RIA-653549

Mesos Frameworks Long Running Services Big Data Processing Aurora, Marathon, Singularity, SSSP Big Data Processing Cray Chapel, Dpark, Exelixi, Hadoop, Hama, MPI, Spark, Storm Batch Scheduling Chronos, Jenkins, JobServer, Torque Data Storage Cassandra, ElasticSearch, Hypertable INDIGO-DataCloud RIA-653549

Cluster configuration - example INDIGO-DataCloud RIA-653549

Conclusions Preliminary tests Deis: Mesos: Installation is simple, but takes too long (~1h for 5 nodes cluster) Poor stability Debugging not so easy Supported apps: twelve-factors, only ONE HTTP port Mesos: Easy to deploy Packages available for Debian/Ubuntu and Redhat/Centos Automatic provisioning available Easy to scale INDIGO-DataCloud RIA-653549

Backup Slides INDIGO-DataCloud RIA-653549

Service Discovery Mesos-DNS (alpha) allows to connect to an application in the Mesos datacenter knowing only its name can be made fault-tolerant by launching with a framework like Marathon, that can monitor application health and re-launch it on failures. Marathon SD allows easy network communication between services Based on HAProxy TCP/HTTP load balancer plus an assistant script that uses Marathon’s REST API to periodically re-generate a HAProxy configuration file INDIGO-DataCloud RIA-653549

Applications in Marathon JSON app definition curl -X POST -H "Content-Type: application/json" http://<master>:8080/v2/apps -d@app.json Marathon hands over execution to Mesos. Mesos executes each task in its own sandbox environment Marathon knows how to handle application resources A typical pattern in the development and deployment cycle is to have your automated build system place the app binary in a location that's downloadable via an URI. Marathon can download resources from a number of sources, supporting the following URI schemes: file: http: https: ftp: ftps: hdfs: s3: INDIGO-DataCloud RIA-653549