Container cluster management solutions Kishore Jagannath System Architect Hewlett Packard Enterprise

Mesos/Marathon Architecture Apache Mesos is a resource manager and provides a datacenter view of a cluster of machines. Works in a Master/Slave architecture. Master schedules job on slaves which is executed by an executor. Masters are connected to service registry such as Zookeeper for leader election and HA. Offers resources to any framework connected on top of it. Marathon is a scheduler/framework providing orchestration capabilities. Mesos Master Assigns jobs to slaves and make resource offers to framework Mesos Slave Runs Mesos agents. Executes tasks and provides resource capacity to master Zookeeper Leader election of Masters Framework Selects and schedules tasks via resource offers. Deploys executors on agents to execute scheduled tasks

Mesos/Marathon Overview Demo

Docker Swarm Architecture docker swarm init <<options>> docker swarm join <<options>> Directly available within Docker engine from docker 1.12. Enable cluster setup in swarm mode with simple commands Create services instead of containers. Managers talk to workers to schedule tasks on the worker. Managers use Raft algorithm for leader election. Worker nodes communicate through gossip protocol. In built support for Service Registry and discovery. No need for external services like consul, etcd. In built Load balancing. Multi host container networking via Overlay networks Secured by default via TLS. Automatic reconciliation to desired state of cluster. Master Schedules tasks on slaves and exposes service commands. Slave Executes containers Internal Key Value Store Inbuilt key values store for master leader election using Raft algorithm

Docker Swarm Overview Demo

Kubernetes Overview APIServer Scheduler Controller Manager ETCD A group of tightly dependent containers within a POD and exposed as a Service. Allows specifying the desired replicas of pods within a service. Inbuilt support for service registry and load balancing Automatic reconciliation to desired state APIServer Interaction point with Kubernetes Scheduler Schedules the jobs on slaves Controller Manager Ensures the desired state of pod replicas ETCD Key value store enabling master election Kubelet Agent running on slave nodes Kube Proxy A small component in slave nodes enabling service discovery across Pods

Kubernetes Overview Demo

Comparison Table Mesos/Marathon Swarm Kubernetes Background Open Source based on Apache Mesos and productized by Mesosphere DCOS Open Source. Docker Community Open Source. Google Workloads Multiple workloads Container based workloads Desired state Supported by Marathon framework Supported by service replicas Supported via replication controllers Unit of scalability Container Pods High availability Application containers distributed on Mesos slaves Application containers distributed on swarm workers Pods distributed among slave nodes Load Balancing load balanced through Mesos DNS or Marathon LB Container tasks abstracted as a service and requests are load balanced. Pods exposed as services and requests are loadbalanced

Mesos/Marathon Swarm Kubernetes Service discovery Dns entries in case of mesos-dns or service ports in marathon-lb(HA Proxy) in each node Services discovered via service names enabled via gossip protocol in target nodes Pods talk to each other via Service names enabled through kube-proxy running in nodes. Key value stores ZooKeeper Inbuilt etcd Rolling update support Yes. Can specify minimal health capacity Yes. Can specify time interval between updates Yes. Can specify time interval between updates. Networking Expose container ports to host ports to expose. Service ports used for managing cluster Container ports not exposed. Service ports can be published. Inbuilt Overlay network Containers within pod share network space. Across pods clusterIp and external through serviceport. UsesFlannel for overlay network Support for stateful applications Yes with Datastax enterprise and RexRay No Yes from kubernetes 1.3. concept of PetSet

Salient Features Mesos/Marathon Docker Swarm Timetested: Apache Mesos as a resource manager has been around long time even before containers Support for containerized as well as non containerized workloads Easy to configure health check of applications apart from nodes. Open architecture. Support for multiple frameworks and multiple schedulers. Docker Swarm Have been around for more than a year but given new life with in built swarm mode and docker services from 1.12 Available inbuilt with docker engine so no additional installation. Super simple to configure just two commands. Support for service registry store, overlay networks and load balancing is inbuilt.

Salient Features(contd.) Will be soon integrated with other Docker solutions such as UCP in future releases. Exposed as Docker cli commands, very easy to learn and adapt for folks familiar with Docker. Kubernetes One of the initial popular clustering solutions around containers. Provides the concept of pods enabling closely dependent containers to communicate in same network space via localhost. Pods, Services, Deployments are discrete pieces and glued through selectors providing more flexibility. Inbuilt service registry and load balancing.