Not Just Another Mesos Framework Kubernetes on Mesos Not Just Another Mesos Framework Da Ma, Software Architect, kube-mesos owner, IBM Yong Feng, Senior Software Architect, IBM Guangya Liu, Senior Software Engineer, IBM

Why “Kubernetes on Mesos”? Container Runtime The container runtime packages and launches application instance in a sandbox with portable and flexible capability. Docker and rkt are container runtimes. Computing, Storage and Network Application Application Resource Management The resource management component provides an abstraction of resources (cpu, mem, …) for application and then allocates/provision resources among tenants and applications. Mesos is an open source resource manager. EGO is an IBM enterprise resource manager. Workload Management The workload management component manages the life cycle of an application as well as access to the application, including service compose, service discovery, load balance Kubernetes and Marathon are workload managers.

Shared state between fws by Optimistic Offer Why “Kubernetes on Mesos”? Shared state Two Level Scheduling ?? Shared state between fws by Optimistic Offer …… Monolithic PBS/SGE/LSF Mesos/YARN/EGO Res mgr & wld mgr tightly coupled Batch wld only 1992 2003 2016 future

“Kubernetes on Mesos” Organization Sponsor: Tim Hockin (Google) Champion: David Eads (Redhat) Owner: Klaus Ma (IBM) Github: kuberntes-incubator/kube-mesos-framework

Spectrum Conductor with IBM is Owner & User !! Watson/Cognitive Container Cloud Spectrum Conductor with Spark IBM Spectrum Conductor for Container GUI Workflow App Store Authentication Authorization k8s Session Scheduler HPC Persistent Volume Service Discovery Installer (Deploy, Reconfigure, HA, Scale, Rolling update) Mesos Master Distributed Key-value Store Service Load Balance Mesos Agent Trouble-shooting Monitor K8s executor container container pod pod pod container container Network Topology Image registry

Community Value IBM Value-add Customer Value Docker Hub Registry holds a repository of 75000+ Docker images Lots of application integrated with Mesos Kubernetes enable micro-service architecture Client unique registry available on premises Security readiness guidance via the Vulnerability Advisor Build-in applications of popular open source projects and IBM enterprise products in App Store Access to the images and application you require to deploy containers that meet your business needs and strategy Open-source, standardized, lightweight, self sufficient container technology Balance workload between on-prem and off-prem Deployment choice with openPOWER and x86_64 Flexibility to choose on-prem and off- prem or mix for your business Build, ship, and run standardized containers Integrated monitoring & logging Elasticity to grow storage & container needs Integrated CI/CD flow Life-cycle management of containers and data volumes  Docker ease of use combined with enterprise-level integrity and confidence

Promoted to incubator, Sep, 2016 Where are we now? v0.2.0 ~ 0.2.2, 2014 v0.3.0 ~ 0.7.1, 2015 v0.7.2, Jan 30, 2016 Promoted to incubator, Sep, 2016 Re-org the code as a stand along project Removed un-used upstream code Sanity check with latest k8s version Updated CI script for daily build Enabling e2e test

No code dependency Get Pods Match Pods and Offers Bind Pods with Host Update Pods status Run Pods by kubelet

IBM’s contribution in Mesos Hierarchy Role (MESOS-6375) Multi-Role Frameworks (MESOS-1763) Scheduling (Pending) Resource Revocation Revocable resources by default Oversubscription for reservation Quota enhancements Optimistic Offers ….. Unified Container: Volume Unified Container: Network (CNI) Pods

(role = *, bigdata-daemons) Support Complex Business Container service BigData Service & Applications Hierarchy Role NS/Quota Spark Session Scheduler Myriad Masters Myriad Slaves (weight) Spark Slaves (weight) Dep 1 Dep 2 Dep 3 ns1 + quota1 ns2 + quota2 ns3 + quota3 Scheduling Kubernetes (role = *, bigdata-daemons) (role = bigdata-comute) Spark with kube-mesos Resource Requirement Mesos Resource Sharing App Area (label: app) BigData Area (label: bigdata) role = * Network/DNS Quota & Preemption role = bigdata-daemon: Reserve resources for HDFS and Yarn/Spark master role = bigdata-comute: Reserve resource for Yarn/Spark agents

Multi-Role & Hierarchy Role BigData ORG App others MR Team Spark Team Proj-1 Proj-2 50% Myriad BigData 40% Total Resources Spark 50% 60% Kubernetes NS-1 NS-2

Resources Requirements hostname=host1 hostname=host1 k8s Spark Myriad k8s Spark Myriad host ??? host1 host1 Mesos Mesos

Revocable Resources Support Mesos revocable resources KUbernetes Other Frameworks Support Mesos revocable resources Add a flag for revocable tasks/Pods Making scheduler to accept revocable offers Killing tasks when notified How to handle forceful eviction? Mesos Mesos Agent Mesos Agent …… Mesos Agent k8s-executor kubelet kubelet

What’s next?? … Support Sharing Policies & Smart Preemption: Revocable resources support (#19529) Scheduling enhancement (# 31068) Support Hierarchical Consumer: Namespace/Quota support/integrate (#31069) Multiple roles support Kube-DNS integrate with external DNS (# 28453) …

Roadmap of “Kuberntes on Mesos” v0.9 release new features v1.0 release Production Ready v0.8 release k8sm refactor v0.7 release new code base Nov, 2016 End of 2016 2017

Thank You !!