1. Not Just Another Mesos Framework
Kubernetes on Mesos
Not Just Another Mesos Framework
Da Ma, Software Architect, kube-mesos owner, IBM
Yong Feng, Senior Software Architect, IBM
Guangya Liu, Senior Software Engineer, IBM

2. Why “Kubernetes on Mesos”?
Container Runtime
The container runtime packages and launches application instance in a sandbox with portable and flexible capability.
Docker and rkt are container runtimes.
Computing, Storage and Network
Application
Application
Resource Management
The resource management component provides an abstraction of resources (cpu, mem, …) for application and then allocates/provision resources among tenants and applications.
Mesos is an open source resource manager.
EGO is an IBM enterprise resource manager.
Workload Management
The workload management component manages the life cycle of an application as well as access to the application, including service compose, service discovery, load balance
Kubernetes and Marathon are workload managers.

3. Shared state between fws by Optimistic Offer
Why “Kubernetes on Mesos”?
Shared state
Two Level Scheduling ??
Shared state between fws by Optimistic Offer ……
Monolithic
PBS/SGE/LSF
Mesos/YARN/EGO
Res mgr & wld mgr
tightly coupled
Batch wld only
1992
2003
2016
future

4. “Kubernetes on Mesos” Organization
Sponsor: Tim Hockin (Google)
Champion: David Eads (Redhat)
Owner: Klaus Ma (IBM)
Github: kuberntes-incubator/kube-mesos-framework

5. Spectrum Conductor with
IBM is Owner & User !!
Watson/Cognitive
Container Cloud
Spectrum Conductor with
Spark
IBM Spectrum Conductor for Container
GUI
Workflow
App Store
Authentication
Authorization
k8s
Session Scheduler
HPC
Persistent Volume
Service Discovery
Installer
(Deploy, Reconfigure, HA, Scale, Rolling update)
Mesos Master
Distributed Key-value Store
Service Load Balance
Mesos Agent
Trouble-shooting
Monitor
K8s executor
container
container
pod
pod
pod
container
container
Network Topology
Image registry

6. Community Value IBM Value-add Customer Value
Docker Hub Registry holds a repository of Docker images
Lots of application integrated with Mesos
Kubernetes enable micro-service architecture
Client unique registry available on premises
Security readiness guidance via the Vulnerability Advisor
Build-in applications of popular open source projects and IBM enterprise products in App Store
Access to the images and application you require to deploy containers that meet your business needs and strategy
Open-source, standardized, lightweight, self sufficient container technology
Balance workload between on-prem and off-prem
Deployment choice with openPOWER and x86_64
Flexibility to choose on-prem and off- prem or mix for your business
Build, ship, and run standardized containers
Integrated monitoring & logging
Elasticity to grow storage & container needs
Integrated CI/CD flow
Life-cycle management of containers and data volumes 
Docker ease of use combined with enterprise-level integrity and confidence

7. Promoted to incubator, Sep, 2016
Where are we now?
v0.2.0 ~ 0.2.2, 2014
v0.3.0 ~ 0.7.1, 2015
v0.7.2, Jan 30, 2016
Promoted to incubator, Sep, 2016
Re-org the code as a stand along project
Removed un-used upstream code
Sanity check with latest k8s version
Updated CI script for daily build
Enabling e2e test

8. No code dependency Get Pods Match Pods and Offers Bind Pods with Host
Update Pods status
Run Pods by kubelet

9. IBM’s contribution in Mesos
Hierarchy Role (MESOS-6375)
Multi-Role Frameworks (MESOS-1763)
Scheduling (Pending)
Resource Revocation
Revocable resources by default
Oversubscription for reservation
Quota enhancements
Optimistic Offers
…..
Unified Container: Volume
Unified Container: Network (CNI)
Pods

10. (role = *, bigdata-daemons)
Support Complex Business
Container service
BigData Service & Applications
Hierarchy Role
NS/Quota
Spark Session
Scheduler
Myriad Masters
Myriad
Slaves
(weight)
Spark
Slaves
(weight)
Dep 1
Dep 2
Dep 3
ns1 +
quota1
ns2 +
quota2
ns3 +
quota3
Scheduling
Kubernetes
(role = *, bigdata-daemons)
(role = bigdata-comute)
Spark with
kube-mesos
Resource Requirement
Mesos
Resource
Sharing
App Area (label: app)
BigData Area (label: bigdata)
role = *
Network/DNS
Quota
&
Preemption
role = bigdata-daemon: Reserve resources for HDFS and Yarn/Spark master
role = bigdata-comute: Reserve resource for Yarn/Spark agents

11. Multi-Role & Hierarchy Role
BigData
ORG
App
others
MR Team
Spark Team
Proj-1
Proj-2
50%
Myriad
BigData
40%
Total Resources
Spark
50%
60%
Kubernetes
NS-1
NS-2

12. Resources Requirements
hostname=host1
hostname=host1
k8s
Spark
Myriad
k8s
Spark
Myriad
host ???
host1
host1
Mesos
Mesos

13. Revocable Resources Support Mesos revocable resources KUbernetes
Other Frameworks
Support Mesos revocable resources
Add a flag for revocable tasks/Pods
Making scheduler to accept revocable offers
Killing tasks when notified
How to handle forceful eviction?
Mesos
Mesos Agent
Mesos Agent ……
Mesos Agent
k8s-executor
kubelet
kubelet

14. What’s next?? … Support Sharing Policies & Smart Preemption:
Revocable resources support (#19529)
Scheduling enhancement (# 31068)
Support Hierarchical Consumer:
Namespace/Quota support/integrate (#31069)
Multiple roles support
Kube-DNS integrate with external DNS (# 28453) …

15. Roadmap of “Kuberntes on Mesos”
v0.9 release new features
v1.0 release
Production Ready
v0.8 release
k8sm refactor
v0.7 release new code base
Nov, 2016
End of 2016
2017

16. Thank You !!