LinuxCon ContainerCon CloudOpen China June 19, 2017 Kubernetes Storage LinuxCon ContainerCon CloudOpen China June 19, 2017 Jing Xu - Software Engineer - Google https://github.com/jingxu97

Agenda Overview of Kubernetes High-level architecture API objects Kubernetes Storage System Concepts of Pod, PV/PVC, SC Three ways to setup and use storage New Features for Storage Local Storage and Capacity Isolation Snapshot, data replication Out-of-tree plugins and more ... Next get into the design and implementation, what are the components and how they work

K8s provides container-centric infrastructure Once specific containers are no longer bound to specific machines/VMs, host-centric infrastructure no longer works Scheduling: Decide where my containers should run Lifecycle and health: Keep my containers running despite failures Scaling: Make sets of containers bigger or smaller Naming and discovery: Find where my containers are now Load balancing: Distribute traffic across a set of containers Storage volumes: Provide data to containers Logging and monitoring: Track what’s happening with my containers Debugging and introspection: Enter or attach to containers Identity and authorization: Control who can do things to my containers K8s is an open source container management platform designed to run enterprise-class, cloud-enabled and web-scalable IT workloads. It follows master-worker achitecture model. Master decides where to run the containers, perform load balancing, keep track of each node status and other functions. On each node, we have deamon called kubelet running to take care of containers lifecycle. Kubernetes provides nice features such as self-healing mechanisms, such as auto-restarting, re-scheduling, and replicating containers then ensure this state is met.

K8s API Objects API Objects: Abstraction of system state apiVersion: v1 kind: Node spec: externalID: "3877693740656810196" podCIDR: 10.100.0.0/24 status: addresses: - address: 10.128.0.2 type: InternalIP - address: 104.198.141.183 type: ExternalIP capacity: cpu: "1" memory: 3794520Ki pods: "110" storage: 16310120Ki conditions: - lastHeartbeatTime: 2017-06-07T02:38:14Z message: RouteController created a route reason: RouteCreated status: "False" API Objects: Abstraction of system state Spec: desired state Status: current state Operations Create/Delete/Update/Get/List Basic Objects: Pod, Volume, Service, Namespace High-level abstractions (controllers): ReplicationSet, StatefulSet, DaemonSet, etc. Control Plane: Make cluster’s current state match the desired state One more thing we go into storage The REST API is the fundamental fabric of Kubernetes. All operations and communications between components are REST API calls handled by the API Server, including external user commands. Consequently, everything in the Kubernetes platform is treated as an API object and has a corresponding entry in the API. A Kubernetes object is a “record of intent”–once you create the object, the Kubernetes system will constantly work to ensure that that object exists. By creating an object, you’re effectively telling the Kubernetes system what you want your cluster’s workload to look like; this is your cluster’s desired state. API Object Example: Node

K8s Architecture users master nodes kubelet API apiserver etcd CLI UI scheduler kubelet controllers users master nodes

K8s API Objects: Pod Pod: small group of containers The atom of scheduling & placement Pod restarts container if it dies Shared namespace share IP address & localhost share IPC, etc. A group of Pods Distributed on different nodes E.g., ReplicaSet, StatefulSet, DaemonSet Content Manager Consumers File Puller Web Server Pod One more thing we go into storage The REST API is the fundamental fabric of Kubernetes. All operations and communications between components are REST API calls handled by the API Server, including external user commands. Consequently, everything in the Kubernetes platform is treated as an API object and has a corresponding entry in the API. 6

K8s API Objects: Pod Pod has problems to use storage apiVersion: v1 kind: Pod metadata: name: sleepypod spec: containers: - name: write-container image: gcr.io/google_containers/busybox command: - //write-data volumeMounts: - name: data mountPath: /data - name: read-container - //read-data pod.yaml Content Manager Consumers File Puller Web Server Pod Pod has problems to use storage Not sharable: Containers can’t share files Not persistent: Container crashes result in loss of data One more thing we go into storage The REST API is the fundamental fabric of Kubernetes. All operations and communications between components are REST API calls handled by the API Server, including external user commands. Consequently, everything in the Kubernetes platform is treated as an API object and has a corresponding entry in the API. $ kubectl create -f pod.yaml

K8s Storage System Volume A directory accessible to all the containers in a pod API objects that has explicit lifetime outlives containers Storage System Supports many types of volumes that can be used by a Pod simultaneously. Persistent, Ephemeral, local storage, networked attached etc. Make storage available for pods. Direct access Static provisioning Dynamic provisioning

K8s Supported Storage (Volume) Persistent (Networked) GCE Persistent Disk AWS Elastic Block Store Azure File Storage Azure Data Disk iSCSI Flocker NFS vSphere GlusterFS Ceph File and RBD Cinder Quobyte Volume FibreChannel VMWare Photon PD Portworx Dell EMC ScaleIO Ephemeral (local) Empty dir (and tmpfs) Expose K8s API Secret ConfigMap DownwardAPI Other Flex (exec a binary) Host path New Local Persistent Storage Life cycle

Direct Access “sleepypod” is created and scheduled on “node-A” apiVersion: v1 kind: Pod metadata: name: sleepypod spec: volumes: - name: data gcePersistentDisk: pdName: panda-disk fsType: ext4 containers: - name: sleepycontainer image: gcr.io/google_containers/busybox command: - sleep - "6000" volumeMounts: mountPath: /data readOnly: false User Pod node-A “sleepypod” is created and scheduled on “node-A” “panda-disk” is attached to “node-A” and mounted to pod’s /data directory

Issues with Direct Access Not portable Need to change pod spec when moving to a different cluster Not scalable Distribute storage information to all users Not secure Storage information is exposed to users Time consuming Need to coordinate between all users User Pod User User

Issues with Direct Access Separate Complexity Admin sets up storage User requests storage Cluster Admin Pod User

PersistentVolumes (PV) K8s: PV and PVC Separate Complexity Admin sets up storage User requests storage Admin: PersistentVolume (PV) Piece of networked storage in the cluster Lifecycle independent of any individual pod User: PersistentVolumeClaim (PVC) Claims request specific size and access modes of storage PersistentVolumes (PV) Cluster Admin PVClaim User

PersistentVolumes (PV) PV and PVC Binding: Automatically match between storage request and available resource Once bound, no other PVC could use that PV PersistentVolumes (PV) Cluster Admin PVClaim Using Pod uses PVC as volumes K8s automatically attach and mount volume to where pod is scheduled User node-A Pod PV/PVCs enable an abstraction between the consumption of storage and the implementation of storage. The PV object is a representation of a piece storage available on the cluster And a PVC is a request for storage on behalf of an application trying to run on the cluster. Let’s look at an example.

PV/PVC Example: Create PV apiVersion: v1 kind: PersistentVolume metadata: name: myPV1 spec: accessModes: - ReadWriteOnce capacity: storage: 10Gi persistentVolumeReclaimPolicy: Retain gcePersistentDisk: fsType: ext4 pdName: panda-disk1 apiVersion: v1 kind: PersistentVolume metadata: name: myPV2 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi persistentVolumeReclaimPolicy: Retain gcePersistentDisk: fsType: ext4 pdName: panda-disk2 pv.yaml First we have to expose the storage available for applications to use We do this by creating PV objects to represent each piece of storage available for applications to use on the cluster. Each PV object contains a description of the storage it exposes, and connection information. Normally cluster or storage administrators create these PV objects, not application developers.

PV/PVC Example: PVC and PV binding apiVersion: v1 kind: PersistentVolumeClaim metadata: name: mypvc namespace: testns spec: accessModes: - ReadWriteOnce resources: requests: storage: 100Gi $ kubectl create -f pv.yaml persistentvolume "pv1" created persistentvolume "pv2" created $ kubectl get pv NAME CAPACITY ACCESSMODES STATUS CLAIM pv1 10Gi RWO Available pv2 100Gi RWO Available $ kubectl create -f pvc.yaml persistentvolumeclaim "mypvc" created $ kubectl get pv NAME CAPACITY ACCESSMODES STATUS CLAIM pv1 10Gi RWO Available pv2 100Gi RWO Bound testns/mypvc Now if an application developer wants to use storage on this cluster, they must first create a request for storage. This is done by creating a PVC object. In the PVC object, you define the minimum requirements for the storage you need: the min disk capacity and required access modes

PVC/PV Example: Pod references volume apiVersion: v1 kind: Pod metadata: name: sleepypod spec: volumes: - name: data gcePersistentDisk: pdName: panda-disk fsType: ext4 containers: - name: sleepycontainer image: gcr.io/google_containers/busybox command: - sleep - "6000" volumeMounts: ... Not portable User does not to change pod spec when moving to a different cluster Not scalable No need to distribute storage information to users and coordinate among them Not secure User does not know any detailed information about storage volumes: - name: data persistentVolumeClaim: claimName: mypvc

PersistentVolumes (PV) Still has issues ... Admin is busy ... Need to pre-create and configure each individual piece of storage User is waiting ... For a new request, has to ask admin to config and create. PersistentVolumes (PV) Cluster Admin PVClaim PV/PVCs enable an abstraction between the consumption of storage and the implementation of storage. The PV object is a representation of a piece storage available on the cluster And a PVC is a request for storage on behalf of an application trying to run on the cluster. Let’s look at an example. User

K8s API: Storage Classes (SC) An API object created by admins to “enable” dynamic provisioning Allows storage (PVs) to be created on-demand in seconds Define the parameters used during volume creation. Type of the storage hdd, ssd. Credential information: keys. PersistentVolumes (PV) Storage Class Cluster Admin PVClaim PV/PVCs enable an abstraction between the consumption of storage and the implementation of storage. The PV object is a representation of a piece storage available on the cluster And a PVC is a request for storage on behalf of an application trying to run on the cluster. Let’s look at an example. User

Dynamic Provisioning and Storage Classes “Selecting” a storage class in PVC triggers dynamic provisioning Default StorageClass is pre-installed Admin and user does not need to do anything if default class is used! kind: StorageClass apiVersion: storage.k8s.io/v1 metadata: name: slow provisioner: kubernetes.io/gce-pd parameters: type: pd-standard -- name: fast type: pd-ssd apiVersion: v1 kind: PersistentVolumeClaim metadata: name: mypvc namespace: testns spec: accessModes: - ReadWriteOnce resources: requests: storage: 100Gi storageClassName: fast pvc.yaml (on k8s 1.6+) To use storage, just as before, the application developer creates a PVC object to request storage. In the PVC object, they now specify the storage class to use. In k8s 1.6, this is a new field in the PVC object. In earlier versions this was an annotation on the PVC object.

Dynamic Provisioning and Storage Classes K8s dynamically provisions storage and creates PV to represent it. Storage Provider $ kubectl create -f storage_class.yaml storageclass "fast" created $ kubectl create -f pvc.yaml persistentvolumeclaim "mypvc" created $ kubectl get pvc --all-namespaces NAMESPACE NAME STATUS VOLUME CAPACITY ACCESSMODES AGE testns mypvc Bound pvc-331d7407-fe18-11e6-b7cd-42010a8000cd 100Gi RWO 6s $ kubectl get pv pvc-331d7407-fe18-11e6-b7cd-42010a8000cd NAME CAPACITY ACCESSMODES RECLAIMPOLICY STATUS CLAIM REASON AGE pvc-331d7407-fe18-11e6-b7cd-42010a8000cd 100Gi RWO Delete Bound testns/mypvc 13m When a PVC object w/a storage class is created, instead of binding to existing available PVs, K8s calls the provisioner defined in the specified storage-class to create a new volume. In this case it calls the GCE PD Plugin which calls out to the GCE API to create a new SSD backed GCE PD Once the new storage volume is provisioned, k8s automatically creates a PV object to represent the new storage, and binds it to the user’s PVC. It’s important to note that dynamically provisioned volumes are created with a “delete” retention policy. That means once the PVC is deleted, the dynamically provisioned volume will automatically be deleted. If you don’t want that to happen you should update the PV object after it is automatically created and change the Retention Policy to “retain” instead of “delete”.

Local Persistent Storage Capacity Management Out-of-Tree​ ​Volume​ ​Drivers Snapshotting, Replication and more to come ...

Out-of-Tree Volume Drivers Before: in-tree plugins Volume plugins are compiled and ship along with kubernetes binaries Hard to maintain, force provider to open source code ... Now: Container Storage Interface Goal: industry wide standard for plugging storage systems into all major container orchestration (CO) systems. Working with Mesos, Cloud Foundry, and Docker to converge on an agreed on design.

Local Persistent Storage Before: Use hostPath to access local storage (a specific location in Pod spec) Issues of portability, accounting, security Now (Alpha feature): Accessing local storage is through local persistent volumes (PV) Discover and provision local PVs automatically Building block for high-performance and distributed apps

K8s Resource Management Resources: Measurable quantities that can be requested, allocated, and consumed. Node-level Capacity: all resources available for user pods and system daemons Allocatable: amount available for only user pods (exclude resources reserved for system daemons) Container-level Request: ensure enough resources for Pod Limit: get terminated if exceeding limit

Local Storage Capacity Isolation Before: No capacity isolation on Storage at all! Local storage is shared → Unlimited log writing cause disk full… Cannot reserve storage for system daemons. Now (Alpha feature): Allocatable storage: allow to reserve storage like CPU and memory EmptyDir isolation: set a limit on emptyDir, enforce limit by evicting Pods Container overlay isolation: set a limit on container overlay usage In Kubernetes, resources are measurable quantities that can be requested, allocated, and consumed. Request: If a pod is successfully scheduled, the container is guaranteed the amount of resources requested. Limit: The pods and its containers will not be allowed to exceed the specified limit

Coming soon Snapshot Provide support for user to backup data, run applications with pre-populate data and recover from failures Quota management Provides rich policies and improve resource usage Logging Better logging infrastructure and storage usage management And more ...

Summary K8s storage management supports variety of volume plugins hides the complexity of managing makes your work portable and scalable provides more and more cool features ...

Kubernetes is Open open community open design Try it! Contribute! open community open design open source open to ideas https://kubernetes.io Code: github.com/kubernetes/kubernetes Chat: slack.k8s.io Twitter: @kubernetesio

Happy Kubernetes Team