1. LinuxCon ContainerCon CloudOpen China June 19, 2017
Kubernetes Storage
LinuxCon ContainerCon CloudOpen China
June 19, 2017
Jing Xu - Software Engineer - Google

2. Agenda Overview of Kubernetes High-level architecture API objects
Kubernetes Storage System
Concepts of Pod, PV/PVC, SC
Three ways to setup and use storage
New Features for Storage
Local Storage and Capacity Isolation
Snapshot, data replication
Out-of-tree plugins and more ...
Next get into the design and implementation, what are the components and how they work

3. K8s provides container-centric infrastructure
Once specific containers are no longer bound to specific machines/VMs, host-centric infrastructure no longer works
Scheduling: Decide where my containers should run
Lifecycle and health: Keep my containers running despite failures
Scaling: Make sets of containers bigger or smaller
Naming and discovery: Find where my containers are now
Load balancing: Distribute traffic across a set of containers
Storage volumes: Provide data to containers
Logging and monitoring: Track what’s happening with my containers
Debugging and introspection: Enter or attach to containers
Identity and authorization: Control who can do things to my containers
K8s is an open source container management platform designed to run enterprise-class, cloud-enabled and web-scalable IT workloads.
It follows master-worker achitecture model. Master decides where to run the containers, perform load balancing, keep track of each node status and other functions. On each node, we have deamon called kubelet running to take care of containers lifecycle. Kubernetes provides nice features such as self-healing mechanisms, such as auto-restarting, re-scheduling, and replicating containers then ensure this state is met.

4. K8s API Objects API Objects: Abstraction of system state
apiVersion: v1
kind: Node
spec:
externalID: " "
podCIDR: /24
status:
addresses:
- address:
type: InternalIP
- address:
type: ExternalIP
capacity:
cpu: "1"
memory: Ki
pods: "110"
storage: Ki
conditions:
- lastHeartbeatTime: T02:38:14Z
message: RouteController created a route
reason: RouteCreated
status: "False"
API Objects:
Abstraction of system state
Spec: desired state
Status: current state
Operations
Create/Delete/Update/Get/List
Basic Objects:
Pod, Volume, Service, Namespace
High-level abstractions (controllers):
ReplicationSet, StatefulSet, DaemonSet, etc.
Control Plane:
Make cluster’s current state match the desired state
One more thing we go into storage
The REST API is the fundamental fabric of Kubernetes. All operations and communications between components are REST API calls handled by the API Server, including external user commands. Consequently, everything in the Kubernetes platform is treated as an API object and has a corresponding entry in the API.
A Kubernetes object is a “record of intent”–once you create the object, the Kubernetes system will constantly work to ensure that that object exists. By creating an object, you’re effectively telling the Kubernetes system what you want your cluster’s workload to look like; this is your cluster’s desired state.
API Object Example: Node

5. K8s Architecture users master nodes kubelet API apiserver etcd CLIUI
scheduler
kubelet
controllers
users
master
nodes

6. K8s API Objects: Pod Pod: small group of containers
The atom of scheduling & placement
Pod restarts container if it dies
Shared namespace
share IP address & localhost
share IPC, etc.
A group of Pods
Distributed on different nodes
E.g., ReplicaSet, StatefulSet, DaemonSet
Content Manager
Consumers
File
Puller
Web
Server
Pod
One more thing we go into storage
The REST API is the fundamental fabric of Kubernetes. All operations and communications between components are REST API calls handled by the API Server, including external user commands. Consequently, everything in the Kubernetes platform is treated as an API object and has a corresponding entry in the API. 6

7. K8s API Objects: Pod Pod has problems to use storage
apiVersion: v1
kind: Pod
metadata:
name: sleepypod
spec:
containers:
- name: write-container
image: gcr.io/google_containers/busybox
command:
- //write-data
volumeMounts:
- name: data
mountPath: /data
- name: read-container
- //read-data
pod.yaml
Content Manager
Consumers
File
Puller
Web
Server
Pod
Pod has problems to use storage
Not sharable: Containers can’t share files
Not persistent: Container crashes result in loss of data
One more thing we go into storage
The REST API is the fundamental fabric of Kubernetes. All operations and communications between components are REST API calls handled by the API Server, including external user commands. Consequently, everything in the Kubernetes platform is treated as an API object and has a corresponding entry in the API.
$ kubectl create -f pod.yaml

8. K8s Storage System Volume
A directory accessible to all the containers in a pod
API objects that has explicit lifetime outlives containers
Storage System
Supports many types of volumes that can be used by a Pod simultaneously.
Persistent, Ephemeral, local storage, networked attached etc.
Make storage available for pods.
Direct access
Static provisioning
Dynamic provisioning

9. K8s Supported Storage (Volume)
Persistent (Networked)
GCE Persistent Disk
AWS Elastic Block Store
Azure File Storage
Azure Data Disk
iSCSI
Flocker
NFS
vSphere
GlusterFS
Ceph File and RBD
Cinder
Quobyte Volume
FibreChannel
VMWare Photon PD
Portworx
Dell EMC ScaleIO
Ephemeral (local)
Empty dir (and tmpfs)
Expose K8s API
Secret
ConfigMap
DownwardAPI
Other
Flex (exec a binary)
Host path
New
Local Persistent Storage
Life cycle

10. Direct Access “sleepypod” is created and scheduled on “node-A”
apiVersion: v1
kind: Pod
metadata:
name: sleepypod
spec:
volumes:
- name: data
gcePersistentDisk:
pdName: panda-disk
fsType: ext4
containers:
- name: sleepycontainer
image: gcr.io/google_containers/busybox
command:
- sleep
- "6000"
volumeMounts:
mountPath: /data
readOnly: false
User
Pod
node-A
“sleepypod” is created and scheduled on “node-A”
“panda-disk” is attached to “node-A” and mounted to pod’s /data directory

11. Issues with Direct Access
Not portable
Need to change pod spec when moving to a different cluster
Not scalable
Distribute storage information to all users
Not secure
Storage information is exposed to users
Time consuming
Need to coordinate between all users
User
Pod
User
User

12. Issues with Direct Access
Separate Complexity
Admin sets up storage
User requests storage
Cluster Admin
Pod
User

13. PersistentVolumes (PV)
K8s: PV and PVC
Separate Complexity
Admin sets up storage
User requests storage
Admin: PersistentVolume (PV)
Piece of networked storage in the cluster
Lifecycle independent of any individual pod
User: PersistentVolumeClaim (PVC)
Claims request specific size and access modes of storage
PersistentVolumes (PV)
Cluster Admin
PVClaim
User

14. PersistentVolumes (PV)
PV and PVC
Binding:
Automatically match between storage request and available resource
Once bound, no other PVC could use that PV
PersistentVolumes (PV)
Cluster Admin
PVClaim
Using
Pod uses PVC as volumes
K8s automatically attach and mount volume to where pod is scheduled
User
node-A
Pod
PV/PVCs enable an abstraction between the consumption of storage and the implementation of storage.
The PV object is a representation of a piece storage available on the cluster
And a PVC is a request for storage on behalf of an application trying to run on the cluster.
Let’s look at an example.

15. PV/PVC Example: Create PV
apiVersion: v1
kind: PersistentVolume
metadata:
name: myPV1
spec:
accessModes:
- ReadWriteOnce
capacity:
storage: 10Gi
persistentVolumeReclaimPolicy: Retain
gcePersistentDisk:
fsType: ext4
pdName: panda-disk1
apiVersion: v1
kind: PersistentVolume
metadata:
name: myPV2
spec:
accessModes:
- ReadWriteOnce
capacity:
storage: 100Gi
persistentVolumeReclaimPolicy: Retain
gcePersistentDisk:
fsType: ext4
pdName: panda-disk2
pv.yaml
First we have to expose the storage available for applications to use
We do this by creating PV objects to represent each piece of storage available for applications to use on the cluster.
Each PV object contains a description of the storage it exposes, and connection information.
Normally cluster or storage administrators create these PV objects, not application developers.

16. PV/PVC Example: PVC and PV binding
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: mypvc
namespace: testns
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 100Gi
$ kubectl create -f pv.yaml
persistentvolume "pv1" created
persistentvolume "pv2" created
$ kubectl get pv
NAME CAPACITY ACCESSMODES STATUS CLAIM
pv Gi RWO Available
pv Gi RWO Available
$ kubectl create -f pvc.yaml
persistentvolumeclaim "mypvc" created
$ kubectl get pv
NAME CAPACITY ACCESSMODES STATUS CLAIM
pv Gi RWO Available
pv Gi RWO Bound testns/mypvc
Now if an application developer wants to use storage on this cluster, they must first create a request for storage.
This is done by creating a PVC object.
In the PVC object, you define the minimum requirements for the storage you need: the min disk capacity and required access modes

17. PVC/PV Example: Pod references volume
apiVersion: v1
kind: Pod
metadata:
name: sleepypod
spec:
volumes:
- name: data
gcePersistentDisk:
pdName: panda-disk
fsType: ext4
containers:
- name: sleepycontainer
image: gcr.io/google_containers/busybox
command:
- sleep
- "6000"
volumeMounts:
...
Not portable
User does not to change pod spec when moving to a different cluster
Not scalable
No need to distribute storage information to users and coordinate among them
Not secure
User does not know any detailed information about storage
volumes:
- name: data
persistentVolumeClaim:
claimName: mypvc

18. PersistentVolumes (PV)
Still has issues ...
Admin is busy ...
Need to pre-create and configure each individual piece of storage
User is waiting ...
For a new request, has to ask admin to config and create.
PersistentVolumes (PV)
Cluster Admin
PVClaim
PV/PVCs enable an abstraction between the consumption of storage and the implementation of storage.
The PV object is a representation of a piece storage available on the cluster
And a PVC is a request for storage on behalf of an application trying to run on the cluster.
Let’s look at an example.
User

19. K8s API: Storage Classes (SC)
An API object created by admins to “enable” dynamic provisioning
Allows storage (PVs) to be created on-demand in seconds
Define the parameters used during volume creation.
Type of the storage
hdd, ssd.
Credential information: keys.
PersistentVolumes (PV)
Storage Class
Cluster Admin
PVClaim
PV/PVCs enable an abstraction between the consumption of storage and the implementation of storage.
The PV object is a representation of a piece storage available on the cluster
And a PVC is a request for storage on behalf of an application trying to run on the cluster.
Let’s look at an example.
User

20. Dynamic Provisioning and Storage Classes
“Selecting” a storage class in PVC triggers dynamic provisioning
Default StorageClass is pre-installed
Admin and user does not need to do anything if default class is used!
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: slow
provisioner: kubernetes.io/gce-pd
parameters:
type: pd-standard --
name: fast
type: pd-ssd
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: mypvc
namespace: testns
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 100Gi
storageClassName: fast
pvc.yaml (on k8s 1.6+)
To use storage, just as before, the application developer creates a PVC object to request storage.
In the PVC object, they now specify the storage class to use.
In k8s 1.6, this is a new field in the PVC object. In earlier versions this was an annotation on the PVC object.

21. Dynamic Provisioning and Storage Classes
K8s dynamically provisions storage and creates PV to represent it.
Storage Provider
$ kubectl create -f storage_class.yaml
storageclass "fast" created
$ kubectl create -f pvc.yaml
persistentvolumeclaim "mypvc" created
$ kubectl get pvc --all-namespaces
NAMESPACE NAME STATUS VOLUME CAPACITY ACCESSMODES AGE
testns mypvc Bound pvc-331d7407-fe18-11e6-b7cd-42010a8000cd 100Gi RWO s
$ kubectl get pv pvc-331d7407-fe18-11e6-b7cd-42010a8000cd
NAME CAPACITY ACCESSMODES RECLAIMPOLICY STATUS CLAIM REASON AGE
pvc-331d7407-fe18-11e6-b7cd-42010a8000cd 100Gi RWO Delete Bound testns/mypvc m
When a PVC object w/a storage class is created, instead of binding to existing available PVs,
K8s calls the provisioner defined in the specified storage-class to create a new volume.
In this case it calls the GCE PD Plugin which calls out to the GCE API to create a new SSD backed GCE PD
Once the new storage volume is provisioned, k8s automatically creates a PV object to represent the new storage, and binds it to the user’s PVC.
It’s important to note that dynamically provisioned volumes are created with a “delete” retention policy. That means once the PVC is deleted, the dynamically provisioned volume will automatically be deleted.
If you don’t want that to happen you should update the PV object after it is automatically created and change the Retention Policy to “retain” instead of “delete”.

22. Local Persistent Storage
Capacity Management
Out-of-Tree​ ​Volume​ ​Drivers
Snapshotting, Replication
and more to come ...

23. Out-of-Tree Volume Drivers
Before: in-tree plugins
Volume plugins are compiled and ship along with kubernetes binaries
Hard to maintain, force provider to open source code ...
Now: Container Storage Interface
Goal: industry wide standard for plugging storage systems into all major container orchestration (CO) systems.
Working with Mesos, Cloud Foundry, and Docker to converge on an agreed on design.

24. Local Persistent Storage
Before:
Use hostPath to access local storage (a specific location in Pod spec)
Issues of portability, accounting, security
Now (Alpha feature):
Accessing local storage is through local persistent volumes (PV)
Discover and provision local PVs automatically
Building block for high-performance and distributed apps

25. K8s Resource Management
Resources:
Measurable quantities that can be requested, allocated, and consumed.
Node-level
Capacity: all resources available for user pods and system daemons
Allocatable: amount available for only user pods (exclude resources reserved for system daemons)
Container-level
Request: ensure enough resources for Pod
Limit: get terminated if exceeding limit

26. Local Storage Capacity Isolation
Before: No capacity isolation on Storage at all!
Local storage is shared → Unlimited log writing cause disk full…
Cannot reserve storage for system daemons.
Now (Alpha feature):
Allocatable storage: allow to reserve storage like CPU and memory
EmptyDir isolation: set a limit on emptyDir, enforce limit by evicting Pods
Container overlay isolation: set a limit on container overlay usage
In Kubernetes, resources are measurable quantities that can be requested, allocated, and consumed.
Request: If a pod is successfully scheduled, the container is guaranteed the amount of resources requested.
Limit: The pods and its containers will not be allowed to exceed the specified limit

27. Coming soon
Snapshot
Provide support for user to backup data, run applications with pre-populate data and recover from failures
Quota management
Provides rich policies and improve resource usage
Logging
Better logging infrastructure and storage usage management
And more ...

28. Summary K8s storage management supports variety of volume plugins
hides the complexity of managing
makes your work portable and scalable
provides more and more cool features ...

29. Kubernetes is Open open community open design
Try it!
Contribute!
open community
open design
open source
open to ideas
Code: github.com/kubernetes/kubernetes
Chat: slack.k8s.io

30. Happy Kubernetes Team