Kubernetes intro

How can we simplify PerfSonar management? Need to run some software on a set of nodes Config is repeated on all nodes Want to autodiscover new nodes New nodes info should be added to config file and propagated Containers + Container orchestration system

Kubernetes definition Kubernetes is ancient Greek for "Helmsman". Root of the word "Governor", "Cybernetics". Kubernetes is a "Container Orchestrator" or "Cluster Manager". Places containers on nodes Recovers automatically from failure Basic monitoring, logging, health checking Enables containers to find each other.

Google grown Based on ideas proven at Google over 10 years Google launches 2 billion containers per week. Part of a larger set of tools that make up the internal Google platform. Everything at google runs in containers

Linux containers

Docker container: layers, union filesystem Docker images are read-only templates from which Docker containers are launched. Only top layer is writeable Copy on write Only changed layers need to be transferred Natively supports ZFS for storing layers (zvol)

We need more than just packing and isolation Scheduling: where should my container run? Lifecycle and health: Keep my containers running despite failures Discovery: Where are my containers now? Monitoring: What’s happening with my containers? Auth{n,z}: Control who can do things to my containers Aggregates: Compose sets of containers into jobs Scaling: Making jobs bigger or smaller Provisioning storage: block volumes, shared FS

PODs Pods are the basic building block of Kubernetes Pods are the smallest deployable units that are created and managed in Kubernetes A Pod is a group of one or more deeply coupled containers Shared network Shared storage Options to run the containers Each pod is assigned a unique, DYNAMIC IP address

Service An interface to a set of replicated pods, which acts as a load-balancer Provides an abstraction layer No need to worry about containers location.

Other units ReplicaSets: Ensures that the number of desired pods "replicas" are running at any time. Deployments: Declarative way to describe the desired state of the application (pods, replica sets). Easy version updates for any software DaemonSets: runs a POD on every node in a cluster StatefulSets: clustered applications Startup/shutdown ordering Stable hostname and storage Clustered MySQL, Zookeeper, Elasticsearch, Cassandra Volume plugins: ConfigMaps, Secrets, HostPath, rook (ceph), nfs, iscsi, gitRepo, …

Networking The pods are scheduled on a flat shared network across all nodes. Each pod can communicate without proxies and translations (NAT) with other pods within the cluster. Several implementations are available: Flannel, Weave, OpenVswitch, Calico, kube-router Kube-router talks BGP Can assign AS to groups of nodes Can export routing rules Supports ExternalIP for services Automatically connects remote nodes with TCP tunnels – doesn’t have to be in same subnet

Our approach to PerfSonar install PerfSonar Master Archive - a Deployment 1 POD, keep it alive, restart if necessary TestPoints - a DaemonSet Runs 1 copy of POD on every node in a cluster Esmond - a Service Points to currently running PerfSonar container Cassandra DB - a StatefulSet Nodes have their stable names, can elect a master node PostgreSQL DB - inside the PerfSonar POD Required, can’t separate

Cluster autoconfig List of testpoints Kubernetes API MeshConfig Schedules testpoint PODs on all nodes in the cluster Config JSON TestPoint PerfSonar Central MA Measurement results TestPoint TestPoint