1. Kubernetes

2. How can we simplify PerfSonar management?
Need to run some software on a set of nodes
Config is repeated on all nodes
Want to autodiscover new nodes
New nodes info should be added to config file and propagated
Containers +
Container orchestration system

3. Kubernetes definition
Kubernetes is ancient Greek for "Helmsman". Root of the word "Governor", "Cybernetics".
Kubernetes is a "Container Orchestrator" or "Cluster Manager".
Places containers on nodes
Recovers automatically from failure
Basic monitoring, logging, health checking
Enables containers to find each other.

4. Google grown Based on ideas proven at Google over 10 years
Google launches 2 billion containers per week.
Part of a larger set of tools that make up the internal Google platform.
Everything at google runs in containers

5. Linux containers

6. Docker container: layers, union filesystem
Docker images are read-only templates from which Docker containers are launched.
Only top layer is writeable
Copy on write
Only changed layers need to be transferred
Natively supports ZFS for storing layers (zvol)

7. We need more than just packing and isolation
Scheduling: where should my container run?
Lifecycle and health: Keep my containers running despite failures
Discovery: Where are my containers now?
Monitoring: What’s happening with my containers?
Auth{n,z}: Control who can do things to my containers
Aggregates: Compose sets of containers into jobs
Scaling: Making jobs bigger or smaller
Provisioning storage: block volumes, shared FS

8. PODs Pods are the basic building block of Kubernetes
Pods are the smallest deployable units that are created and managed in Kubernetes
A Pod is a group of one or more deeply coupled containers
Shared network
Shared storage
Options to run the containers
Each pod is assigned a unique, DYNAMIC IP address

9. Service
An interface to a set of replicated pods, which acts as a load-balancer
Provides an abstraction layer
No need to worry about containers location.

10. Other units
ReplicaSets: Ensures that the number of desired pods "replicas" are running at any time.
Deployments: Declarative way to describe the desired state of the application (pods, replica sets).
Easy version updates for any software
DaemonSets: runs a POD on every node in a cluster
StatefulSets: clustered applications
Startup/shutdown ordering
Stable hostname and storage
Clustered MySQL, Zookeeper, Elasticsearch, Cassandra
Volume plugins: ConfigMaps, Secrets, HostPath, rook (ceph), nfs, iscsi, gitRepo, …

11. Networking
The pods are scheduled on a flat shared network across all nodes.
Each pod can communicate without proxies and translations (NAT) with other pods within the cluster.
Several implementations are available: Flannel, Weave, OpenVswitch, Calico, kube-router
Kube-router talks BGP
Can assign AS to groups of nodes
Can export routing rules
Supports ExternalIP for services
Automatically connects remote nodes with TCP tunnels – doesn’t have to be in same subnet

12. Our approach to PerfSonar install
PerfSonar Master Archive - a Deployment
1 POD, keep it alive, restart if necessary
TestPoints - a StatefulSet
Runs 1 copy of POD on every node in a cluster
Esmond - a Service
Points to currently running PerfSonar container
Cassandra DB - a StatefulSet
Nodes have their stable names, can elect a master node
PostgreSQL DB - inside the PerfSonar POD
Required, can’t separate

13. Cluster autoconfig List of testpoints Kubernetes API MeshConfig
Schedules testpoint PODs on all nodes in the cluster
Config JSON
TestPoint
PerfSonar Central MA
Measurement results
TestPoint
TestPoint