Choosing the containerized cloud provisioning tool that best suits your need Good morning everyone & welcome to the session today as we are going to talk about "Choosing the container cloud provisioning tool that best suits your need". Before we get started on this, let me quickly introduce myself, I am Pradeep Kumar and I work as a Solution Architect at Ericsson India. I also part of the AT&T's Network Cloud project built on Airship and OpenStack. I also would want to mention about my colleague Soumitra, who unfortunately could not make it to the session today due to personal reasons, and also works on the AT&T’s Network cloud initiative. Smruti Soumitra Khuntia, Solution Architect Pradeep Kumar K S, Solution Architect Hemanth Kumar Nakkina, Solution Architect Ericsson

Agenda Why containers for cloud provisioning? 2018-02-21 Agenda Why containers for cloud provisioning? Containerized OpenStack deployment Key attributes of Containerized Deployments Airship overview Kolla overview TripleO overview Comparison Pradeep

Why containers for cloud provisioning? 2018-02-21 Why containers for cloud provisioning? Microservices help to decompose complex operations into smaller composable parts Quicker to bring them up and down Containers can scale, upgrade and monitored independently Increasing efficiency, reliability and scalability Opensource and thus fosters innovation and rapid development Pradeep

Containerised OpenStack deployment 2018-02-21 Containerised OpenStack deployment Run either Openstack-helm which uses Helm to install OpenStack on Kubernetes. It is a full lifecycle management solution that lets users easily deploy and manage individual OpenStack components or a full stack or Kolla-Ansible or TripleO, which are a set of ansible playbooks used to deploy the complete set of all the OpenStack services on containers.

Key features of Containerized Cloud Provisioning tools 2018-02-21 Key features of Containerized Cloud Provisioning tools Container technologies/tools Deployment strategies Infrastructure Configurations Logging, Monitoring, Health & Auditing Bare metal provisioning Container Networking Security Storage Support Day 2 Operations & Upgrades Pradeep

Containerized Cloud Deployment Solutions 2018-02-21 Containerized Cloud Deployment Solutions Pradeep

Introduction to Airship 2017-11-06 Introduction to Airship Airship is a collection of loosely coupled but interoperable open source tools that declaratively automate cloud provisioning. Airship manages the full lifecycle of data center infrastructure to deliver a production-grade Kubernetes cluster with Helm deployed artifacts, including OpenStack-Helm from raw bare metal infrastructure.   Airship allows operators to manage their infrastructure deployments and lifecycle through the declarative YAML documents that describe an Airship environment. Airship provides the flexibilities of container based infrastructure. Containers and Helm charts are the basic unit of deployment for all software including Airship itself, pushing software orchestration logic to the edge. Expanding the software stack is as simple as adding new charts to Airship declarations. Airship is simple to operate. Infrastructure is managed through declarative YAML files and there is one workflow for both deployments and updates. Airship does not require operators to develop their own set of complex orchestration tooling to automate Airship. Airship provides resiliency advantage. All jobs and services are running as containers, provide health status, and are healed by Kubernetes supervision by taking full advantage of native Kubernetes resiliency. Airship is self-hosted. The Airship components themselves are deployed as Helm charts and run as services within Kubernetes. This allows them to be upgraded like any other software component in the system. © Ericsson AB 2017

2017-11-06 Architecture Deckhand – Layering, substitution, parent-child relation of documents Armada – Tiller, define chart groups – sequence/parallel execution of charts, timeouts. Ex: ceph/ucp, keystone/mariadb/ceph … chaos Drydock – Declarative definition of baremetal hosts- bios configuration, raid configuration. Host os nic configuration bond etc, os disks Uses MAAS .. Future cluster api metalkube ironic Shipyard – Acyclic directed graph, uses airflow, community argo workflow © Ericsson AB 2017

2017-11-06 Airship Workflow © Ericsson AB 2017

Airship Workflow 4) Manually provision ”genesis” node 2017-11-06 Airship Workflow 4) Manually provision ”genesis” node central site site 1) Specify all info in YAML files Application templates Openstack helm charts Kubernetes helm charts Security Generate Passphrases, certificates PKI structure Site info Node hardware Disks, storage (ceph) setup Ports and VLANs (Oob/PXE/OAM/Data) site 5) Run the seed script Self hosted Kubernetes cluster Install all undercloud components - Bare metal deployer Install applications - Openstack control plane - (Other applications) Transform node to first cluster controller Genesis/ Controller1 Controller2 ~350 Airship supports strategies where we can define to Deploy all controllers first and if 80% are successful then proceed with deployment of parallel 10 worker nodes Keytakeaways: Declarative YAML definitions for your site for a smaller change. Kind of control the DE has to define your site using declarative YAMLs. Diff of changes in deckhand If you want to bring in new service/component, its just change of YAML manifests provided if you have helm chart. Infrastructure as Config. Process for your lifecycle management stays same. Controller3 3) Generate seed script 2) Combine & validate Controller4 Worker site.yaml Genesis.sh (40MB) 2) Combine & validate Worker © Ericsson AB 2017

Kolla 2018-02-21 Pre-reqs: We need ansible on the control node Python and pip to run ansible and kolla Security modules for ansible such as gcc, libssl, libffi Passwordless ssh configured between the nodes Kolla images

Kolla Consists of three main projects Kolla-OpenStack (Kolla) 2018-02-21 Kolla Consists of three main projects Kolla-OpenStack (Kolla) Provides Dockerfiles for each service and each process in the services Container external configurations Kolla Images Ansible code to build those containers Lets you build your own images on your own private registry or use public images Kolla-Ansible Kolla provides production-ready containers and deployment tools Ansible code to deploy, start, configure and re-configure/upgrade a system Allows for complete customization Uses lightweight Docker containers to deploy OpenStack services using Ansible. Kolla provides production-ready containers and deployment tools for operating OpenStack clouds that are scalable, fast, reliable and upgradable using community best practices. Kolla-Ansible allows for complete customization which permits operators to deploy OpenStack quickly even with minimal experience and modify OpenStack configurations to suit the operator’s requirements

Kolla images build process

Kolla Deployment Model/Workflow Setup the baseline OS Get OS Installed Install required tools Clone or pip install the Kolla tools Configure and install Docker Ensure shared mounts are enabled Configure an insecure registry or pass TLS certificates information for local registry

Kolla Deployment Model/Workflow Setup globals.yml and passwords.yml files Globals is a way to turn off and on services Define network and storage interactions with the underlying system. Passwords defines default passwords, keys, UUIDs etc. Can be generated automatically using kolla-genpwd tool Setup multi node inventory Kolla-ansible bootstrap-servers –i inventory_file Kolla-ansible deploy –i inventory_file

TripleO Deployment Overview 2017-11-06 TripleO Deployment Overview Deploys, Updates Monitors Undercloud OpenStack TripleO Overcloud OpenStack Deployment and Management Cloud Knowns as Undercloud Responsible for deployment of overcloud Provide infrastructure command and control Only applicable for cloud operator Conducts all lifecycle management Tenant Facing Cloud Knowns as Overcloud The cloud responsible to run tenant workloads Visible to Tenant administrator Provides cloud resources to run tenant Apps/VNFs. © Ericsson AB 2017

Physical View

TripleO Deployment Flow

Components of Containerized TripleO Pre-Built Docker Images Kolla Configuration Generation docker-puppet.py Container management Paunch Updates and Upgrades Ansible with TripleO Heat templates

Config generation for containerized services 2018-02-21 Config generation for containerized services docker-puppet.json Generates configs Bootstrap container Puppet docker-puppet.py Docker-puppet.py generates configs for each service and available through puppet hiera

Kolla Start Kolla start is responsible for: 2018-02-21 Kolla Start Kolla start is responsible for: Copying configuration files Setting permissions Starting the Service process config.json Service Container Kolla Start copy config set permission Key takeaways: TripleO follows the reference architecture of TripleO Vm based deployments which is pretty much proven one. Converts all system services to container based services. Uses light weight container management tools instead of kubernetes paunch Podman will be new tools for container management which supports OCI compatible images on kubelet, TripleO is going towards kubernetes in a step based manner ensuring smooth upgrades Bringing a new service needs modifying tripleo-heat-template projects – ansible scripts Config dir Start Process

Comparisons Project Container Type Supported Containers Deployment Type Kolla-Ansible Docker Kolla Containers Ansible TripleO Airship LOCI Containers Kubernetes and Helm

Container Technologies/Tools 2018-02-21 Container Technologies/Tools Airship Uses Kubernetes Helm charts ( from OpenStack Helm ) Loci for image builds TripleO Paunch Podman/Docker Kolla-ansible Kolla for image builds Paunch : Utility to launch and manage containers using YAML based configuration data Declarative in yaml file which can deploy containers and also manage them. It is also idempotent in behavior e.g: paunch --verbose apply --file examples/hello-world.yml --config-id hi

Infrastructure configurations 2018-02-21 Infrastructure configurations Airship Defines YAML manifests at global, type, site level Reuse the files defined at global and type Only site level manifests get modified for each site Everything is a YAML configuration. Complex initially to understand the YAML manifests YAML manifests are released by Airship community as part of Treasuremap project every month On the brighter side, the manifests provide you control to define settings at a very low level TripleO Defines manifests in TripleO-heat-templates under puppet/services docker/services extraconfig/service YAML files with custom keys to modify Airship - Any new component on Airship is just an addition of a manifest configuration (YAML) and helm charts without really modifying existing code TripleO – To add a new component, we will need to implement a new ansible playbook for the associated component. For e.g. a lot of new components provide helm charts along with them which can be easily leveraged on Airship (like Multus)

Deployment strategy Airship Deployment is performed via Genesis node which bootstraps the kubernetes cluster with control and compute nodes Genesis node itself becomes one of the control nodes post the initial bootstrap process This would mean that under cloud components are always running on the actual site. By this virtue, the site manifests remain on Deckhand database registry This would provide the current/actual state of under cloud within the site itself Thus not needing a separate hardware for managing under-cloud LCM. TripleO Deployment of the under cloud happens via seed node which bootstraps other nodes using an under-cloud OpenStack Under-cloud OpenStack will be used to deploy the OpenStack overcloud using heat templates and Kolla Under cloud would be more of a plug & play and can be shut-off and turned-on whenever required.

Bare metal server management 2018-02-21 Bare metal server management Airship MAAS Plugin for Drydock iDRAC/iLO management using IPMI plugin, redfish plugin for drydock Integration with Kubernetes cluster-api with Metal 3‘ with Airship ( WIP) TripleO Uses Ironic project for bootstrapping bare metal servers Update configuration in heat templates

Networking Airship Uses calico by default for control plane networking Calico provides ability to set network policies Can deploy any Kubernetes CNI plugins by changing the Airship manifests, provided helm charts are available for CNI TripleO Networking in Containerized TripleO is similar to Non Containerized TripleO Host based networking is leveraged for all containerized services Docker *Host* Driver is utilized.

Security Airship Utilizes all the Kubernetes native security features 2018-02-21 Security Airship Utilizes all the Kubernetes native security features Calico network policies Setting security context for POD/container ( privileges/non-privileged, linux capabilities) AppArmor Seccomp TripleO Audit rules Firewall management rules AIDE ( File directory integrity checker) K8s native security features like: Pod security policies, auth and authorization ,rbac, data encryption,resource access restriction A lot of 3rd party apps exist which can be integrated to secure K8s Least privilege access control, pod-pod communications, container runtime AppArmor ("Application Armor") is a Linux kernel security module that allows the system administrator to restrict programs' capabilities with per-program profiles. Profiles can allow capabilities like network access, raw socket access, and the permission to read, write, or execute files on matching path SecComp: is a Linux feature that allows a userspace program to set up syscall filters. Triple0 AIDE (Advanced Intrusion Detection Environment) is a file and directory integrity checker. It is used as medium to reveal possible unauthorized file tampering / changes.

Logging, Monitoring & Auditing Airship Uses EFK stack ( Fluentd, ElasticSearch, Kibana) TripleO Uses EFK stack ( Fluentd*, ElasticSearch, Kibana) Monitoring Uses Prometheus, ElasticSearch, Grafana Uses Sensu* *might be deprecated in future releases

Storage Airship Supports Ceph and NFS storage drivers TripleO Support Ceph and other NFS storage drivers Under-cloud can be configured to use Swift/Cinder Object Storage as image backend

Updates & Upgrades Airship 2018-02-21 Updates & Upgrades Airship Typical process is to use the same set of Under cloud tools to do Day 2 operations and upgrades Update the YAML manifests and deploy the new manifests with Shipyard Host impacting components ( Kubelet, Docker, kernel upgrades) using Argo Workflows ( WIP ) Upgrades for most of the services are handled via helm charts TripleO Ansible scripts to perform updates, pre/post update operations triggered via OpenStack overcloud command Ansible scripts for the under cloud upgrade (using FFU) FFU - Upgrading a TripleO deployment from Newton to Queens is done by first executing a minor update in both undercloud and overcloud, to ensure that the system is using the latest Newton release. After that, the undercloud is upgraded to the target version Queens. This will then be used to upgrade the overcloud.

2018-02-21