Seungkyu Ahn | SKTelecom Airship Airship: Making Lifecycle Management for Open Infrastructure Repeatable and Predictable Rodolfo Pacheco | AT&T Alan Meadows | AT&T Seungkyu Ahn | SKTelecom 5/21/2018 3 © 2018 AT&T Intellectual Property.  All Rights Reserved.  AT&T, the Globe logo, Mobilizing Your World and DirecTV are registered trademarks and service marks of AT&T Intellectual Property and/or AT&T affiliated companies.  All other marks are the property of their respective owners.

AGENDA WHAT IS AIRSHIP? THE SUB-PROJECTS AIRSHIP UNDERCLOUD HOW DOES IT WORK PROJECT ADOPTION 2 © 2018 AT&T Intellectual Property.  All Rights Reserved.  AT&T, the Globe logo, Mobilizing Your World and DirecTV are registered trademarks and service marks of AT&T Intellectual Property and/or AT&T affiliated companies.  All other marks are the property of their respective owners.

WHAT IS AIRSHIP? Security Containerization Quality We have chosen the name Airship for the nautically themed collection of interoperable open- source tools/services that provide for automated cloud provisioning and management by establishing an under cloud platform (UCP) leveraging Kubernetes. The focus of this project is the implementation of a declarative platform to introduce OpenStack on Kubernetes (OOK), and the lifecycle management of the resulting cloud, with the scale, speed, resiliency, flexibility and operational predictability demanded of Network Clouds. 8 © 2018 AT&T Intellectual Property.  All Rights Reserved.  AT&T, the Globe logo, Mobilizing Your World and DirecTV are registered trademarks and service marks of AT&T Intellectual Property and/or AT&T affiliated companies.  All other marks are the property of their respective owners.

WHY AIRSHIP? DECLARATIVE ONE WORKFLOW FOR LIFE CYCLE MANAGEMENT Security Containerization Quality To evolve how we deliver our cloud platform as well as manage the lifecycle of the software running there – including OpenStack -- we created Airship. DECLARATIVE Sites are declared using YAML. This includes both hard assets such as network configuration and bare metal hosts as well as soft assets like helm charts, their overrides, and container images. You manage the document and Airship implements it. CONTAINERS AS THE NEW UNIT OF SOFTWARE DELIVERY Containers are the unit of software delivery for Airship. Everything is a container. This allows us to progress environments from development, to testing, and finally to production with confidence. ONE WORKFLOW FOR LIFE CYCLE MANAGEMENT We needed a system that was predictable with life cycle management at its core. This meant ensuring we had one workflow that handled both initial deployments as well as future site updates. In other words, there should be virtually nothing different when interacting with a new deployment or providing an an update to an existing site. FLEXIBLE FOR DIFFERENT ARCHITECTURES AND SOFTWARE Airship is delivering environments both very small and large with a wide range of configurations. We use Airship to manage our entire cloud platform, not just OpenStack. 8 © 2018 AT&T Intellectual Property.  All Rights Reserved.  AT&T, the Globe logo, Mobilizing Your World and DirecTV are registered trademarks and service marks of AT&T Intellectual Property and/or AT&T affiliated companies.  All other marks are the property of their respective owners.

THE OPEN SOURCE PROJECTS AIRSHIP LEVERAGES OR INTEGRATES WITH Security Containerization Quality * = Future OpenStack-Helm: is a OpenStack project to provide a collection of Helm charts that simply, resiliently, and flexibly deploy OpenStack and related services on Kubernetes. Kubernetes Helm: is a tool for managing Kubernetes charts. Charts are packages of pre-configured Kubernetes resources. Kubernetes (K8s): is a open-source platform for managing containerized workloads and services, that facilitates both declarative configuration and automation. Barbican: Barbican is a REST API designed for the secure storage, provisioning and management of secrets. Keystone: Keystone provides authentication, authorization and service discovery mechanisms via HTTP primarily for use by projects in the OpenStack family. *Ironic: Ironic consists of an API and plug-ins for managing and provisioning physical machines in a security-aware and fault- tolerant manner. *Mogan: Mogan is an OpenStack project which offers bare metals as first class resources to users, supporting variety of bare metal provisioning drivers including Ironic. 8 © 2018 AT&T Intellectual Property.  All Rights Reserved.  AT&T, the Globe logo, Mobilizing Your World and DirecTV are registered trademarks and service marks of AT&T Intellectual Property and/or AT&T affiliated companies.  All other marks are the property of their respective owners.

THE AIRSHIP SUB-PROJECTS Security Containerization Quality Treasure Map: is the documentation project that outlines a reference architecture for automated cloud provisioning and management, leveraging the Airship interoperable open- source tools. Shipyard: is the directed acyclic graph controller for Kubernetes and OpenStack control plane life cycle management, and a component of the Undercloud Platform (UCP). Drydock: is a python REST orchestrator to translate a YAML host topology to a provisioned set of hosts and provide a set of post-provisioning instructions. Divingbell: is a lightweight solution for 1) Bare metal configuration management for a few very targeted use cases and 2) Bare metal package manager orchestration Deckhand: is a storage service for YAML-based configuration documents, which are managed through version control and automatically validated. Deckhand provides users with a variety of different document types that describe complex configurations using the features listed below. 8 © 2018 AT&T Intellectual Property.  All Rights Reserved.  AT&T, the Globe logo, Mobilizing Your World and DirecTV are registered trademarks and service marks of AT&T Intellectual Property and/or AT&T affiliated companies.  All other marks are the property of their respective owners.

THE AIRSHIP SUB-PROJECTS (Continued) Security Containerization Quality Armada: is a tool for managing multiple Helm charts with dependencies by centralizing all configurations in a single Armada YAML and providing life-cycle hooks for all Helm releases. Note: Armada is an enhancement to the helm client. Still using the helm tiller service, we deploy multiple charts with data- driven yamls. Berth: is a deliberately minimalist VM runner for Kubernetes. Promenade : is a tool for bootstrapping a resilient Docker, and Kubernetes cluster and managing its life-cycle via Helm charts. Pegleg: Pegleg is a document aggregator that provides early linting and validations via Deckhand for documents that can be consumed by Airship. 8 © 2018 AT&T Intellectual Property.  All Rights Reserved.  AT&T, the Globe logo, Mobilizing Your World and DirecTV are registered trademarks and service marks of AT&T Intellectual Property and/or AT&T affiliated companies.  All other marks are the property of their respective owners.

Undercloud Platform (UCP) Security Containerization Quality AIRSHIP - AN UNDERCLOUD PLATFORM (UCP) ENABLED NETWORK CLOUD Baremetal Host OS OCI Compliant Container Runtime | Kubelet Kubernetes Logging Monitoring & Alerting Keystone SDN Controller Glance Horizon Nova Neutron Cinder Heat Barbican Ranger Swift Security & Operations Ceph Calico Divingbell Drydock Shipyard Deckhand Promenade Berth OpenStack-Helm (OSH) Helm Armada VNF ONAP Cloud Platform Undercloud Platform (UCP) Runtime The blue highlights the Airship projects that facilitate the creation and life-cycle management of an undercloud platform that is used to enable a OpenStack based Network Cloud with the scale, speed, flexibility and operational predictability this infrastructure must deliver. The green highlights the Open Source projects that Airship leverages/ integrates with to deploy the undercloud platform. End to End CI/CD Different host profiles, ref. arch., storage devices -> RvW document We’re developing software to abstract hardware details (separations of concerns), allowing flexibility Projects we’re building have to work across 118 permutations (accounting for variability in physical hardware) continuously. This is one reason why lab requirements are the way they are. How do we achieve this? It’s because of our pipelines (enable delivery) © 2018 AT&T Intellectual Property.  All Rights Reserved.  AT&T, the Globe logo, Mobilizing Your World and DirecTV are registered trademarks and service marks of AT&T Intellectual Property and/or AT&T affiliated companies.  All other marks are the property of their respective owners. 8

File System / Partitions File System / Partitions How do we use Airship| What do we mean by declarative Ingress Chart Fluentd Chart Hardware lnfo IAM (Keystone) Chart mariadb Chart Kibana Chart Baremetal Shipyard Chart keystone Chart ElasticSearch Chart nova Chart OSH LOGGING MONITORING ALARMING AIRSHIP CNI CALICO – DNS - NTP SDS CEPH Kubernetes Network Deckhand Chart horizon Chart Fluentd Collector Chart OSH Network Link Drydock Chart swift Nova compute openvswitch cinder Chart Prometheus Chart horizon cinder memcached etcd Armada Chart Host Profile memcached Chart prometheus grafana exporter alert mgr. IAM deckhand drydock armada Promenade Chart Promenade Chart’s etcd Chart Grafana Chart core dns ntpd glance rabbitmq neutron SWITCH FABRIC libvirt swift Chart ceph mgr cephfs ceph osd ingress mariadb Kubernetes Chart DivingBell Chart K8s proxy keystone nova k8s controller Kube Exporter Chart glance Chart fluentd kibana elasticsrch collector Docker Kubelet neutron Chart shipyard promenade divingbell Node Exporter Chart SDS (Ceph) Chart calico node calico etcd rabbitmq Chart VF(s) VF(s) VF(s) ceph mon ceph rbd NIC ceph rgw NIC NIC CNI(Calico) Chart AlertMgr Chart Nova compute Chart k8s api OS etcd k8s scheduler File System / Partitions File System / Partitions Coredns Chart Host NUMA NUMA Openvswitch Chart ntpd Chart Libvirt chart CORE CORE CORE CORE CORE CORE

How do we use Airship| How do we bootstrap the cloud? Hardware lnfo IAM (Keystone) Chart keystone nova horizon ElasticSearch Fluentd Collector SDS (Ceph) CNI(Calico) Baremetal Cloud iam Cloud Harbour(**) Build Genesis.sh deckhand Diving bell drydock armada Calico Armada Apply Boostrap manifest promenade armada Generate shipyard Ceph osd armada Calico etcd Kubelet armada genesis.sh Hardware lnfo (Keystone) Chart IAM keystone nova horizon ElasticSearch Fluentd Collector SDS (Ceph) CNI(Calico) Baremetal Docker K8s contrl EXECUTE Docker Kubelet Ceph rbd genesis.sh Hardware lnfo (Keystone) Chart IAM keystone nova horizon ElasticSearch Fluentd Collector SDS (Ceph) CNI(Calico) Baremetal Retrieve images Docker Kubernetes tiller Kubelet K8s sched Ceph mon pegleg Artifactory Hardware lnfo (Keystone) Chart IAM keystone nova horizon ElasticSearch Fluentd Collector SDS (Ceph) CNI(Calico) Baremetal genesis.sh Prepare Manifest OS K8s api Jenkins Host genesis.sh Hardware lnfo (Keystone) Chart IAM keystone nova horizon ElasticSearch Fluentd Collector SDS (Ceph) CNI(Calico) Baremetal Kubernetes Docker genesis.sh Hardware lnfo (Keystone) Chart IAM keystone nova horizon ElasticSearch Fluentd Collector SDS (Ceph) CNI(Calico) Baremetal OS Deliver Host Press the button Genesis Done – Airship available

How do we use Airship| How do we deploy a cloud? Hardware lnfo IAM (Keystone) Chart keystone nova horizon ElasticSearch Fluentd Collector SDS (Ceph) CNI(Calico) Baremetal Deploy Strategy Cloud Host Host Compute Host Cloud Harbour (**) keystone Authenticate rabbitmq iam Deploy Compute Host groups K8s Scheduler Balance POD’s cinder deckhand glance promenade Commit Docs Artifactory OS Docker Kubelet Kubernetes drydock neutron fluentd armada Calico Diving bell libvirt shipyard Ceph osd ingress Hardware lnfo IAM (Keystone) Chart keystone nova horizon ElasticSearch Fluentd Collector SDS (Ceph) CNI(Calico) Baremetal kibana Calico etcd Gather Manifest For Cloud Docker Kubelet Kubernetes Hardware lnfo IAM (Keystone) Chart keystone nova horizon ElasticSearch Fluentd Collector SDS (Ceph) CNI(Calico) Baremetal K8s contrl Diving bell pegleg Jenkins nova prometheus Ceph osd OS Ceph rbd Fluentd collect Kubernetes tiller Kubernetes Docker Kubelet Docker Kubelet Kubernetes Docker Kubelet K8s sched mariadb Calico etcd Calico Nova compute Hardware lnfo IAM (Keystone) Chart keystone nova horizon ElasticSearch Fluentd Collector SDS (Ceph) CNI(Calico) Baremetal Docker Kubelet Kubernetes OS Ceph mon OS OS Kubernetes K8s api K8s api OS K8s proxy Hardware lnfo (Keystone) IAM Chart keystone nova horizon ElasticSearch Fluentd Collector SDS (Ceph) CNI(Calico) Baremetal Docker OS Trigger Deploy Action Deliver Manifest documents Host Press to deploy

How do we use Airship| How do we update a cloud? Hardware lnfo IAM (Keystone) Chart keystone nova horizon ElasticSearch Fluentd Collector SDS (Ceph) CNI(Calico) Baremetal Update Docs (*) Cloud Change Configuration Add new documents Host Host Compute Host Cloud Harbour (**) keystone keystone rabbitmq Authenticate iam neutron neutron cinder deckhand glance promenade Artifactory drydock Hardware lnfo (Keystone) IAM Chart keystone nova horizon ElasticSearch Fluentd Collector SDS (Ceph) CNI(Calico) Baremetal neutron fluentd armada Calico kibana Diving bell libvirt shipyard Ceph osd ingress Calico etcd Hardware lnfo IAM (Keystone) Chart keystone nova horizon ElasticSearch Fluentd Collector SDS (Ceph) CNI(Calico) Baremetal Gather updated Manifest K8s contrl Hardware lnfo (Keystone) IAM Chart keystone nova horizon ElasticSearch Fluentd Collector SDS (Ceph) CNI(Calico) Baremetal Diving bell nova prometheus Ceph osd pegleg Jenkins Ceph rbd Fluentd collect Kubernetes tiller Kubernetes Kubernetes Nova compute Nova compute Docker Kubelet Docker Kubelet Docker Kubelet K8s sched mariadb Calico etcd Calico Ceph mon Nova compute Hardware lnfo IAM (Keystone) Chart keystone nova horizon ElasticSearch Fluentd Collector SDS (Ceph) CNI(Calico) Baremetal OS OS Kubernetes K8s api K8s api OS K8s proxy Hardware lnfo (Keystone) IAM Chart keystone nova horizon ElasticSearch Fluentd Collector SDS (Ceph) CNI(Calico) Baremetal Docker Press to Update OS Trigger Update Action Host

+ HOW DO WE USE AIRSHIP | Documents, how are they managed? Pegleg REPOSITORY documents LAYERS global type sites software schemas profiles secrets cruiser unicycle Cloud(X) Cloud(Y) charts host hardware passphrase certs software profiles software profiles physical osh airship charts host charts host baremetal network osh osh + Pegleg Deckhand

THE CONTRIBUTORS Security Containerization Quality The Airship sub-projects started in the open through active collaboration both in the CNCF and OpenStack communities. An initial collaboration with SK telecom and Intel is growing. Community & Vendor branch – make parallel Add our branch to show us merging with vendor Community continues to move forward Have to re-base fairly often in order to keep updated with Vendor version, impacts influence further up the branch Illustrate this complexity Jared to look for replacement Working off Master today Convey internal burden on left – lessened on right side (benefit of community) Actively working with a community mindset © 2018 AT&T Intellectual Property.  All Rights Reserved.  AT&T, the Globe logo, Mobilizing Your World and DirecTV are registered trademarks and service marks of AT&T Intellectual Property and/or AT&T affiliated companies.  All other marks are the property of their respective owners. 14

(Unified Manifest & Life Cycle Hooks) (Logging & Monitoring) SK Telecom Case Study: TACO (SKT All Container OpenStack) TACO = OpenStack-Helm + Armada + SKT All-Flash Ceph + SKT In-House Tools OpenStack-Helm (OpenStack) Baremetal Provisioning (SKT Tool) Armada (Unified Manifest & Life Cycle Hooks) Asset Management (SKT Tool) Kubernetes Deployment (Kubespray) OpenStack-Helm (Logging & Monitoring) Network Configuration (SKT Tool) SKT Helm Chart (SKT Internal Ops Tools) From the collaboration with AT&T within Community, SKT is building TACO (SKT All Container OpenStack). TACO consists of OpenStack-Helm, Armada, SKT AF (All-Flash) Ceph, SKT In-House Tools including Analytics / Operation Automation Tools. One of benefits of Airship is that you can select sub-project you want to leverage, and easily integrate with your existing pipeline. In SKT, we use Armada to orchestrate most of control planes and operation tools to deploy OpenStack. I will describe more detail on how we leverage armada, especially in SKT’s internal CICD pipeline. SKT AF-Ceph Deployment 15

SK Telecom Case Study: How do we use Armada (a sub-project of Airship) CI Pipeline Basically, SKT uses Armada manifest as a center of CI Pipeline. We simplifies CI pipeline with a single unified Armada manifest, yet incorporate flexibility to do only necessary thing per each change. For instance, (윗 그림의 Submit PR for Chart A를 가르키며) when a developer changes something in Nova chart and does a PR, (Jenkins와 Armada CLI 그림을 가르키며) Jenkins triggers a job to launch Jenkins slave node, which has armada-cli in it. the slave node fetches a changed manifest file and openstack-helm chart, (k8s cluster 그림을 가르키며) then remotely call Armada API in a k8s cluster to start deploying and testing what has been changed. In this stage, it uses - - set option to only deploy “Nova”. What is better in this scenarios is, Armada will take care of any dependency to deploy Nova service, since it is already pre-defined in manifest. In this way, each changes are testing only necessary part, but still maintain a single unified manifest that can be utilized in full deployment step. For CD Pipeline, we are maintaining and versioning an Armada manifest for each deployment site, In this way, we don’t need to change anything in openstack-helm chart to deploy different openstack. We just need to maintain a manifest per deployment site, and that manifest is a single source of truth for operators to look at. CI 파이프라인의 경우, 각각의 팀원이 각각 다른 차트를 담당하고 있기 때문에, 차트 별로 PR이 submit되고 테스트 또한 변경된 부분만 테스트할 수 있어야 한다. (매번 전체 테스트를 하면 시간이 너무 많이 걸리므로..) 공통의 단일 manifest 파일을 사용하여 이러한 파이프라인을 구현하기 위해 apply 시점에 “–set” 으로 deploy할 차트 범위를 제한하게 하였다. 상세 플로우를 설명하자면, 우선 manifest 파일을 수정 후 PR 을 올리게 되면, 자동으로 Jenkins job이 trigger되고 armada-cli (client) 가 내장된 slave 노드가 launch된다. Slave 는 내부 repository에서 (skt) manifest 파일과 openstack-helm 차트를 fetch한 후 apply 명령을 사용하여 Armada API를 호출하여 deploy 과정을 수행한다. 위에서 말한 대로 “--set” 옵션으로 배포할 차트 범위를 제한하는데, 수정된 차트 및 해당 차트가 배포되기 위해 필요한 의존성 있는 차트들도 함께 배포하게 된다. (의존성 리스트는 pre-define되어 있음) CD 파이프라인의 경우, 우리는 production A, B, C등 각각 다른 환경 별로 별도의 manifest 파일을 관리한다. 이를 통해 (공통의) openstack-helm 차트 자체를 수정하지 않고 각 환경별로 다른 형상으로 openstack을 배포할 수 있다. “A” manifest “B” manifest “C” manifest Chart A Production A Production B Production C Openstack-helm CD Pipeline 16

SK Telecom Case Study: TACO (OpenStack-Helm & Armada) Adoption SKT Private Cloud Service : Infrastructure Service for Dev & QA (3Q 2018) VDI Service : Internal VDI Service (4Q 2018), Public VDI Service (2019) Big Data & Analytics Service : Development (2018), Production (2019) NFV (Telco Infra) : Green Field Production (2019) Important Things are Working together in the community to enhance Armada and OpenStack-Helm Keeping “Upstream First” strategy for mutual benefits Going a step further to review other sub-projects in Airship to boost possible collaboration Reference Implementation Information TACO All In One Deployment Script with Armada https://github.com/sktelecom-oslab/taco-scripts https://github.com/sktelecom-oslab/armada-manifests SKT have several production deployment happening in 2018 and 2019. Four important case are, First, SKT Private Cloud Service in 2018, Second, VDI Service for both Internal Use and Public VDI Service mainly targeting Education and Public Sector. Third one is for Big Data and Analytics Platform, and Lastly, for NFV Infrastructure targeting 2019 green field production deployment. SKT will continue to collaborate in OpenStack community to enhance OpenStack-Helm and Armada based on our production experiences, and SKT will keep “Upstream First” Policy both for Armada and OpenStack-Helm. In addition, SKT will start reviewing other projects in Airship for the any feasibility to leverage and contribute more. We believe working together in this great community is always better than doing alone. In addition, SKT provides TACO All-In-One script to deploy OpenStack with Armada manifest. It is mostly very similar to what we use internally. You will be able to do hands-on experience to use Armada and OpenStack-Helm.

HOW TO GET STARTED WHERE TO GO TRY IT OUT Visit https://airshipit.org Security Containerization Quality WHERE TO GO Visit https://airshipit.org TRY IT OUT https://github.com/openstack/airship-in-a-bottle CI/CD STATUS Project Pep8/Flake8 Bandit Docs Coverage Unit Tests Shipyard Yes 55% 234 Drydock 43% 126 Armada 65% 134 Deckhand 76% 257 Promenade 64% 3 8 © 2018 AT&T Intellectual Property.  All Rights Reserved.  AT&T, the Globe logo, Mobilizing Your World and DirecTV are registered trademarks and service marks of AT&T Intellectual Property and/or AT&T affiliated companies.  All other marks are the property of their respective owners.

HELP WITH OS API ALIGNMENT AND FEEDBACK TEMPEST INTEGRATION CALL TO ACTION Security Containerization Quality USER FEEDBACK HELP WITH OS API ALIGNMENT AND FEEDBACK TEMPEST INTEGRATION OPENSTACK/REQUIREMENTS WORK IRONIC DRYDOCK INTEGRATION KUBERNETES BACKED BY KEYSTONE 8 © 2018 AT&T Intellectual Property.  All Rights Reserved.  AT&T, the Globe logo, Mobilizing Your World and DirecTV are registered trademarks and service marks of AT&T Intellectual Property and/or AT&T affiliated companies.  All other marks are the property of their respective owners.

LINKS TO AIRSHIP SUB-PROJECT CODE & DOCUMENTATION (GITHUB) Questions? Alan Meadows (alan.meadows@att.com) Rodolfo Pacheco (rodolfo.pacheco@att.com) LINKS TO AIRSHIP SUB-PROJECT CODE & DOCUMENTATION (GITHUB) Treasure Map (http://treasuremap.readthedocs.io/en/latest/) Shipyard (https://github.com/openstack/airship-shipyard) Drydock (https://github.com/openstack/airship-drydock) Divingbell (https://github.com/openstack/airship-divingbell) Deckhand (https://github.com/openstack/airship-deckhand) Armada (https://github.com/openstack/airship-armada) Promenade (https://github.com/openstack/airship-promenade) Berth (https://github.com/openstack/airship-berth) Pegleg (https://github.com/openstack/airship-pegleg) Three Ways to Learn More - https://airshipit.org - IRC: #airshipit - Mailing List: lists.airshipit.org © 2018 AT&T Intellectual Property.  All Rights Reserved.  AT&T, the Globe logo, Mobilizing Your World and DirecTV are registered trademarks and service marks of AT&T Intellectual Property and/or AT&T affiliated companies.  All other marks are the property of their respective owners. 20