ENTER THE TITLE OF YOUR OPENSTACK EFFECTIVE KUBERNETES HELM ENGINEERING OpenStack-Helm Lessons Learned ENTER THE TITLE OF YOUR OPENSTACK PRESENTATION Matt McEuen, AT&T Seunkyu Ahn, SK Telecom Jaesuk Ahn, SK Telecom OpenStack Vancouver Summit 2018 – 5/23/2018

AGENDA Dependencies Reuse Configuration Ingress & Logging Security Containerization Quality Dependencies Reuse Configuration Ingress & Logging ** Overview (1min) ** - What is Helm - What is OpenStack-Helm ** Dependency Management (1 min) ** Overview of built-in Helm dependencies Use of Stackanetes' K8s-entrypoint ** Engineering for Reuse (2-3 min) ** Using a chart as a template library Give example of short parameterized manifest template Give overview of Helm Toolkit Long term, interested in refactoring non-OpenStack-specific parts into an external The approaches below leverage consistency in values across charts ** Resource Types (2-3min) ** Principle: all configuration must be injectable via Values.yaml overrides Generating config (e.g. nova.conf) directly from YAML (give example) oslo, ini, env variables note: Helm has a built-in toYaml function Inject config files https://github.com/openstack/openstack-helm/blob/master/ldap/values.yaml#L114 ** Production Example (1 min) ** ** Questions (1 min) ** © 2018 SK Telecom Co, LTD. All Rights Reserved. © 2018 AT&T Intellectual Property.  All Rights Reserved.  AT&T, the Globe logo, Mobilizing Your World and DirecTV are registered trademarks and service marks of AT&T Intellectual Property and/or AT&T affiliated companies.  All other marks are the property of their respective owners.

Quick level-set Kubernetes: cluster-based container orchestration Security Containerization Quality Kubernetes: cluster-based container orchestration Helm: Kubernetes packaging and deployment management OpenStack-Helm: Helm charts for OpenStack services and related tooling Airship: An Open Infrastructure platform for OpenStack © 2018 SK Telecom Co, LTD. All Rights Reserved. © 2018 AT&T Intellectual Property.  All Rights Reserved.  AT&T, the Globe logo, Mobilizing Your World and DirecTV are registered trademarks and service marks of AT&T Intellectual Property and/or AT&T affiliated companies.  All other marks are the property of their respective owners.

Dependencies # requirements.yaml dependencies: - name: nginx Security Containerization Quality Out of box Helm dependency management: A couple of shortcomings for OpenStack use: Dependencies aren’t “shared” across deployed charts (think: Keystone) Dependencies don’t have visibility into when resources have been bootstrapped (think: Keystone) (example from https://github.com/kubernetes/helm/blob/master/docs/helm/helm_dependency.md) # requirements.yaml dependencies: - name: nginx version: "1.2.3" repository: "https://example.com/charts" - name: memcached version: "3.2.1" repository: "https://another.example.com/charts" © 2018 SK Telecom Co, LTD. All Rights Reserved. © 2018 AT&T Intellectual Property.  All Rights Reserved.  AT&T, the Globe logo, Mobilizing Your World and DirecTV are registered trademarks and service marks of AT&T Intellectual Property and/or AT&T affiliated companies.  All other marks are the property of their respective owners.

Dependencies Stackanetes’ kubernetes-entrypoint to the rescue! Security Containerization Quality Stackanetes’ kubernetes-entrypoint to the rescue! # Glance chart values.yaml dependencies: static: api: jobs: - glance-storage-init - glance-db-sync - glance-rabbit-init - glance-ks-user - glance-ks-endpoints # Glance chart deloyment-api.yaml initContainers: {{ tuple $envAll “api” $mounts_glance_api_init | include "helm-toolkit.snippets.kubernetes_entrypoint_init_container" | indent 8 }} © 2018 SK Telecom Co, LTD. All Rights Reserved. © 2018 AT&T Intellectual Property.  All Rights Reserved.  AT&T, the Globe logo, Mobilizing Your World and DirecTV are registered trademarks and service marks of AT&T Intellectual Property and/or AT&T affiliated companies.  All other marks are the property of their respective owners.

reuse # requirements.yaml dependencies: - name: helm-toolkit Security Containerization Quality OpenStack-Helm uses Helm dependencies for its shared library chart: Helm-Toolkit # requirements.yaml dependencies: - name: helm-toolkit repository: http://localhost:8879/charts version: 0.1.0 © 2018 SK Telecom Co, LTD. All Rights Reserved. © 2018 AT&T Intellectual Property.  All Rights Reserved.  AT&T, the Globe logo, Mobilizing Your World and DirecTV are registered trademarks and service marks of AT&T Intellectual Property and/or AT&T affiliated companies.  All other marks are the property of their respective owners.

reuse Helm-Toolkit functions (examples): Security Containerization Quality Helm-Toolkit functions (examples): Endpoints: values.yaml-driven endpoint helper functions Manifests: full manifests that are 95% similar across charts Oslo database init / sync jobs Keystone endpoint / service / user setup jobs Kubernetes ingress service definitions Scripts: reusable Shell or Python scripts Keystone endpoint / service / user setup Rally test harness for helm test use Snippets: reusable pieces of templated manifest Values.yaml-driven labelling Prometheus annotation setup Utils: GoTpl helper functions joinListWithComma Deep YAML merge function © 2018 SK Telecom Co, LTD. All Rights Reserved. © 2018 AT&T Intellectual Property.  All Rights Reserved.  AT&T, the Globe logo, Mobilizing Your World and DirecTV are registered trademarks and service marks of AT&T Intellectual Property and/or AT&T affiliated companies.  All other marks are the property of their respective owners.

reuse Helm-Toolkit manifest example: _job-ks-user.yaml.tpl Security Containerization Quality Helm-Toolkit manifest example: _job-ks-user.yaml.tpl # Glance’s job-ks-user.yaml template {{- if .Values.manifests.job_ks_user }} {{- $ksUserJob := dict "envAll" . "serviceName" "glance" -}} {{ $ksUserJob | include "helm-toolkit.manifests.job_ks_user" }} {{- end }} # Rendered job-ks-user job apiVersion: batch/v1 kind: Job metadata: creationTimestamp: 2018-04-25T02:49:40Z labels: application: glance component: ks-user controller-uid: 4cacab22-4833-11e8-baec-3c528210de87 job-name: glance-ks-user release_group: glance name: glance-ks-user … (177 lines) … © 2018 SK Telecom Co, LTD. All Rights Reserved. © 2018 AT&T Intellectual Property.  All Rights Reserved.  AT&T, the Globe logo, Mobilizing Your World and DirecTV are registered trademarks and service marks of AT&T Intellectual Property and/or AT&T affiliated companies.  All other marks are the property of their respective owners.

Configuration Security Containerization Quality Principle: All configuration should be values.yaml (overrides) driven Most config files can be generated directly from values.yaml Helm supports toYaml out of box OpenStack-Helm’s Helm-Toolkit adds functions to generate Oslo config Ini files Kubernetes environment variables # Nova values.yaml conf: nova: DEFAULT: default_ephemeral_format: ext4 ram_allocation_ratio: 1.0 disk_allocation_ratio: 1.0 cpu_allocation_ratio: 3.0 state_path: /var/lib/nova … © 2018 SK Telecom Co, LTD. All Rights Reserved. © 2018 AT&T Intellectual Property.  All Rights Reserved.  AT&T, the Globe logo, Mobilizing Your World and DirecTV are registered trademarks and service marks of AT&T Intellectual Property and/or AT&T affiliated companies.  All other marks are the property of their respective owners.

Configuration Security Containerization Quality Principle: All configuration should be values.yaml (overrides) driven When needed, full configuration files can be specified via values.yaml # LDAP values.yaml data: sample: | dn: ou=People,dc=cluster,dc=local objectclass: organizationalunit ou: People description: We the People … © 2018 SK Telecom Co, LTD. All Rights Reserved. © 2018 AT&T Intellectual Property.  All Rights Reserved.  AT&T, the Globe logo, Mobilizing Your World and DirecTV are registered trademarks and service marks of AT&T Intellectual Property and/or AT&T affiliated companies.  All other marks are the property of their respective owners.

INgress Security Containerization Quality There are two types of ingress controllers: cluster mode and namespace mode. Cluster mode : DaemonSet type and vip # Ingress values.yaml deployment: mode: cluster type: DaemonSet cluster: class: “nginx-cluster” … network: host_namespace: true vip: manage: true interface: eth0 addr: 192.168.30.10/32 © 2018 SK Telecom Co, LTD. All Rights Reserved.

INgress Security Containerization Quality The namespace mode recognizes only the ingress of the corresponding namespace. Namespace mode : Deployment type # Ingress values.yaml deployment: mode: namespace type: Deployment … network: host_namespace: false # Set to true if used with cluster mode. © 2018 SK Telecom Co, LTD. All Rights Reserved.

Logging A default architecture is a 3 tier architecture. Security Containerization Quality A default architecture is a 3 tier architecture. Fluent-bit (each node) -> Fluentd (aggregator) -> ElasticSearch openstack-helm-infra / fluent-logging / values.yaml # fluent-logging values.yaml conf: fluentbit: - service: … - containers_tail: - kube_filter: - fluentd_output: header: output Name: forward Match: “*” Host: fluentd-svc Port: 5170 © 2018 SK Telecom Co, LTD. All Rights Reserved.

Logging It can be customized with a simple architecture. Security Containerization Quality It can be customized with a simple architecture. Fluent-bit (each node) -> ElasticSearch # fluent-logging values.yaml conf: fluentbit: - service: … - containers_tail: - kube_filter: - fluentd_output: header: output Name: es Match: “*” Host: elasticsearch-svc Port: 80 Logstash_Format: On HTTP_User: “admin” HTTP_Passwd: “changeme” © 2018 SK Telecom Co, LTD. All Rights Reserved.

Thank you! Questions ? Security Containerization Quality © 2018 SK Telecom Co, LTD. All Rights Reserved. © 2018 AT&T Intellectual Property.  All Rights Reserved.  AT&T, the Globe logo, Mobilizing Your World and DirecTV are registered trademarks and service marks of AT&T Intellectual Property and/or AT&T affiliated companies.  All other marks are the property of their respective owners.