1. ENTER THE TITLE OF YOUR OPENSTACK
EFFECTIVE KUBERNETES HELM ENGINEERING
OpenStack-Helm Lessons Learned
ENTER THE TITLE OF YOUR OPENSTACK
PRESENTATION
Matt McEuen, AT&T
Seunkyu Ahn, SK Telecom
Jaesuk Ahn, SK Telecom
OpenStack Vancouver Summit 2018 – 5/23/2018

2. AGENDA Dependencies Reuse Configuration Ingress & Logging Security
Containerization
Quality
Dependencies
Reuse
Configuration
Ingress & Logging
** Overview (1min) **
- What is Helm
- What is OpenStack-Helm
** Dependency Management (1 min) **
Overview of built-in Helm dependencies
Use of Stackanetes' K8s-entrypoint
** Engineering for Reuse (2-3 min) **
Using a chart as a template library
Give example of short parameterized manifest template
Give overview of Helm Toolkit
Long term, interested in refactoring non-OpenStack-specific parts into an external
The approaches below leverage consistency in values across charts
** Resource Types (2-3min) **
Principle: all configuration must be injectable via Values.yaml overrides
Generating config (e.g. nova.conf) directly from YAML (give example)
oslo, ini, env variables
note: Helm has a built-in toYaml function
Inject config files
** Production Example (1 min) **
** Questions (1 min) **
© 2018 SK Telecom Co, LTD. All Rights Reserved.
© 2018 AT&T Intellectual Property.  All Rights Reserved.  AT&T, the Globe logo, Mobilizing Your World and DirecTV are registered trademarks and service marks of AT&T Intellectual Property and/or AT&T affiliated companies.  All other marks are the property of their respective owners.

3. Quick level-set Kubernetes: cluster-based container orchestration
Security
Containerization
Quality
Kubernetes: cluster-based container orchestration
Helm: Kubernetes packaging and deployment management
OpenStack-Helm: Helm charts for OpenStack services and related tooling
Airship: An Open Infrastructure platform for OpenStack
© 2018 SK Telecom Co, LTD. All Rights Reserved.
© 2018 AT&T Intellectual Property.  All Rights Reserved.  AT&T, the Globe logo, Mobilizing Your World and DirecTV are registered trademarks and service marks of AT&T Intellectual Property and/or AT&T affiliated companies.  All other marks are the property of their respective owners.

4. Dependencies # requirements.yaml dependencies: - name: nginx
Security
Containerization
Quality
Out of box Helm dependency management:
A couple of shortcomings for OpenStack use:
Dependencies aren’t “shared” across deployed charts (think: Keystone)
Dependencies don’t have visibility into when resources have been bootstrapped (think: Keystone)
(example from
# requirements.yaml
dependencies:
- name: nginx
version: "1.2.3"
repository: "
- name: memcached
version: "3.2.1"
repository: "
© 2018 SK Telecom Co, LTD. All Rights Reserved.
© 2018 AT&T Intellectual Property.  All Rights Reserved.  AT&T, the Globe logo, Mobilizing Your World and DirecTV are registered trademarks and service marks of AT&T Intellectual Property and/or AT&T affiliated companies.  All other marks are the property of their respective owners.

5. Dependencies Stackanetes’ kubernetes-entrypoint to the rescue!
Security
Containerization
Quality
Stackanetes’ kubernetes-entrypoint to the rescue!
# Glance chart values.yaml
dependencies:
static:
api:
jobs:
- glance-storage-init
- glance-db-sync
- glance-rabbit-init
- glance-ks-user
- glance-ks-endpoints
# Glance chart deloyment-api.yaml
initContainers:
{{ tuple $envAll “api” $mounts_glance_api_init |
include "helm-toolkit.snippets.kubernetes_entrypoint_init_container" | indent 8 }}
© 2018 SK Telecom Co, LTD. All Rights Reserved.
© 2018 AT&T Intellectual Property.  All Rights Reserved.  AT&T, the Globe logo, Mobilizing Your World and DirecTV are registered trademarks and service marks of AT&T Intellectual Property and/or AT&T affiliated companies.  All other marks are the property of their respective owners.

6. reuse # requirements.yaml dependencies: - name: helm-toolkit
Security
Containerization
Quality
OpenStack-Helm uses Helm dependencies for its shared library chart:
Helm-Toolkit
# requirements.yaml
dependencies:
- name: helm-toolkit
repository:
version: 0.1.0
© 2018 SK Telecom Co, LTD. All Rights Reserved.
© 2018 AT&T Intellectual Property.  All Rights Reserved.  AT&T, the Globe logo, Mobilizing Your World and DirecTV are registered trademarks and service marks of AT&T Intellectual Property and/or AT&T affiliated companies.  All other marks are the property of their respective owners.

7. reuse Helm-Toolkit functions (examples):
Security
Containerization
Quality
Helm-Toolkit functions (examples):
Endpoints: values.yaml-driven endpoint helper functions
Manifests: full manifests that are 95% similar across charts
Oslo database init / sync jobs
Keystone endpoint / service / user setup jobs
Kubernetes ingress service definitions
Scripts: reusable Shell or Python scripts
Keystone endpoint / service / user setup
Rally test harness for helm test use
Snippets: reusable pieces of templated manifest
Values.yaml-driven labelling
Prometheus annotation setup
Utils: GoTpl helper functions
joinListWithComma
Deep YAML merge function
© 2018 SK Telecom Co, LTD. All Rights Reserved.
© 2018 AT&T Intellectual Property.  All Rights Reserved.  AT&T, the Globe logo, Mobilizing Your World and DirecTV are registered trademarks and service marks of AT&T Intellectual Property and/or AT&T affiliated companies.  All other marks are the property of their respective owners.

8. reuse Helm-Toolkit manifest example: _job-ks-user.yaml.tpl Security
Containerization
Quality
Helm-Toolkit manifest example: _job-ks-user.yaml.tpl
# Glance’s job-ks-user.yaml template
{{- if .Values.manifests.job_ks_user }}
{{- $ksUserJob := dict "envAll" . "serviceName" "glance" -}}
{{ $ksUserJob | include "helm-toolkit.manifests.job_ks_user" }}
{{- end }}
# Rendered job-ks-user job
apiVersion: batch/v1
kind: Job
metadata:
creationTimestamp: T02:49:40Z
labels:
application: glance
component: ks-user
controller-uid: 4cacab e8-baec-3c528210de87
job-name: glance-ks-user
release_group: glance
name: glance-ks-user
… (177 lines) …
© 2018 SK Telecom Co, LTD. All Rights Reserved.
© 2018 AT&T Intellectual Property.  All Rights Reserved.  AT&T, the Globe logo, Mobilizing Your World and DirecTV are registered trademarks and service marks of AT&T Intellectual Property and/or AT&T affiliated companies.  All other marks are the property of their respective owners.

9. Configuration
Security
Containerization
Quality
Principle: All configuration should be values.yaml (overrides) driven
Most config files can be generated directly from values.yaml
Helm supports toYaml out of box
OpenStack-Helm’s Helm-Toolkit adds functions to generate
Oslo config
Ini files
Kubernetes environment variables
# Nova values.yaml
conf:
nova:
DEFAULT:
default_ephemeral_format: ext4
ram_allocation_ratio: 1.0
disk_allocation_ratio: 1.0
cpu_allocation_ratio: 3.0
state_path: /var/lib/nova …
© 2018 SK Telecom Co, LTD. All Rights Reserved.
© 2018 AT&T Intellectual Property.  All Rights Reserved.  AT&T, the Globe logo, Mobilizing Your World and DirecTV are registered trademarks and service marks of AT&T Intellectual Property and/or AT&T affiliated companies.  All other marks are the property of their respective owners.

10. Configuration
Security
Containerization
Quality
Principle: All configuration should be values.yaml (overrides) driven
When needed, full configuration files can be specified via values.yaml
# LDAP values.yaml
data:
sample: |
dn: ou=People,dc=cluster,dc=local
objectclass: organizationalunit
ou: People
description: We the People …
© 2018 SK Telecom Co, LTD. All Rights Reserved.
© 2018 AT&T Intellectual Property.  All Rights Reserved.  AT&T, the Globe logo, Mobilizing Your World and DirecTV are registered trademarks and service marks of AT&T Intellectual Property and/or AT&T affiliated companies.  All other marks are the property of their respective owners.

11. INgress
Security
Containerization
Quality
There are two types of ingress controllers: cluster mode and namespace mode.
Cluster mode : DaemonSet type and vip
# Ingress values.yaml
deployment:
mode: cluster
type: DaemonSet
cluster:
class: “nginx-cluster” …
network:
host_namespace: true
vip:
manage: true
interface: eth0
addr: /32
© 2018 SK Telecom Co, LTD. All Rights Reserved.

12. INgress
Security
Containerization
Quality
The namespace mode recognizes only the ingress of the corresponding namespace.
Namespace mode : Deployment type
# Ingress values.yaml
deployment:
mode: namespace
type: Deployment …
network:
host_namespace: false # Set to true if used with cluster mode.
© 2018 SK Telecom Co, LTD. All Rights Reserved.

13. Logging A default architecture is a 3 tier architecture.
Security
Containerization
Quality
A default architecture is a 3 tier architecture.
Fluent-bit (each node) -> Fluentd (aggregator) -> ElasticSearch
openstack-helm-infra / fluent-logging / values.yaml
# fluent-logging values.yaml
conf:
fluentbit:
- service: …
- containers_tail:
- kube_filter:
- fluentd_output:
header: output
Name: forward
Match: “*”
Host: fluentd-svc
Port: 5170
© 2018 SK Telecom Co, LTD. All Rights Reserved.

14. Logging It can be customized with a simple architecture.
Security
Containerization
Quality
It can be customized with a simple architecture.
Fluent-bit (each node) -> ElasticSearch
# fluent-logging values.yaml
conf:
fluentbit:
- service: …
- containers_tail:
- kube_filter:
- fluentd_output:
header: output
Name: es
Match: “*”
Host: elasticsearch-svc
Port: 80
Logstash_Format: On
HTTP_User: “admin”
HTTP_Passwd: “changeme”
© 2018 SK Telecom Co, LTD. All Rights Reserved.

15. Thank you! Questions ? Security Containerization Quality
© 2018 SK Telecom Co, LTD. All Rights Reserved.
© 2018 AT&T Intellectual Property.  All Rights Reserved.  AT&T, the Globe logo, Mobilizing Your World and DirecTV are registered trademarks and service marks of AT&T Intellectual Property and/or AT&T affiliated companies.  All other marks are the property of their respective owners.