Lifecycle Management for the Multi-Access Edge Cloud: Experience with CORD Larry Peterson

Challenges Configurable and Extensible Platform Supports rapid deployment of new services It’s a Cloud, not a fixed-function solution Disaggregation implies fine-grain components Multiple Stakeholders with different requirements Developers: Tight development loops + Realistic emulation environments Integrators: Flexible way to build and certify composite services Operators: Lifecycle management + Configure/Control of production systems

Challenge CORD Use Cases (Trials) Extensible Platform Building Blocks VOLTHA xRAN Fabric vEPC …

Background CORD is a multi-access edge cloud Built using commodity servers and white-box switches/access devices (PON, RAN) Runs both scalable cloud services and disaggregated Telco services (BNG, EPC) Configured as Base Platform + One or more Service Profiles CORD adopts a model-based design philosophy System’s run-time behavior is defined by a set of models Decouple Service Control Plane (set of model) and Service Data Plane (set of images) CORD is an open source project Contributions managed through Gerrit Technical Steering Team curates releases

Data Center Data Center N W E WAN Routers Switching Fabric Compute Storage For illustration purposes; sizing is done independent of general design. 10GPON also an option (as is Mobile-CORD and Enterprise-CORD).

Multi-Access Edge I/O I/O PON RAN Core For illustration purposes; sizing is done independent of general design. 10GPON also an option (as is Mobile-CORD and Enterprise-CORD).

Architecture – Functional View CORD Controller vOLT Controller vSG vRouter R-CORD vCDN OpenStack ONOS Controller Service Graph for Residential CORD

Architecture – Component View CORD Controller Service Control Plane – Provision Services – Enforce Policy – Assemble & Control Data Paths Ctrl App Ctrl App Ctrl App Ctrl App Ctrl App Ctrl App Ctrl App Ctrl App Ctrl App Ctrl App Ctrl App VNF ONOS Kubernetes (OpenStack) Service Data Plane – Server-based (VMs, Containers) – Switch-based (SDN Control Apps) – Highly disaggregated – White-Box (OCP) Hardware OCP Hardware

Architecture – Operational View GUI REST API TOSCA VIM Event Bus XOS Core DB Sync Sync Sync Sync ONOS Open Stack Backend Control Apps, VMs, and Containers Runs as a Set of Micro-Services (Managed by Kubernetes – not shown)

Automated Configuration Workflow TOSCA Workflows – Provision & Configure Services – Runtime Operation CORD Controller Protobuf-based Models – Schema that Model Services – Core set Loaded at Boot Time – Dynamically Updated at Runtime Ctrl App Ctrl App Ctrl App Ctrl App Ctrl App Ctrl App Ctrl App Ctrl App Ctrl App Ctrl App Ctrl App VNF ONOS Kubernetes (OpenStack) Helm Charts – Containers that Implement Services – Core set Loaded at Boot Time – Dynamically Updated at Runtime OCP Hardware Kubernetes (optionally MAAS)

(Development/Production) Multi-Stage Pipeline On-Board Configure Build CORD Controller OCP Hardware Operational System (Development/Production)

Multi-Stage Pipeline On-Board Configure Build Service On-Board Configure Build CORD Controller OCP Hardware Gerrit On-Board – Make individual components available for inclusion in CORD. Service – 〈Xproto-Model, Helm-Chart, TOSCA-Workflow, VNF-Image〉 e.g., volt, vsg, vrouter,… Result – Commit source code to Gerrit

Profile = Composite Service Multi-Stage Pipeline Service Profile On-Board Configure Build CORD Controller OCP Hardware Gerrit Configure – Specify how a set of components are assembled into a solution. Profile – 〈Xproto-Model, Helm-Chart, TOSCA-Workflow〉 e.g., base-cord, rcord, mcord, ecord,… Result – Commit source code to Gerrit Insight: Profile = Composite Service

Multi-Stage Pipeline On-Board Configure Build Service Profile Target On-Board Configure Build CORD Controller OCP Hardware Gerrit DockerHub Build – Produce the set of Docker images that control the target POD. Target – 〈Profile, Scenario〉 e.g., local, single, mock, cord, ciab,… Result – Set of container images to DockerHub + Fully Qualified Helm-Charts

Multi-Stage Pipeline On-Board Configure Build Service Profile Target On-Board Configure Build CORD Controller OCP Hardware D Gerrit DockerHub Deploy – Manage lifecycle of containers on the target POD. Input – Helm-Charts (from Build Stage) Result – Operational POD (ready to be provisioned)

Multi-Stage Pipeline On-Board Configure Build Operate (Test) Workflow Provision Service Profile Target Configure On-Board Configure Build CORD Controller OCP Hardware D Operate (Test) Gerrit DockerHub Operate (Test) – Run regression/integration tests against commits/builds. Workflow – Parameterized TOSCA Result – Provisioned POD (ready to carry carry traffic)

Multi-Stage Pipeline Development Time Build Time Run-Time On-Board Service Profile Target Workflow On-Board Configure Build CORD Controller OCP Hardware D Operate (Test) Gerrit DockerHub Versioning Development Time Build Time Run-Time

Multi-Stage Pipeline Development Time Build Time Run-Time On-Board Service Profile Target Workflow On-Board Configure Build CORD Controller OCP Hardware D Jenkins (140 Tests) CP Workload Gerrit DockerHub DP Workload Smoke tests run on each commit. Integration tests run nightly on 22 physical PODS worldwide. Development Time Build Time Run-Time

(Self-Service Portal) Operational View Back Office Subscriber Database Cloud-based Automation Tools Operators Telemetry/Diagnostic Data Control (Self-Service Portal) Subscribers Data Plane Configure/Provision CORD Controller OCP Hardware (CORD Controller assembles per-subscriber service chains.)

Upgrade Service Helm-Charts On-Board Configure Build Operate (Test) CORD Controller OCP Hardware D Operate (Test) Gerrit DockerHub

Upgrade Service Helm-Charts specify versions and dependencies Semantics versioning at the granularity of services Each CORD Release corresponds to a tested combination of services Kubernetes manages incremental rollout/rollback Deploys new “VNF” image Deploys new Synchronizer image Synchronizer loads new models into XOS XOS manages upgrade of the service control plane Migrates database Generates backward compatible APIs

Approach to Lifecycle Management Kubernetes is responsible for Service Data Plane Support for implementing services (scale up/down, HA) Support for incremental upgrades (rollout/rollback) XOS is responsible for Service Control Plane Support for configuring and controlling services Support for incremental upgrades (transitioning state/interfaces)

XOS’s Role In Lifecycle Management Uniformity – Defines a system-wide layer of abstraction, making it possible to: Integrate into an existing operating environment as a single unified entity Extend the set of services without introducing OAM silos Reason about invariants and eliminate unstated assumptions that lead to errors Declarative – Represents services with models, making it possible to: Create composite services by composing multiple micro-services Create logical services that do not necessarily map onto micro-services Create federated or distributed services across multiple cloud instantiations Granularity – Tracks per-subscriber context, making it possible to: Control end-to-end service chains at the granularity of subscribers Correlate diagnostic/monitoring info at the granularity of subscribers Allocate resources and isolate performance on a per-subscriber (class) basis Migrate service chains in support of mobility

XOS Is… xproto – A declarative language for specifying models. Protocol Buffers: extended to support inheritance, relationships, and predicates xosgenx – An extensible toolchain to enforce models on operational system. Targets: APIs, Access Control, ORM, Synchronizer Framework,… core.xproto – A default (and malleable) set of core models. Models: Service, ServiceDependency, ServiceInstance, ServiceInstanceLink… Chart.yaml – A Helm Chart (plus set of container images) to deploy XOS. Micro-services: xos-core, xos-gui, xos-tosca, xos-db, xos-ws, redis,…

XOS Constructed from Micro-Services GUI REST API TOSCA VIM Views (UIs) CORD Controller (XOS) Event Bus XOS Core DB Data Model Synchronizers Backend Services and Resources

Example Model and Policy policy grant_policy < ctx.user.is_admin | exists Privilege:Privilege.object_type = obj.object_type & Privilege.object_id = obj.object_id & Privilege.accessor_type = "User" & Privilege.accessor_id = ctx.user.id & Privilege.permission = "role:admin" > message Privilege::grant_policy (XOSBase) { required int32 accessor_id = 1 [null = False]; required string accessor_type = 2 [null = False, max_length=1024]; required int32 controller_id = 3 [null = True]; required int32 object_id = 4 [null = False]; required string object_type = 5 [null = False, max_length=1024]; required string permission = 6 [null = False, default = "all", max_length=1024]; required string granted = 7 [content_type = "date", auto_now_add = True, max_length=1024]; required string expires = 8 [content_type = "date", null = True, max_length=1024]; }

XOS Generative Toolchain GUI REST API VIM TOSCA Generated Code – API Tests – Northbound Interfaces – Enforce Security Policy – Object Relation Mapper – Synchronizer Framework Event Bus XOS Core DB Sync Sync Sync Sync

Core Models Service Slice (Resources) (Control) Service Slice Controller Service Slice Instance (Resources) (Control) Controller Service Slice Instance (Resources) (Control) ONOS vRouter

Core Models Service Graph R-CORD vOLT vSG vRouter Controller R-CORD Controller Controller Controller vOLT vSG vRouter Service Instance Service Instance Service Instance Subscriber CORD Service Chain = At the granularity of subscribers (or subscriber classes)

Core Models EPC-as-a-Service eNB Composite Services Network Slicing Controller SPGW Control vMME vSPGW-c vSPGW-u Controller vHSS eNB Composite Services Network Slicing

XOS: Another Perspective Use Cases (Trials) Abstraction Layer XOS Building Blocks VOLTHA xRAN Fabric vEPC …