1. Lifecycle Management for the Multi-Access Edge Cloud: Experience with CORD
Larry Peterson

2. Challenges Configurable and Extensible Platform
Supports rapid deployment of new services
It’s a Cloud, not a fixed-function solution
Disaggregation implies fine-grain components
Multiple Stakeholders with different requirements
Developers: Tight development loops + Realistic emulation environments
Integrators: Flexible way to build and certify composite services
Operators: Lifecycle management + Configure/Control of production systems

3. Challenge CORD Use Cases (Trials) Extensible Platform Building Blocks
VOLTHA
xRAN
Fabric
vEPC …

4. Background CORD is a multi-access edge cloud
Built using commodity servers and white-box switches/access devices (PON, RAN)
Runs both scalable cloud services and disaggregated Telco services (BNG, EPC)
Configured as Base Platform + One or more Service Profiles
CORD adopts a model-based design philosophy
System’s run-time behavior is defined by a set of models
Decouple Service Control Plane (set of model) and Service Data Plane (set of images)
CORD is an open source project
Contributions managed through Gerrit
Technical Steering Team curates releases

5. Data Center Data Center N W E WAN Routers Switching Fabric Compute
Storage
For illustration purposes; sizing is done independent of general design.
10GPON also an option (as is Mobile-CORD and Enterprise-CORD).

6. Multi-Access Edge I/O I/O PON RAN Core
For illustration purposes; sizing is done independent of general design.
10GPON also an option (as is Mobile-CORD and Enterprise-CORD).

7. Architecture – Functional View
CORD Controller
vOLT
Controller
vSG
vRouter
R-CORD
vCDN
OpenStack
ONOS
Controller
Service Graph for Residential CORD

8. Architecture – Component View
CORD Controller
Service Control Plane
– Provision Services
– Enforce Policy
– Assemble & Control Data Paths
Ctrl
App
Ctrl
App
Ctrl
App
Ctrl
App
Ctrl
App
Ctrl
App
Ctrl
App
Ctrl
App
Ctrl
App
Ctrl
App
Ctrl
App
VNF
ONOS
Kubernetes
(OpenStack)
Service Data Plane
– Server-based (VMs, Containers)
– Switch-based (SDN Control Apps)
– Highly disaggregated
– White-Box (OCP) Hardware
OCP
Hardware

9. Architecture – Operational View
GUI
REST
API
TOSCA
VIM
Event
Bus
XOS
Core DB
Sync
Sync
Sync
Sync
ONOS
Open
Stack
Backend Control Apps, VMs, and Containers
Runs as a Set of Micro-Services
(Managed by Kubernetes – not shown)

10. Automated Configuration
Workflow
TOSCA Workflows
– Provision & Configure Services
– Runtime Operation
CORD Controller
Protobuf-based Models
– Schema that Model Services
– Core set Loaded at Boot Time
– Dynamically Updated at Runtime
Ctrl
App
Ctrl
App
Ctrl
App
Ctrl
App
Ctrl
App
Ctrl
App
Ctrl
App
Ctrl
App
Ctrl
App
Ctrl
App
Ctrl
App
VNF
ONOS
Kubernetes
(OpenStack)
Helm Charts
– Containers that Implement Services
– Core set Loaded at Boot Time
– Dynamically Updated at Runtime
OCP
Hardware
Kubernetes (optionally MAAS)

11. (Development/Production)
Multi-Stage Pipeline
On-Board
Configure
Build
CORD Controller
OCP Hardware
Operational System
(Development/Production)

12. Multi-Stage Pipeline On-Board Configure Build
Service
On-Board
Configure
Build
CORD Controller
OCP Hardware
Gerrit
On-Board – Make individual components available for inclusion in CORD.
Service – 〈Xproto-Model, Helm-Chart, TOSCA-Workflow, VNF-Image〉
e.g., volt, vsg, vrouter,…
Result – Commit source code to Gerrit

13. Profile = Composite Service
Multi-Stage Pipeline
Service
Profile
On-Board
Configure
Build
CORD Controller
OCP Hardware
Gerrit
Configure – Specify how a set of components are assembled into a solution.
Profile – 〈Xproto-Model, Helm-Chart, TOSCA-Workflow〉
e.g., base-cord, rcord, mcord, ecord,…
Result – Commit source code to Gerrit
Insight:
Profile = Composite Service

14. Multi-Stage Pipeline On-Board Configure Build
Service
Profile
Target
On-Board
Configure
Build
CORD Controller
OCP Hardware
Gerrit
DockerHub
Build – Produce the set of Docker images that control the target POD.
Target – 〈Profile, Scenario〉
e.g., local, single, mock, cord, ciab,…
Result – Set of container images to DockerHub + Fully Qualified Helm-Charts

15. Multi-Stage Pipeline On-Board Configure Build
Service
Profile
Target
On-Board
Configure
Build
CORD Controller
OCP Hardware D
Gerrit
DockerHub
Deploy – Manage lifecycle of containers on the target POD.
Input – Helm-Charts (from Build Stage)
Result – Operational POD (ready to be provisioned)

16. Multi-Stage Pipeline On-Board Configure Build Operate (Test)
Workflow
Provision
Service
Profile
Target
Configure
On-Board
Configure
Build
CORD Controller
OCP Hardware D
Operate
(Test)
Gerrit
DockerHub
Operate (Test) – Run regression/integration tests against commits/builds.
Workflow – Parameterized TOSCA
Result – Provisioned POD (ready to carry carry traffic)

17. Multi-Stage Pipeline Development Time Build Time Run-Time On-Board
Service
Profile
Target
Workflow
On-Board
Configure
Build
CORD Controller
OCP Hardware D
Operate
(Test)
Gerrit
DockerHub
Versioning
Development Time
Build Time
Run-Time

18. Multi-Stage Pipeline Development Time Build Time Run-Time On-Board
Service
Profile
Target
Workflow
On-Board
Configure
Build
CORD Controller
OCP Hardware D
Jenkins
(140 Tests)
CP Workload
Gerrit
DockerHub
DP Workload
Smoke tests run on
each commit.
Integration tests run
nightly on 22 physical
PODS worldwide.
Development Time
Build Time
Run-Time

19. (Self-Service Portal)
Operational View
Back Office
Subscriber
Database
Cloud-based
Automation Tools
Operators
Telemetry/Diagnostic Data
Control
(Self-Service Portal)
Subscribers
Data Plane
Configure/Provision
CORD Controller
OCP Hardware
(CORD Controller assembles
per-subscriber service chains.)

20. Upgrade Service Helm-Charts On-Board Configure Build Operate (Test)
CORD Controller
OCP Hardware D
Operate
(Test)
Gerrit
DockerHub

21. Upgrade Service Helm-Charts specify versions and dependencies
Semantics versioning at the granularity of services
Each CORD Release corresponds to a tested combination of services
Kubernetes manages incremental rollout/rollback
Deploys new “VNF” image
Deploys new Synchronizer image
Synchronizer loads new models into XOS
XOS manages upgrade of the service control plane
Migrates database
Generates backward compatible APIs

22. Approach to Lifecycle Management
Kubernetes is responsible for Service Data Plane
Support for implementing services (scale up/down, HA)
Support for incremental upgrades (rollout/rollback)
XOS is responsible for Service Control Plane
Support for configuring and controlling services
Support for incremental upgrades (transitioning state/interfaces)

23. XOS’s Role In Lifecycle Management
Uniformity – Defines a system-wide layer of abstraction, making it possible to:
Integrate into an existing operating environment as a single unified entity
Extend the set of services without introducing OAM silos
Reason about invariants and eliminate unstated assumptions that lead to errors
Declarative – Represents services with models, making it possible to:
Create composite services by composing multiple micro-services
Create logical services that do not necessarily map onto micro-services
Create federated or distributed services across multiple cloud instantiations
Granularity – Tracks per-subscriber context, making it possible to:
Control end-to-end service chains at the granularity of subscribers
Correlate diagnostic/monitoring info at the granularity of subscribers
Allocate resources and isolate performance on a per-subscriber (class) basis
Migrate service chains in support of mobility

24. XOS Is… xproto – A declarative language for specifying models.
Protocol Buffers: extended to support inheritance, relationships, and predicates
xosgenx – An extensible toolchain to enforce models on operational system.
Targets: APIs, Access Control, ORM, Synchronizer Framework,…
core.xproto – A default (and malleable) set of core models.
Models: Service, ServiceDependency, ServiceInstance, ServiceInstanceLink…
Chart.yaml – A Helm Chart (plus set of container images) to deploy XOS.
Micro-services: xos-core, xos-gui, xos-tosca, xos-db, xos-ws, redis,…

25. XOS Constructed from Micro-Services
GUI
REST
API
TOSCA
VIM
Views (UIs)
CORD Controller (XOS)
Event
Bus
XOS
Core DB
Data Model
Synchronizers
Backend Services and Resources

26. Example Model and Policy
policy grant_policy < ctx.user.is_admin
| exists Privilege:Privilege.object_type = obj.object_type
& Privilege.object_id = obj.object_id
& Privilege.accessor_type = "User"
& Privilege.accessor_id = ctx.user.id
& Privilege.permission = "role:admin" >
message Privilege::grant_policy (XOSBase)
{ required int32 accessor_id = 1 [null = False];
required string accessor_type = 2 [null = False, max_length=1024];
required int32 controller_id = 3 [null = True];
required int32 object_id = 4 [null = False];
required string object_type = 5 [null = False, max_length=1024];
required string permission = 6 [null = False, default = "all", max_length=1024];
required string granted = 7 [content_type = "date", auto_now_add = True, max_length=1024];
required string expires = 8 [content_type = "date", null = True, max_length=1024]; }

27. XOS Generative Toolchain
GUI
REST
API
VIM
TOSCA
Generated Code
– API Tests
– Northbound Interfaces
– Enforce Security Policy
– Object Relation Mapper
– Synchronizer Framework
Event
Bus
XOS
Core DB
Sync
Sync
Sync
Sync

28. Core Models Service Slice (Resources) (Control) Service Slice
Controller
Service
Slice
Instance
(Resources)
(Control)
Controller
Service
Slice
Instance
(Resources)
(Control)
ONOS
vRouter

29. Core Models Service Graph R-CORD vOLT vSG vRouter
Controller
R-CORD
Controller
Controller
Controller
vOLT
vSG
vRouter
Service
Instance
Service
Instance
Service
Instance
Subscriber
CORD
Service Chain = At the granularity of subscribers (or subscriber classes)

30. Core Models EPC-as-a-Service eNB Composite Services Network Slicing
Controller
SPGW
Control
vMME
vSPGW-c
vSPGW-u
Controller
vHSS
eNB
Composite Services
Network Slicing

31. XOS: Another Perspective
Use Cases
(Trials)
Abstraction
Layer
XOS
Building
Blocks
VOLTHA
xRAN
Fabric
vEPC …