1. Isasku, Srini, Alex, Ramki, Seshu, Bin Hu, Munish, Gil, Victor
Kubernetes Support for VM & Container based VNFs - Request to approve Phase 1 in Casablanca
Isasku, Srini, Alex, Ramki, Seshu, Bin Hu, Munish, Gil, Victor

2. ONAP – Support for K8S based remote regions
APP Orchestrators
OSS
BSS
Current support: Openstack based remote Clouds, Support multiple Openstack variations – Windriver Titanium, VMWare VIO, Native Newton, Ocata. Only VM based VNFs.
ONAP
Multi Cloud Service
SDNC
(Fabric/WAN Control)
Need support for containerized VNFs.
Need support where K8S is used for both VMs and Containers (To avoid multiple controls and also to utilize same compute nodes for both VMs and containers)
Site
(With K8S for both VMs and Containers)
Site
(With Openstack VIM)
Site (*)
(Two logical edges – Openstack for VMs and K8S for containers)

3. ONAP – VM and Container networking (Uniform)
APP Orchestrators
OSS
BSS
Current: Networks are created and VM are placed in various networks (VLs and CPs – Virtual Links and Connection Points).
ONAP
Multi Cloud Service
SDNC
(Fabric Control)
Need: Some workloads can be containers and networking is expected to be extended to containers
Site (edge/cloud)
POD
POD VM VM C C C
K8S Master
Network1 (VLAN/VXLAN/GENEVE)
Network Controller
Network N (VLAN/VXLAN/GENEVE)
NFVI/VMM
vSwitch (e.g OVSDPDK) networking
SRIOV networking
Compute Servers – Hardware (CPU/Memory/SRIOV-NIC/FPGA/GPU) etc…

4. ONAP – VM and Container Storage (Uniform)
APP Orchestrators
OSS
BSS
Volumes are expected to be extended to Containers.
ONAP
Multi Cloud Service
SDNC
(Fabric Control)
Site – Edge/Cloud
K8S Master
POD
POD VM VM
Network Controller C C C
NFVI/VMM
Ceph Storage
Compute Servers – Hardware (CPU/Memory/SRIOV-NIC/FPGA/GPU) etc…

5. ONAP – Service Orchestration language (Model Driven – Model being TOSCA based)
NSD
Current (Being defined in Modeling committee) - Standardizing on TOSCA based Service description
Moving away from Openstack HOT templates.
Multi-Cloud OS plugins are expected to convert TOSCA VNFC information to Openstack API
Child NSD
VNFD
VNFD VL VL
Need:
Continue with TOSCA based Service description for VMs and containers managed by K8S.
Multi-Cloud K8S plugin convert from VNFC information K8S API at run time.
Multi Cloud to convert from VL and CP to Network controller (eg. OVN)
(No K8S Yaml, Helm in the 1st phase)
VNFC/VDU
(VM based)
VNFC/VDU VM
VNFC/VDU
(POD)
VNFC/VDU
(POD)
Image
Compute
Storage
Network
HPA
Monitoring
Scaling
User data C1 C2
Image
Compute, Storage
Network, HPA
Monitoring, Scaling
User data

6. Current flow - HEAT based Service Orchestration
A Service consists of multiple HOTs
Each HOT is a VNF, set of VLs and CPs
NSD Instantiation time
Service instantiation request
SO decomposes service to VNFs.
For each VNF
Requests OOF to get best site and flavors (for each VDU) to place VNF.
OOF with the help of A&AI and Policy, identifies best site and flavors for each VDU
Responds to SO. SO replaces flavors in HOT
Requests Multi-Cloud service (with HOT template as input) to bring VMs in remote site.
Not shown: SO & Multi-Cloud store Virtual inventory information in A&AI. OOF and Multi-Cloud interaction wrt capacity checks and SDN-C interaction from SO. 1 3 SO
OOF 2 5 4
AAI
Policy frame work 6
Multi-Cloud Service
Remote Cloud regions

7. NSD Instantiation time
TOSCA based workflow (Simplified) – Discussions in progress in community
A service represented as NSD (a nested NSD)
A NSD can contain multiple VNFs.
Each VNF can contain multiple VDUs
NSD Instantiation time 1
Service instantiation request
SO decomposes NSD to child NSDs, decompose individual VNFs of each NSD, decompose to VNFCs for each VNF (uses TOSCA parser)
For each VL and VNFC in VNFs
Requests OOF to get best site and flavor.
OOF with the help of A&AI and Policy, identifies best site and flavors
Responds to SO. SO replaces HPA information from the flavor data.
Requests Multi-Cloud service (with VNFC TOSCA fragment as input) to bring VLs, Volumes and workloads in remote site. 3 SO
OOF 2 5 4
AAI
Policy frame work 6
Multi-Cloud Service
Remote Cloud regions

8. Phase 1 Scope - Focus on Multi-Cloud Service
Multi- Cloud Service changes
Test with VMs and containers.
Test using scripts on top of Multi-Cloud Service(simulating SO actions)
Identify TOSCA modeling changes.
Provide feedback to bigger architecture to support K8S.
Testing
DP Container: DPDK based Router
IOT infrastructure: EdgeXFoundry services as containers
Ensure that both VM and containers can exist on same network and works when placed on same compute node.
Model Driven API:
TOSCA fragments in API
API categories : Compute, Storage and Networks.
K8S Plugin (Go lang)
Converts TOSCA to K8S API data – Deployment, Service, Endpoint
Converts TOSCA VL fragment to OVN API, SRIOV API, flannel API
Uses OPNFV TOSCA parser (Python Service – Modifications are needed) – Eventually use External TOSCA service that is being considered as a separate project in ONAP.
Site (Mostly deployment related challenges)
Virtlet for VMs, Dockers for Containers.
OVN for networking
NSD Instantiation time
Test Service (Simulating SO)
Multi Cloud Service
Existing API
Model Driven API
Openstack Plugins – VIO, Titanium, OS ocata & newton
K8S Plugin + OVN Plugin
TOSCA parser
VFW VM
Router Cntnr
EdgeX
K8S Master
OVN N Controller
Minion Docker, virtlet, OVN S Controller, OVS-DPDK
Site - Edge/Cloud

9. Asks from Architecture Subcommittee
Approve Phase 1
Approve repository creation
Next steps after approval:
Detailed architecture specifications – Model Driven API, K8S+OVN Plugin, TOSCA Parser
Document test cases
Work with Multi-Cloud PTL/Team
Request contributors and create JIRA EPICs/stories.

11. Items that came up in various meetings – For Discussions
Co-existence of VMs and containers in NSD and granularity
Would a VNF contain all VMs or container or is the mix needed?
Would VMs and container can come up on same compute node?
TOSCA Orchestration
Support for TOSCA capabilities to represent container workloads
Support for Helm charts for VNFC
Support for POD deployment spec for VNFC
Networking:
ONAP controlling network across VMs and containers – Which CNI?
Multi-Cloud plugin to program CNI
Placement decisions (Should not bypass OOF)
Taking advantage of SO and OOF for placement decisions. Can SO decompose NSD to VNFC level? (Some work to be done at SO and OOF)
Does SO call Multi-Cloud VIM for each VNFC? It needs to be for container support (Yes)
Would all workloads in a VNF in one location or would there be requirement to put them in different regions?
Overlapping features between K8S and ONAP - Monitoring, Replication, Liveness checks etc.. Use ONAP, Use K8S?
Bare-metal containers – Yes
Containers in VMs – Topics for discussions
Support for other Container orchestration systems (Dockerswarm) & Orchestration management systems (Openshift, Rancher etc…)
Others???