Isasku, Srini, Alex, Ramki, Seshu, Bin Hu, Munish, Gil, Victor Kubernetes Support for VM & Container based VNFs - Request to approve Phase 1 in Casablanca Isasku, Srini, Alex, Ramki, Seshu, Bin Hu, Munish, Gil, Victor

ONAP – Support for K8S based remote regions APP Orchestrators OSS BSS Current support: Openstack based remote Clouds, Support multiple Openstack variations – Windriver Titanium, VMWare VIO, Native Newton, Ocata. Only VM based VNFs. ONAP Multi Cloud Service SDNC (Fabric/WAN Control) Need support for containerized VNFs. Need support where K8S is used for both VMs and Containers (To avoid multiple controls and also to utilize same compute nodes for both VMs and containers) Site (With K8S for both VMs and Containers) Site (With Openstack VIM) Site (*) (Two logical edges – Openstack for VMs and K8S for containers)

ONAP – VM and Container networking (Uniform) APP Orchestrators OSS BSS Current: Networks are created and VM are placed in various networks (VLs and CPs – Virtual Links and Connection Points). ONAP Multi Cloud Service SDNC (Fabric Control) Need: Some workloads can be containers and networking is expected to be extended to containers Site (edge/cloud) POD POD VM VM C C C K8S Master Network1 (VLAN/VXLAN/GENEVE) Network Controller Network N (VLAN/VXLAN/GENEVE) NFVI/VMM vSwitch (e.g OVSDPDK) networking SRIOV networking Compute Servers – Hardware (CPU/Memory/SRIOV-NIC/FPGA/GPU) etc…

ONAP – VM and Container Storage (Uniform) APP Orchestrators OSS BSS Volumes are expected to be extended to Containers. ONAP Multi Cloud Service SDNC (Fabric Control) Site – Edge/Cloud K8S Master POD POD VM VM Network Controller C C C NFVI/VMM Ceph Storage Compute Servers – Hardware (CPU/Memory/SRIOV-NIC/FPGA/GPU) etc…

ONAP – Service Orchestration language (Model Driven – Model being TOSCA based) NSD Current (Being defined in Modeling committee) - Standardizing on TOSCA based Service description Moving away from Openstack HOT templates. Multi-Cloud OS plugins are expected to convert TOSCA VNFC information to Openstack API Child NSD VNFD VNFD VL VL Need: Continue with TOSCA based Service description for VMs and containers managed by K8S. Multi-Cloud K8S plugin convert from VNFC information K8S API at run time. Multi Cloud to convert from VL and CP to Network controller (eg. OVN) (No K8S Yaml, Helm in the 1st phase) VNFC/VDU (VM based) VNFC/VDU VM VNFC/VDU (POD) VNFC/VDU (POD) Image Compute Storage Network HPA Monitoring Scaling User data C1 C2 Image Compute, Storage Network, HPA Monitoring, Scaling User data

Current flow - HEAT based Service Orchestration A Service consists of multiple HOTs Each HOT is a VNF, set of VLs and CPs NSD Instantiation time Service instantiation request SO decomposes service to VNFs. For each VNF Requests OOF to get best site and flavors (for each VDU) to place VNF. OOF with the help of A&AI and Policy, identifies best site and flavors for each VDU Responds to SO. SO replaces flavors in HOT Requests Multi-Cloud service (with HOT template as input) to bring VMs in remote site. Not shown: SO & Multi-Cloud store Virtual inventory information in A&AI. OOF and Multi-Cloud interaction wrt capacity checks and SDN-C interaction from SO. 1 3 SO OOF 2 5 4 AAI Policy frame work 6 Multi-Cloud Service Remote Cloud regions

NSD Instantiation time TOSCA based workflow (Simplified) – Discussions in progress in community A service represented as NSD (a nested NSD) A NSD can contain multiple VNFs. Each VNF can contain multiple VDUs NSD Instantiation time 1 Service instantiation request SO decomposes NSD to child NSDs, decompose individual VNFs of each NSD, decompose to VNFCs for each VNF (uses TOSCA parser) For each VL and VNFC in VNFs Requests OOF to get best site and flavor. OOF with the help of A&AI and Policy, identifies best site and flavors Responds to SO. SO replaces HPA information from the flavor data. Requests Multi-Cloud service (with VNFC TOSCA fragment as input) to bring VLs, Volumes and workloads in remote site. 3 SO OOF 2 5 4 AAI Policy frame work 6 Multi-Cloud Service Remote Cloud regions

Phase 1 Scope - Focus on Multi-Cloud Service Multi- Cloud Service changes Test with VMs and containers. Test using scripts on top of Multi-Cloud Service(simulating SO actions) Identify TOSCA modeling changes. Provide feedback to bigger architecture to support K8S. Testing DP Container: DPDK based Router IOT infrastructure: EdgeXFoundry services as containers Ensure that both VM and containers can exist on same network and works when placed on same compute node. Model Driven API: TOSCA fragments in API API categories : Compute, Storage and Networks. K8S Plugin (Go lang) Converts TOSCA to K8S API data – Deployment, Service, Endpoint Converts TOSCA VL fragment to OVN API, SRIOV API, flannel API Uses OPNFV TOSCA parser (Python Service – Modifications are needed) – Eventually use External TOSCA service that is being considered as a separate project in ONAP. Site (Mostly deployment related challenges) Virtlet for VMs, Dockers for Containers. OVN for networking NSD Instantiation time Test Service (Simulating SO) Multi Cloud Service Existing API Model Driven API Openstack Plugins – VIO, Titanium, OS ocata & newton K8S Plugin + OVN Plugin TOSCA parser VFW VM Router Cntnr EdgeX K8S Master OVN N Controller Minion Docker, virtlet, OVN S Controller, OVS-DPDK Site - Edge/Cloud

Asks from Architecture Subcommittee Approve Phase 1 Approve repository creation Next steps after approval: Detailed architecture specifications – Model Driven API, K8S+OVN Plugin, TOSCA Parser Document test cases Work with Multi-Cloud PTL/Team Request contributors and create JIRA EPICs/stories.

Items that came up in various meetings – For Discussions Co-existence of VMs and containers in NSD and granularity Would a VNF contain all VMs or container or is the mix needed? Would VMs and container can come up on same compute node? TOSCA Orchestration Support for TOSCA capabilities to represent container workloads Support for Helm charts for VNFC Support for POD deployment spec for VNFC Networking: ONAP controlling network across VMs and containers – Which CNI? Multi-Cloud plugin to program CNI Placement decisions (Should not bypass OOF) Taking advantage of SO and OOF for placement decisions. Can SO decompose NSD to VNFC level? (Some work to be done at SO and OOF) Does SO call Multi-Cloud VIM for each VNFC? It needs to be for container support (Yes) Would all workloads in a VNF in one location or would there be requirement to put them in different regions? Overlapping features between K8S and ONAP - Monitoring, Replication, Liveness checks etc.. Use ONAP, Use K8S? Bare-metal containers – Yes Containers in VMs – Topics for discussions Support for other Container orchestration systems (Dockerswarm) & Orchestration management systems (Openshift, Rancher 2.0 etc…) Others???