Deploy Containerized OPNFV Cluster Efficiently Using Daisy Installer –- Speed Up Image Distribution Using Multicast Zhijiang Hu hu.zhijiang@zte.com.cn Ya Zhou zhou.ya@zte.com.cn

Daisy Introduction From the user perspective: Deploy containerized OpenStack/OPNFV Web UI Cluster Configuration DB Version Manager Auto Scaling(Future Rlease)

Daisy Introduction From the developer perspective:Extensions for Kolla WebUI : Django framework Core component: daisy-api,daisy-registry,daisy-client Authentication: Using Keystone Upstream: kolla-ansible

OPNFV E Release Version Manager Maintain Kolla images as tarball package with a version id Maintain package/version in database Kolla image replication

Kolla image Replication Motivation When deploying a large OpenStack cluster, we would like to take the advantage of UDP multicast to prevent the network bottleneck caused by distributing container images from one node to all target hosts using unicast.

Kolla image Replication Question: Can the distributed docker registry server really speed up OpenStack deployment?

Kolla image Replication Test Environment Setup Dell R710, 12 Core (24 Thread) X5670 @ 2.93GHz, 128G Ram, SAS 300G * 5(Raid 5) 20 VMs, 4 CPUs, 4G Ram, 50G Disk space, OS pre-installed NIC: Intel 82576 SR-IOV

Kolla Image Replication Test Network Setup Daisy Server Control1 Compute1 Compute. ComputeN Installation Network(PXE, Mcast) Public/Private API Internet Access During Installation Tenant Network(VxLAN Tunnel) External Network Installation Network2 With SR-IOV NICs

Kolla image Replication Test case 1:Deploy 20Nodes(VMs)

Kolla image Replication Test case 1:Deploying 20VMs Multicast takes time. I/O, CPU is the bottleneck

Kolla image Replication Test case 2:Pure Multicast Performance Tests

Kolla image Replication Transmit a 4GB file to 13 VMs There is almost no difference between the cost of sending image to 1 nodes and sending files to 13 nodes in SR-IOV+Jumbo Frame scenario.

Kolla image Replication Summarize: Distribute Docker registries using multicast to is a good way to boost containerized OPNFV deployment performance in a large B/M cluster environment.

Kolla image Replication Next Step Test on BM POD Release in OPNFV Euphrates Will also cover OS installation traffic

Multicast Tool:Jasmine Description:Just A Simple Multicast engINE Usage: Server side: ./jasmines <InterfaceIP> <Number of Clients> < /path/to/read/image/file/from Client side: ./jasminec <InterfaceIP> <ServerIP> > /path/to/save/file/to (with disk I/O) ./jasminec <InterfaceIP> <ServerIP> > /dev/null (without disk I/O)

Jasmine Utilize Jasmine In Daisy4NFV 1. Daisy server still setup the monopolistic docker registry server on Daisy server as a fail-safe option. 2. Daisy server, as the jasmine server, prepares the image file to be transmitted, and count how many target hosts, as the jasmine client, should receive the image file simultaneously. 3. Jasmine clients tell the jasmine server about ready to receive the image. 4. Jasmine server transmits Kolla image tarball over UDP multicast channel. 5. Jasmine clients request re-transmission if needed. Then jasmine will re-transmit packets. 6. Jasmine clients report to jasmine server after received the whole tarball successfully. 7. Daisy server setups docker registry server on each target hosts based upon received Kolla image. 8. Daisy server setups Kolla ansible to use 127.0.0.1 as the registry server IP so that the real docker image retrieving activities will only take place inside target hosts.

Jasmine Doc/code location Spec Design Code Demo Video https://gerrit.opnfv.org/gerrit/gitweb?p=daisy.git;a=blob;f=docs/developer/spec/multicast.rst;h=ba31 4d3a1f24dd71763d2dc8b9ee45c6a188373d;hb=refs/heads/master Design https://gerrit.opnfv.org/gerrit/gitweb?p=daisy.git;a=blob;f=docs/developer/design/multicast.rst;h=89 422fe60311686252b09011efef7f370dabd128;hb=refs/heads/master Code https://gerrit.opnfv.org/gerrit/gitweb?p=daisy.git;a=tree;f=code/jasmine;h=ede3d94f2f011d9078f3c2 2c3eaa050ede8709cf;hb=refs/heads/master Demo Video http://120.24.17.215/daisy_deploy_openstack.avi

Thank you! Jasmine