Nikolas Hermanns Jose Lausuch Testing, CI Gating & Community Fast Feedback: The Challenge of Integration Projects Nikolas Hermanns Jose Lausuch

Outline Challenges of the SDNVPN project XCI patchset verification Virtual testing setup Functest test challenge

Outline Challenges of the SDNVPN project Consume ODL too late Functest challenges – a change happens in every scenario for every installer Overview about Fast Feedback – per patch-set Show setup with 6 virtual cluster running in parallel. Snapshot concept – low disk io Cpu isolation Network drawing Screenshots

Integration Project (e.g. SDNVPN) Consume many different Project Different release dates Different Bug tracking Different source code handling Different programming language Etc. SDNVPN ODL Fuel OVS Quagga Apex

Challenge of Integration Projects (e.g. ODL in OPNFV) Bug1 Bug2 Bug 1 found and solved Bug 2 found but not valid for master anymore Bug 2 was not worth the effort Bug 3 will not be found until next release Bug1 Bug2 OPNFV-Integration Bug finding ODL-Master ODL-Stable Bug3 x ODL-Master And yes odl is a great product but as any other project it has bugs But lets assume there are only 2 bugs in the release (I know hard assumption) Bug1 easy one change on stable and on master Bug2 Only valid on stable time ODL-Release ODL-Release ODL is consumed too late! Vendor integration Harden Test Solution integration and testing Community development In-house development

Challenge of Integration Projects (e.g. ODL in OPNFV) Dev A expert in Part of ODL Int A expert in Fuel but doesn’t know much about ODL code base Int A needs to find and understand the bug! Int A need to talk to Dev A and explain the found bug Might be Dev A does not even remember Developer A Integrator A Bug 1 Bug1 time ODL-Release ODL-Release

Solutions Solution 1: More test cases in consumed project Integrator needs to convince and explain the test to Developer Test case might be too E2Eish to be created in consumed project Solution 2: XCI patchset verification Automated integration (Consume latest state) Fast feedback (reducing from months to < 1 hour) No interaction from integrator

Feedback Installer Project e.g. Apex New Patchset Snapshot ODL OPNFV New Patchset (Commit) Positive -> Merge commit Negative -> Prevent merge Investigate the logs/Find bugs Create Virtual Testing Platform Use standard testing framework e.g. Functest Build ODL artifact Unit test Download/Install new artifact Feedback 5 min 10 min 15 min 35 min

x6 RAM and CPU limiting factor Bare Metal (48 threads/256 GB RAM/500 GB SSD) Thread pool – 8 vCPU 10 GB Diskspace 40 GB RAM Own OVS networks vJenkins slave vController vCompute vController vCompute x6 vController vCompute vCompute vJenkins slave vController vCompute RAM and CPU limiting factor Very low disk IO cause no installation process ~2 mins until Openstack/ODL services up Easy exchange of snapshot vController vCompute vController vCompute vCompute Snapshot Controller SnapshotCompute ~ 100 GB Diskspace

Using the Snapshot Using the base concept of qemu-img vController >1GB disk consumption per vHost vController vController Using the base concept of qemu-img vController vController SnapshotController ~ 100 GB Diskspace

Jenkins-Slave running CPU pinning vCompute 4 3 8 vCPUs ? -> YES! – sharing is the key. 1 vController Under-provisioned vJenkins slave Start vCmp vCtrl Testing started Testing more API CPU-Load 8 vCPU Jenkins-Slave running Started vCmp vCtrl time

Cpu pinning Other vPod do not get effected

Networking OVS for complete isolation internet Bare metal host vJenkins slave vController vCompute OVS for complete isolation vPods using same IPs so every vPod has its own networks Jenkins-slave for Gateway doing NAT NAT admin storage private public host

x6 RAM and CPU limiting factor Bare Metal (48 threads/256 GB RAM/500 GB SSD) Thread pool – 8 vCPU 10 GB Diskspace 40 GB RAM Own OVS networks vJenkins slave vController vCompute vController vCompute x6 vController vCompute vCompute vJenkins slave vController vCompute RAM and CPU limiting factor Very low disk IO cause no installation process ~2 mins until Openstack/ODL services up Easy exchange of snapshot vController vCompute vController vCompute vCompute Snapshot Controller SnapshotCompute ~ 100 GB Diskspace

Example of application: Functest test new code on OPNFV deployment pep8 docs pylint unit tests py35 py27

Example of application: Functest Functest bug might break CI Needs a way to test each change before merging Fast feedback to the commit owner Easy to validate framework changes on Functest Can be tested on a basic nosdn-nofeature scenario

Example of application: Functest Challenges: Changes on Feature tests  need specific scenario deployed Increase verification time of each patchset ( + time to build a docker image) Identify where the change is (framework, test case, …) to trigger an appropriate test case

Thank You Nikolas Hermanns Nikolas.Hermanns@ericsson.com Jose Lausuch jose.lausuch@ericsson.com