2. Nikolas Hermanns Jose Lausuch
Testing, CI Gating & Community Fast Feedback: The Challenge of Integration Projects
Nikolas Hermanns
Jose Lausuch

3. Outline Challenges of the SDNVPN project XCI patchset verification
Virtual testing setup
Functest test challenge

4. Outline Challenges of the SDNVPN project
Consume ODL too late
Functest challenges – a change happens in every scenario for every installer
Overview about Fast Feedback – per patch-set
Show setup with 6 virtual cluster running in parallel.
Snapshot concept – low disk io
Cpu isolation
Network drawing
Screenshots

5. Integration Project (e.g. SDNVPN)
Consume many different Project
Different release dates
Different Bug tracking
Different source code handling
Different programming language
Etc.
SDNVPN
ODL
Fuel
OVS
Quagga
Apex

6. Challenge of Integration Projects (e.g. ODL in OPNFV)
Bug1
Bug2
Bug 1 found and solved
Bug 2 found but not valid for master anymore
Bug 2 was not worth the effort
Bug 3 will not be found until next release
Bug1
Bug2
OPNFV-Integration
Bug finding
ODL-Master
ODL-Stable
Bug3 x
ODL-Master
And yes odl is a great product but as any other project it has bugs
But lets assume there are only 2 bugs in the release (I know hard assumption)
Bug1 easy one change on stable and on master
Bug2 Only valid on stable
time
ODL-Release
ODL-Release
ODL is consumed too late!
Vendor integration
Harden Test
Solution integration and testing
Community development
In-house development

7. Challenge of Integration Projects (e.g. ODL in OPNFV)
Dev A expert in Part of ODL
Int A expert in Fuel but doesn’t know much about ODL code base
Int A needs to find and understand the bug!
Int A need to talk to Dev A and explain the found bug
Might be Dev A does not even remember
Developer A
Integrator A
Bug 1
Bug1
time
ODL-Release
ODL-Release

8. Solutions Solution 1: More test cases in consumed project
Integrator needs to convince and explain the test to Developer
Test case might be too E2Eish to be created in consumed project
Solution 2: XCI patchset verification
Automated integration (Consume latest state)
Fast feedback (reducing from months to < 1 hour)
No interaction from integrator

9. Feedback Installer Project e.g. Apex New Patchset
Snapshot
ODL
OPNFV
New Patchset
(Commit)
Positive -> Merge commit
Negative -> Prevent merge Investigate the logs/Find bugs
Create Virtual Testing Platform
Use standard testing framework e.g. Functest
Build ODL artifact
Unit test
Download/Install new artifact
Feedback
5 min
10 min
15 min
35 min

10. x6 RAM and CPU limiting factor
Bare Metal (48 threads/256 GB RAM/500 GB SSD)
Thread pool – 8 vCPU
10 GB Diskspace
40 GB RAM
Own OVS networks
vJenkins slave
vController
vCompute
vController
vCompute x6
vController
vCompute
vCompute
vJenkins slave
vController
vCompute
RAM and CPU limiting factor
Very low disk IO cause no installation process
~2 mins until Openstack/ODL services up
Easy exchange of snapshot
vController
vCompute
vController
vCompute
vCompute
Snapshot Controller
SnapshotCompute
~ 100 GB Diskspace

11. Using the Snapshot Using the base concept of qemu-img vController
>1GB disk consumption per vHost
vController
vController
Using the base concept of qemu-img
vController
vController
SnapshotController
~ 100 GB Diskspace

12. Jenkins-Slave running
CPU pinning
vCompute 4
3 8 vCPUs ? -> YES! – sharing is the key. 1
vController
Under-provisioned
vJenkins slave
Start vCmp vCtrl
Testing started
Testing more API
CPU-Load
8 vCPU
Jenkins-Slave running
Started vCmp vCtrl
time

13. Cpu pinning
Other vPod do not get effected

14. Networking OVS for complete isolation
internet
Bare metal host
vJenkins slave
vController
vCompute
OVS for complete isolation
vPods using same IPs so every vPod has its own networks
Jenkins-slave for Gateway doing NAT
NAT
admin
storage
private
public
host

15. x6 RAM and CPU limiting factor
Bare Metal (48 threads/256 GB RAM/500 GB SSD)
Thread pool – 8 vCPU
10 GB Diskspace
40 GB RAM
Own OVS networks
vJenkins slave
vController
vCompute
vController
vCompute x6
vController
vCompute
vCompute
vJenkins slave
vController
vCompute
RAM and CPU limiting factor
Very low disk IO cause no installation process
~2 mins until Openstack/ODL services up
Easy exchange of snapshot
vController
vCompute
vController
vCompute
vCompute
Snapshot Controller
SnapshotCompute
~ 100 GB Diskspace

16. Example of application: Functest
test new code on OPNFV
deployment
pep8
docs
pylint
unit tests
py35
py27

17. Example of application: Functest
Functest bug might break CI
Needs a way to test each change before merging
Fast feedback to the commit owner
Easy to validate framework changes on Functest
Can be tested on a basic nosdn-nofeature scenario

18. Example of application: Functest
Challenges:
Changes on Feature tests  need specific scenario deployed
Increase verification time of each patchset ( + time to build a docker image)
Identify where the change is (framework, test case, …) to trigger an appropriate test case

19. Thank You
Nikolas Hermanns Jose Lausuch