OPNFV testing strategy Testing working group 23/06/2017 Reporter for the testing working group Gabriel (Bottlenecks) Morgan (Functest)

Stakes We (test projects) mainly validate a release so far Tooling for stress/resiliency/robustness testing is available in the different frameworks but under used Stress tests have been introduced in Danube (but not in CI) Stress / robustness / resiliency / long duration tests are key

Towards a stress test strategy Community discussions (etherpad, mail) https://wiki.opnfv.org/display/bottlenecks/Sress+Testing+over+OPNFV+Platform

Towards a stress test strategy 1. Test Cases Discussed in Danube Categories Test Case Data-plane Traffic for a virtual or bare metal POD TC1 –Determine baseline for throughput TC2 - Determine baseline for CPU limit Life-cycle Events for VM pairs/stacks TC3 – Perform life-cycle events for ping TC4 – Perform life-cycle events for throughput TC5 – Perform life-cycle events for CPU limit 2. Test Cases Planning in Euphrates (Under Discussion) Categories Test Case Scaling Yardstick Scale-out test Scale-up test Compute & Memory VSPerf & StorPerf Set up VMs until maximum throughput reached Test of different Nova schedulers for different compute nodes Run VSPERF and record numbers (throughput, latency, etc.) Run StorPerf and record numbers (throughput, latency, etc.) Run both at the same time and compare numbers. Cooperation with Bottlenecks Project as load manager for the planned test cases is also under discussion

Towards a stress test strategy 2 Test Cases Implememnted in Danube TC1 - Determine baseline for throughput TC3 - Perform life-cycle events for ping

Towards a stress test strategy TC3 measures the reliability/stability of the system under large number of concurrent requests/traffic. Problem detected on OPNFV & commercial solutions

Cross Project Stress Some Rough Numbers for General View VSPERF: Throughput: 500 mbit/s StorPerf: Bandwidth: 500 mbit/s Combined Tenant / Storage Network VSPERF: Throughput: 1000 mbit/s StorPerf: Bandwidth: 1000 mbit/s Tenant Network Storage Network Bottlenecks could act like load manager and monitoring system behaviors

Theory versus reality

Going beyond release verification Impossible to include stress / robustness tests in CI chain because target version delivered late and instable due to proximity to upstream 6 months cadence...(assuming that we would like to perform tests over weeks)

Needs 2 CI chains (as it is today…) CI chain master => release validation (as it is today, i.e. deploy/functest/yardstick looping on different scenarios) CI chain « stable » Focus on « generic » scenarios first (starting with os-nosdn-nofeature-ha) Reinstallation on demand (CI still needed for clean reinstallation but reinstallation not automated to allow long duration tests + troubleshooting) Schedule to be created by the testing group (allocate N week for project X, N’ for project N’...)

Questions for the TSC Any feedback/comments ? How to manage that from a release perspective...release management process will be incomplete as testing group would be able to work in // on 2 releases (decoupling release validation and release qualification) Quid on any commitment from the installers can the installer deal with 2 versions in // ? Shall we directly work on bifrost/openstack-ansible ? Is it compatible with Infra resource management (Infra group) ? Testing working group is elaborating its stress strategy, shall it eb validated by TSC ?