R2 Test strategy

Test strategy Testing is still a key challenge for OPNFV All the projects must manage their test strategy (unit, fonctional, security, performance) Several specific test projects have been validated by TSC -Define testcases -Perform tests not covered by a single project -Create tooling -Study Performance end to end

R1 In R1: -BGS managed its sanity tests -Functest created and managed 4 test suites -Octopus was in charge to run Functest at the end of a fresh installation -Pharos was in charge of providing documentation on hardware and testbeds

R2 -More testcases (vIMS, ?) -Better coverage -Detect missing test domains -Analytics system

Pharos Common VPerfYardstickQTIPTransformers Octopus Functest In R1 Note: to be discussed Common could be managed by Functest (avoid creation of new project..) DB -Traffic profile -Results … Overview of the different projects Test projects Testbed

Project Interactions Test projects Pharos (federation of testbeds) Octopus (Continuous Integration) Test are run on Tests are integrated in CI Pharos exposes Available labs Octopus triggers build/deploy/tests on pharos labs Artifact Consumes results generated by CI Push results

 Is not a test project  Is a testbed management project –Develop a tool to dynamically retrieve configuration (hardware, network, tooling) –Manage history of test campaigns –Create Guides –HW Requirements –Best practices –Create a tool to manage testbed resource –Create a Dashboard (describing the available testbeds, their capabilities,…) Pharos

Test projects ProjectScopeR1R2 deliverables CommonCommon functions for testing project -definition of default configuration for tools -definition of format for output files - Collection of test profiles/payloads,.. - coverage - Collection of results + DB + Analytics - Test Dashboard -Prescription (format, configuration,..) -Test Dashboard – API for Pharos – Test coverage – testcase search engine - Analytics tooling through API for any test project - traffic profile definition and collection Functest (Morgan Richomme) - new functional tests - integration of upstream suites (R1) - update of existing suites - Tempest + rally suite - ODL suite - vPing - 4 suites integrated in CI - vIMS testcase - vSBC? -new testcases in upstream suite -Move from Tempest smoke to full? Vperf (Mike Lynch) Vswitch perf testingDev & tests done but not integrated in CI for v1 New tests Integration in CI Yardstick (Ana Cunha) In -VM testing Tooling Test suites - Methodology for Infrastructure verification from VNF perspective, aligned with ETSI-NFV TST001 - Generic test cases for Compute, Storage and Networking areas - OPNFV Infrastructure verification from VTC and SFC perspective - Mobility Traffic Profile as Test Stimuli Qtip (Wenjing Chu) Transformers (Michael Wiegers) Note: For R2 Functest (Functional & Common Testing) will manage common part

Test Coverage by component ComponentOPNFV projectExternal tool OpenStackFunctestTempest, Rally ODLFunctest/ODLRobot KVMYardstick OVFYardstick VNFFunctest (vPing, vIMS, vXXX) Yardstick Spirent/Ixia tooling CephFunctest?, Yardstick

Test Coverage by domain DomainOPNFV projectComments Security InstallationBGS OAM ETSI domains -vEPC -vIMS -Mobile base stations -Fixed Access Network -vHGW -vCDNs -Network Functions Virtualisation IaaS -Virtual Network Function ‘aaS -Virtual Network PaaS -VNF Forwarding Graphs Functest Yardstick Is it possible to automate/integrate some of the 36 ETSI PoCs into OPNFV? OpenStack Telco WG testcases g/wiki/TelcoWorkingGrou p#Use_Case_Definition

Common needs: Big data / analytics Problem: Thanks to CI and test projects we will get lots of results logs coming from different testbeds (pharos federation including different hardware, configuration), the idea is to perform some analytics on these tool sets to study the influence of hardware on performance The tool could include: -API for collecting the results of all the tests (from the artifact) -A way to associate a toolset and a configuration (description needed from Pharos) -A NoSQL DB to store the results -Analytic tooling -Analytics script -Display and share results

Reference Testcases Problem Each OPNFV subproject is responsible for its test strategy. Specific test projects have been created. It is difficult to get the big picture of the testcases by checking project by project. List of testcases shall be automatically generated. It shall be possible to search somewhere if testcase has already been run, in which project. It would also be interesting to have an history of the testcases (testcase X from project Y is run oin testbed Z1,Z2, ….) The task will thus mean: -Automation of production of a document collecting all the testcases -Keywords + search engine system: goal know if testcase already considered, which suite, how to automate it,.. -Template for test case documentation

Questions