R2 Test strategy

Test strategy Testing is still a key challenge for OPNFV All the projects must manage their test strategy (unit, fonctional, security, performance) Several specific test projects have been validated by TSC  Define testcases  Perform tests not covered by a single project  Create tooling -Study Performance end to end

R1 In R1:  BGS managed its sanity tests  Functest created and managed 4 test suites  Octopus was in charge to run Functest at the end of a fresh installation -Pharos was in charge of providing documentation on hardware and testbeds

R2  More testcases (vIMS, ?)  Better coverage  Detect missing test domains -Analytics system

Octopus Pharos Releng VPerfYardstickQTIPTransformers Functest In R1 Note: to be discussed Common could be managed by Releng Releng DB For Result reporting/storage/analy tics Traffic profile straorea … Overview of the different projects Test projects Testbed

Project Interactions Test projects Pharos (federation of testbeds) Octopus Tests are integrated in CI through Releng scripts Releng provides Scripting/automation/DB for result reporting/storage/analytics Pharos exposes - available labs (lab topology, tooling,..) -history of test campaigns Tests are run on Pharos testbed(s) Releng provides -scripts to triggers build/deploy OPNFV solution on pharos labs -Scripting/automation/DB for installation build/deploy reporting/storage/analytics Pharos generates community dashboard (aggregation of info of build/deploy and tests) Artifact Releng DB Releng

 Is not a test project  Is a testbed management project –Develop a tool to dynamically retrieve configuration (hardware, network, tooling) –Manage history of test campaigns –Create Guides –HW Requirements –Best practices –Create a tool to manage testbed resource –Create a Dashboard (describing the available testbeds, their capabilities,…) Pharos

Releng ProjectScopeR1R1R2 deliverables Releng (Fatih Degirmenci) Common functions  Collection of test profiles/payloads,.. - Collection of results + DB + Analytics - Test Dashboard Part of Octopus, help creating CI jobs - Analytics tooling through API for any test project - traffic profile definition and collection

Test projects ProjectScopeR1R2 deliverables Functest (Morgan Richomme) Common functions for testing project -definition of default configuration for tools -definition of format for output files - coverage - Test Dashboard - new functional tests - integration of upstream suites (R1) - update of existing suites - Tempest + rally suite - ODL suite - vPing - 4 suites integrated in CI -Prescription (format, configuration,..) -Test Dashboard (API for Pharos) -Test coverage -testcase search engine - vIMS testcase - vSBC?  new testcases in upstream suite -Move from Tempest smoke to full? Vperf (Mike Lynch) Vswitch perf testingDev & tests done but not integrated in CI for v1 New tests Integration in CI Yardstick (Ana Cunha) In -VM testing Tooling Test suites - Methodology for Infrastructure verification from VNF perspective, aligned with ETSI-NFV TST001 - Generic test cases for Compute, Storage and Networking areas - OPNFV Infrastructure verification from VTC and SFC perspective - Mobility Traffic Profile as Test Stimuli Qtip (Wenjing Chu) Transformers (Michael Wiegers)

TestScopeR1R2 deliverables SystemFunctional component/subsystem/en d-to-end Functest - Tempest - ODL Functest - Tempest - ODL - VIMS? - … System testing from a user (i.e. VNF) perspective Functest - VPing Yardstick PerformanceOPNFV components VPerf Vperf OPNFV system level testing Functest (Rally)Qtip? Functest (Rally) Security? Test projects

Test Coverage by component ComponentOPNFV projectExternal tool OpenStackFunctestTempest, Rally ODLFunctest/ODLRobot KVMYardstick OVFYardstick VNFFunctest (vPing, vIMS, vXXX) Yardstick Spirent/Ixia tooling CephFunctest?, Yardstick

Test Coverage by domain DomainOPNFV projectComments Security InstallationBGS OAM ETSI domains  vEPC  vIMS  Mobile base stations  Fixed Access Network  vHGW  vCDNs  Network Functions Virtualisation IaaS  Virtual Network Function ‘aaS  Virtual Network PaaS -VNF Forwarding Graphs Functest Yardstick Is it possible to automate/integrate some of the 36 ETSI PoCs into OPNFV? OpenStack Telco WG testcaseshttps://wiki.openstack.org/wiki/T elcoWorkingGroup#Use_Case_ Definition

Common needs: Big data / analytics => Releng Problem: Thanks to CI and test projects we will get lots of results logs coming from different testbeds (pharos federation including different hardware, configuration), the idea is to perform some analytics on these tool sets to study the influence of hardware on performance The tool could include:  API for collecting the results of all the tests (from the artifact)  A way to associate a toolset and a configuration (description needed from Pharos)  A NoSQL DB to store the results  Analytic tooling  Analytics script  Display and share results

Reference Testcases => Releng + Functest Problem Each OPNFV subproject is responsible for its test strategy. Specific test projects have been created. It is difficult to get the big picture of the testcases by checking project by project. List of testcases shall be automatically generated. It shall be possible to search somewhere if testcase has already been run, in which project. It would also be interesting to have an history of the testcases (testcase X from project Y is run oin testbed Z1,Z2, ….) The task will thus mean:  Automation of production of a document collecting all the testcases  Keywords + search engine system: goal know if testcase already considered, which suite, how to automate it,.. -Template for test case documentation

Questions