Testbed and Test Projects Overview Pharos, VNF Deployment Functional Testing (Functest), Benchmarking (QTIP) Morgan Richomme, Trevor Cooper, Palani Chinnakannan, Wenjing Chu, Fu Qiao February 24, 2015

Test & Performance Sub-group Objectives and Scope OPNFV is an original project mainly based on the integration of Open Source upstream projects OPNFV main goal is to define an Open Source baseline for NFV Testing is a key activity for achieving OPNFV objectives

Test & Performance Sub-group Projects and Scope TSC approved projects: Pharos, Functest, Qtip, Characterize vSwitch Performance Testing Scope Installation/upgrade (deployment) Representative & automated VNF functional test suites Performance: Performance targets and baselines to be defined and determined (e.g. compare bare-metal) Robustness, HA, disaster recovery NVFI/VNF interoperability with legacy network architecture Security Testing with proprietary … VNFs, Test frameworks (SW infrastructure and hw tools)

Project Relationships

Test - Top Level Projects Need a platform to test? Need guidelines to setup your own test-bed? Want to know what has been already tested? Want to reproduce a test-suite on your test-bed? Pharos Need functional test suite? Want to add functional tests to existing test suite? Functest Need tools and tests to benchmark your OPNFV platform? Want to benchmark my VNF on OPNFV platform? Qtip

PHAROS

Pharos Overview The Pharos project deals with the creation of a distributed and federated NFV test capability that will be hosted by a number of companies in the OPNFV community The goals include Tracking availability/capacity of community platforms Specifying capability of different community platforms Managing timeslots and tools to perform tests Sharing the results and the best practices Supporting test campaigns of community projects (e.g. functional testing project, Qtip, BGS, Oscar,…) Automated test environment Integration with CI (test-bed independent)

Pharos Main Deliverables Comments Release Operational testing capabilities for the communities Intel & E/// test-beds already available 1 Test-bed governance / code of conduct Document if you want to set up and open your own tesbed for the community Hardware recommendation Document to help you building your first OPNFV platform Test-bed Installation guide Test-bed Administration guide 2 Best Practices / FAQ On the wiki of FAQ section to answer to any question related to the test-beds Tooling references & description Wiki page referencing the tooling and scripts for OPNFV testing (linked to Qtip project) Inventory tool Tool to give an overview of the test-bed Dashboard Tool to centralize the test campaign, inform any member of the federation and encourage reproduction of test campaigns to consolidate results 1 or 2

Pharos Demos Remote test-bed access Access environment being used by BGS Project Intel hosted environment … https://wiki.opnfv.org/get_started/intel_hosting Discuss tools / methods for remote collaboration Software Fast-path Service Quality Metrics Project focused on measurement of Telco Traffic and Performance KPIs in the fastpath Demo uses RFC2889 (Frame Error Filtering) to show MAC/PHY error and discard statistics

China Mobile OPNFV Testlab Open Source Testlab Provide at least 4 PODs for OPNFV testing Provide multiple hardware support Provide internet access for lab access Provide testing and measurement equipments Partners: Open source Platform

Open discussion 1: Test Dashboard Goals of the Dashboard Inform (who is doing what, when) Simplify reproduction of test campaigns Identify contacts / escalations Facilitate best practices (tools, scripts, …) Compare results (same campaign on different configurations) Identify covered (and non covered) domains Dashboard Contributors Community test-bed managers Testers

Release 1 Functest

Rls 1 - Assumptions and Test Steps CI deploys to Pharos compliant environment (hw infrastructure) Jenkins owned by CI (incl. installation scripts for master and slave modes) Test-bed runs Jenkins slave connected to CI environment (not for Rls 1) BGS constrains test configuration Test steps Deploy NW Spin up VNF (loop-back) Verify functionality of VNF -> ping test Return result (pass/fail)

Rls 1 - Functest Plan Task breakdown with milestones, leaders / owners Identify a simple VNF to spin up Identify Tempest test-suites (Open Stack) 5 tests identified Deliver script to CI for invoking tests Setup Tempest Method to modify and reuse test-cases (as needed) Create new tests (as needed) Setup Jenkins in test-environment (integrated with CI after Rls 1) Documentation

Rls 1 – Test Artifacts Prerequisites / hardware requirements / … Installation and user guides Test suite Community test-bed dashboard Tool for maintaining testbed inventory/capability

Community Needs

Community needs / organization How to collect community need regarding testing? Mailing list, dedicated wiki page, dashboard? Shared test-bed resources – tools to manage? How to manage requests for specific project requirements? Specific hardware Bare metal Vendor specific Hardware Acceleration / Network cards Specific configurations (stack selection, Multi sites, VSwitch tuning..) Specific test equipment (Spirent, IXIA, RobotFramework, SIPP, CLIF, ….)

Community needs / organization cont. How to share the results? Dashboard How to encourage test campaign reproduction? Any test campaign shall be automated and reproduced on any community platform Reference test suites, tooling and communicate on test campaign How to share best practices?

Thank You

Governance Keep it simple Each company may already have some internal rules, do not create new ones FAQ with a text sample shall be produced

Governance proposal This laboratory is a controlled environment made available to OPNFV community members by XXX for improving OPNFV solution and testing interoperability with other manufacturers’ equipment for the OPNFV community. This testing gives community members the advantages of working together to test interoperability. Access to the laboratory is granted by the laboratory administrator. The procedure to get credentials, the documentation (architecture, user guide), the testbed inventory (hardware, bare metal capabilities, available tooling) and the local policy (physical access, specific code of conduct) shall be described in a dedicated wiki page. Members are encouraged to share as many results as possible. The results of community tests would be collected into the Pharos dashboard. The Pharos project will disseminate the test results so any testbed of the federation could replay and consolidate test campaigns with different combination of hardware/software. Specific tests by members using proprietary components to help improving the overall quality of the OPNFV solution are possible. Members participating in such interoperability event, dry run, practice run, certification wave, wave rehearsals must not (either before, during or after the scheduled development lab use or interoperability event, dry run, practice run, certification wave, wave rehearsal or while performing daily tasks at any of XXX test development laboratories): - Access any other members’ equipment or other facilities at XXX without expressed written permission from an authorized XXX’ project manager. - Discuss any other members information and/or interoperability problems with third parties or the press without XXX’ prior written authorization. - Reveal other members’ test results, either directly or implicitly, to his/her own company, third parties or the press. - Share or copy confidential information or intellectual property other than in accordance with the applicable OPNFV policy. Testing resources reservation shall be managed at the testbed level by the testbed administrator. The TSC may re-prioritize test campaigns if necessary.