1. Testbed and Test Projects Overview
Pharos, VNF Deployment Functional Testing (Functest), Benchmarking (QTIP)
Morgan Richomme, Trevor Cooper, Palani Chinnakannan, Wenjing Chu, Fu Qiao
February 24, 2015

2. Test & Performance Sub-group Objectives and Scope
OPNFV is an original project mainly based on the integration of Open Source upstream projects
OPNFV main goal is to define an Open Source baseline for NFV
Testing is a key activity for achieving OPNFV objectives

3. Test & Performance Sub-group Projects and Scope
TSC approved projects: Pharos, Functest, Qtip, Characterize vSwitch Performance
Testing Scope
Installation/upgrade (deployment)
Representative & automated VNF functional test suites
Performance: Performance targets and baselines to be defined and determined (e.g. compare bare-metal)
Robustness, HA, disaster recovery
NVFI/VNF interoperability with legacy network architecture
Security
Testing with proprietary …
VNFs, Test frameworks (SW infrastructure and hw tools)

4. Project Relationships

5. Test - Top Level Projects
Need a platform to test?
Need guidelines to setup your own test-bed?
Want to know what has been already tested?
Want to reproduce a test-suite on your test-bed?
Pharos
Need functional test suite?
Want to add functional tests to existing test suite?
Functest
Need tools and tests to benchmark your OPNFV platform?
Want to benchmark my VNF on OPNFV platform?
Qtip

6. PHAROS

7. Pharos Overview
The Pharos project deals with the creation of a distributed and federated NFV test capability that will be hosted by a number of companies in the OPNFV community
The goals include
Tracking availability/capacity of community platforms
Specifying capability of different community platforms
Managing timeslots and tools to perform tests
Sharing the results and the best practices
Supporting test campaigns of community projects (e.g. functional testing project, Qtip, BGS, Oscar,…)
Automated test environment
Integration with CI (test-bed independent)

8. Pharos Main Deliverables
Comments
Release
Operational testing capabilities for the communities
Intel & E/// test-beds already available 1
Test-bed governance / code of conduct
Document if you want to set up and open your own tesbed for the community
Hardware recommendation
Document to help you building your first OPNFV platform
Test-bed Installation guide
Test-bed Administration guide 2
Best Practices / FAQ
On the wiki of FAQ section to answer to any question related to the test-beds
Tooling references & description
Wiki page referencing the tooling and scripts for OPNFV testing (linked to Qtip project)
Inventory tool
Tool to give an overview of the test-bed
Dashboard
Tool to centralize the test campaign, inform any member of the federation and encourage reproduction of test campaigns to consolidate results
1 or 2

9. Pharos Demos Remote test-bed access
Access environment being used by BGS Project
Intel hosted environment …
Discuss tools / methods for remote collaboration
Software Fast-path Service Quality Metrics
Project focused on measurement of Telco Traffic and Performance KPIs in the fastpath
Demo uses RFC2889 (Frame Error Filtering) to show MAC/PHY error and discard statistics

10. China Mobile OPNFV Testlab
Open Source Testlab
Provide at least 4 PODs for OPNFV testing
Provide multiple hardware support
Provide internet access for lab access
Provide testing and measurement equipments
Partners:
Open source Platform

11. Open discussion 1: Test Dashboard
Goals of the Dashboard
Inform (who is doing what, when)
Simplify reproduction of test campaigns
Identify contacts / escalations
Facilitate best practices (tools, scripts, …)
Compare results (same campaign on different configurations)
Identify covered (and non covered) domains
Dashboard Contributors
Community test-bed managers
Testers

12. Release 1 Functest

13. Rls 1 - Assumptions and Test Steps
CI deploys to Pharos compliant environment (hw infrastructure)
Jenkins owned by CI (incl. installation scripts for master and slave modes)
Test-bed runs Jenkins slave connected to CI environment (not for Rls 1)
BGS constrains test configuration
Test steps
Deploy NW
Spin up VNF (loop-back)
Verify functionality of VNF -> ping test
Return result (pass/fail)

14. Rls 1 - Functest Plan Task breakdown with milestones, leaders / owners
Identify a simple VNF to spin up
Identify Tempest test-suites (Open Stack)
5 tests identified
Deliver script to CI for invoking tests
Setup Tempest
Method to modify and reuse test-cases (as needed)
Create new tests (as needed)
Setup Jenkins in test-environment (integrated with CI after Rls 1)
Documentation

15. Rls 1 – Test Artifacts Prerequisites / hardware requirements / …
Installation and user guides
Test suite
Community test-bed dashboard
Tool for maintaining testbed inventory/capability

16. Community Needs

17. Community needs / organization
How to collect community need regarding testing?
Mailing list, dedicated wiki page, dashboard?
Shared test-bed resources – tools to manage?
How to manage requests for specific project requirements?
Specific hardware
Bare metal
Vendor specific
Hardware Acceleration / Network cards
Specific configurations (stack selection, Multi sites, VSwitch tuning..)
Specific test equipment (Spirent, IXIA, RobotFramework, SIPP, CLIF, ….)

18. Community needs / organization cont.
How to share the results?
Dashboard
How to encourage test campaign reproduction?
Any test campaign shall be automated and reproduced on any community platform
Reference test suites, tooling and communicate on test campaign
How to share best practices?

19. Thank You

20. Governance Keep it simple
Each company may already have some internal rules, do not create new ones
FAQ with a text sample shall be produced

21. Governance proposal
This laboratory is a controlled environment made available to OPNFV community members by XXX for improving OPNFV solution and testing interoperability with other manufacturers’ equipment for the OPNFV community.
This testing gives community members the advantages of working together to test interoperability.
Access to the laboratory is granted by the laboratory administrator. The procedure to get credentials, the documentation (architecture, user guide), the testbed inventory (hardware, bare metal capabilities, available tooling) and the local policy (physical access, specific code of conduct) shall be described in a dedicated wiki page.
Members are encouraged to share as many results as possible. The results of community tests would be collected into the Pharos dashboard. The Pharos project will disseminate the test results so any testbed of the federation could replay and consolidate test campaigns with different combination of hardware/software.
Specific tests by members using proprietary components to help improving the overall quality of the OPNFV solution are possible. Members participating in such interoperability event, dry run, practice run, certification wave, wave rehearsals must not (either before, during or after the scheduled development lab use or interoperability event, dry run, practice run, certification wave, wave rehearsal or while performing daily tasks at any of XXX test development laboratories):
- Access any other members’ equipment or other facilities at XXX without expressed written permission from an authorized XXX’ project manager.
- Discuss any other members information and/or interoperability problems with third parties or the press without XXX’ prior written authorization.
- Reveal other members’ test results, either directly or implicitly, to his/her own company, third parties or the press.
- Share or copy confidential information or intellectual property other than in accordance with the applicable OPNFV policy.
Testing resources reservation shall be managed at the testbed level by the testbed administrator. The TSC may re-prioritize test campaigns if necessary.