1. www.opendaylight.org Test and Performance Integration Group

2. www.opendaylight.org Test Achievements 2 Sept 2013 June 2015 Beryllium Release First Release Performance report Helium Release First Performance test: OpenFlow (Cbench) Sept 2014 Integration group New project scope: distribution and CI system test Feb 2014 Hydrogen Release First system test: OpenFlow, OVSDB and VTN Manager Feb 2016 Lithium Release First Cluster test First SSP suite for OpenFlow

3. www.opendaylight.org  Build test: Junit, integration, SingleFeature  System test: feature, multi-feature  SSP tests: Stability, Scalability, Performance  Cluster tests: Data consistency, HA Continuous Integration Tests 3

4. www.opendaylight.org CI Test Strategy 4 Code Merge Auto-Release Merge build Distribution feature test cluster test For all projects Code SubmitVerify build Build test cluster test For all projects in downstream feature test For all projects in downstream Build test Distribution-check test Auto-Release test cluster test Daily trigger SSP tests Multi-feature test For all projects Multi-feature test

5. www.opendaylight.org  Build test: https://jenkins.opendaylight.org/releng/view/Merge-Jobs/  System test: https://jenkins.opendaylight.org/releng/view/CSIT-1node/  SSP tests: https://wiki.opendaylight.org/view/CrossProject:Integration_Group:Performanc e_Test:Results  Cluster tests: https://jenkins.opendaylight.org/releng/view/CSIT-3node/ CI Tests Results 5

6. www.opendaylight.org  Real network test scenarios.  Uniform and dedicated test environment.  More than one environment to compare results.  Tests documented and step-by-step reproducible.  Other SDN controllers results added for reference.  Recommendations for ODL best performance. Beryllium Performance Report 6

7. www.opendaylight.org  OpenFlow  End-to-End performance  Southbound performance  Netconf  Northbound performance  Southbound scale  Southbound performance  End-to-End performance  OVSDB  Southbound scale  BGP  Data Ingest performance  Data Advertise performance  PCEP  LSP Read performance  LSP Update performance Protocol Summary 7

8. www.opendaylight.org  Setup 1 Test Environment 8 Hardware: CPU: Intel(R) Xeon(R) CPU E5-2690 v2 @ 3.00GHz, 40 cores Total RAM: 128GB OS: Ubuntu Linux 4.2.0-30-generic x86_64 Network: Every test component is run directly on the machine all communication happens over the 127.0.0.1/8 loopback network. JVM: Oracle 1.8.0_73 8G of java heap Default garbage collector

9. www.opendaylight.org  Setup 2 Test Environment 9 Hardware: CPU: 2 Intel Xeon E5-2699 v3sz @ 2.30 GHz, 72 cores Total RAM: 64GB OS: Fedora release 23 Linux 4.2.3-300.fc23.x86_64 x86_64 Network: Test component is run directly on the machine all communication happens over 127.0.0.1/8 loopback network Exception: OVSDB Scale Tests which use multiple other identical systems to scale test tools JVM: OpenJDK 64-Bit 1.8.0_72 8G of java heap Default garbage collector

10. www.opendaylight.org  Controller Software  OpenDaylight Beryllium release.  OpenDaylight Beryllium with OF Boron plugin.  ONOS Falcon pre-release (RC3).  FloodLight 1.2.  NOTE: Data for other SDN controllers is only for general comparative reference. They should not be considered as definitive maximum performance values for those platforms. Test Environment 10

11. www.opendaylight.org  Test procedure:  Configure 100k flows and wait until we get confirmation of flows being programmed.  Remove flows and wait for confirmation.  Step-by-step guide: https://wiki.opendaylight.org/view/Openflow:Testing OpenFlow E2E Performance 11

12. www.opendaylight.org  Beryllium plugin:  Flow Add Controller Rate= 2k flows/s  100k flows Switch program Time= 53 s  100k flows Confirmation time= 23-143 s  Boron plugin:  Flow Add flow Controller Rate= 2k flows/s  100k flows Switch program Time= 53 s  100k flows Confirmation time= 7-10 s  ONOS:  Flow Add Controller Rate= 2.4k flows/s  100k flows Switch program Time= 43-61 s  100k flows Confirmation time= 32-404 s  Floodlight:  Flow Add Controller Rate= 2k flows/s  100k flows Switch program Time= 54 s OpenFlow E2E Performance 12

13. www.opendaylight.org  Beryllium plugin:  Single flow per REST call:  Flow Add Controller Rate= 2k flows/s  100k flows Switch program Time= 53 s  100k flows Confirmation time= 23-143 s  200 flows per REST call:  Flow Add Controller Rate= 10k flows/s  100k flows Switch program Time= 11 s  100k flows Confirmation time= 23-143 s  Boron plugin:  Single flow per REST call:  Flow Add Controller Rate= 2k flows/s  100k flows Switch program Time= 53 s  100k flows Confirmation time= 7-10 s  200 flows per REST call:  Flow Add Controller Rate= 10k flows/s  100k flows Switch program Time= 11 s  100k flows Confirmation time= 7-10 s OpenFlow E2E Performance 13

14. www.opendaylight.org  Test procedure:  Configure 100k flows in the OpenFlow network.  Restart network (mininet).  Wait until we get confirmation of flows being re-programmed.  Step-by-step guide: https://wiki.opendaylight.org/view/Openflow:Testing OpenFlow SB Performance 14

15. www.opendaylight.org  Beryllium plugin:  Flow Add Switch Rate= 10-20k flows/s  100k flows Confirmation time= 160-204 s  Boron plugin:  Flow Add Switch Rate= 10k flows/s  100k flows Confirmation time= 7-10 s  ONOS:  Flow Add Switch Rate= 1-14k flows/s (flows were missing)  100k flows Confirmation time= 60-64s  Floodlight:  Flow Add Switch Rate= 10-20k flows/s  100k flows Confirmation time= No API found for this OpenFlow SB Performance 15

16. www.opendaylight.org  Test procedure:  ODL starts with simple l2fib models and NETCONF northbound enabled.  External client writes lots of l2fib entries in ODL Datastore using NETCONF northbound.  The client measures time since 1st request is sent until last response is received.  Step-by-step guide: https://wiki.opendaylight.org/view/NETCONF:Testing Netconf NB Performance 16

17. www.opendaylight.org  Beryllium Release:  Single l2fib entry per Netconf request:  l2fib Add Rate with TCP client= 7k l2fib/s  l2fib Add Rate with SSH client= 6.6k l2fib/s  100 l2fib entries per Netconf request:  l2fib Add Rate with TCP client= 42k l2fib/s  l2fib Add Rate with SSH client= 38.5k l2fib/s Netconf NB Performance 17

18. www.opendaylight.org  Test procedure:  Start netconf-testtool with desired amount of netconf servers.  Testtool generates initial configuration for OpenDaylight.  Check OpenDaylight connects to all of the simulated devices (TCP or SSH).  Step-by-step guide: https://wiki.opendaylight.org/view/NETCONF:Testing Netconf SB Scale 18

19. www.opendaylight.org  Beryllium release:  Devices connected via TCP= 28,000  Device connected via SSH= 14,000 Netconf SB Scale 19

20. www.opendaylight.org  Test procedure:  OpenDaylight mounts a simulated NETCONF device.  Small ODL application triggers a NETCONF notification stream for new mount point.  Simulated device starts sending routes with a certain number of prefixes into ODL.  Application waits until all notifications have been processed.  Step-by-step guide: https://wiki.opendaylight.org/view/NETCONF:Testinghttps://wiki.opendaylight.org/view/NETCONF:Testing Netconf SB Performance 20

21. www.opendaylight.org  Beryllium Release:  Single prefix per Netconf notification:  Prefix Learn Rate with TCP= 10.7k prefixes/s  Prefix Learn Rate with SSH= 9.8k prefixes/s  10 prefixes per Netconf notification:  Prefix Learn Rate with TCP= 20k prefixes/s  Prefix Learn Rate with SSH= 16.4k prefixes/s Netconf SB Performance 21

22. www.opendaylight.org  Test procedure:  OpenDaylight mounts a simulated NETCONF device.  External REST client starts sending prefixes via RESTCONF.  ODL application transforms the requests into device specific models and writes to the device.  The REST client waits until all of its requests are handled in RESTCONF.  Step-by-step guide: https://wiki.opendaylight.org/view/NETCONF:Testinghttps://wiki.opendaylight.org/view/NETCONF:Testing Netconf E2E Performance 22

23. www.opendaylight.org  Beryllium Release:  Single prefix per Netconf notification:  Prefix Learn Rate with TCP= 3.2k prefixes/s  Prefix Learn Rate with SSH= 2.3k prefixes/s  50 prefixes per Netconf notification:  Prefix Learn Rate with TCP= 85.3k prefixes/s  Prefix Learn Rate with SSH= 61k prefixes/s Netconf E2E Performance 23

24. www.opendaylight.org  Test procedure:  Start multiple OVS systems and configure OpenDaylight as the OVS manager.  Run mininet in the OVS system to generate many OVS bridges.  Check all OVS systems connect to OpenDaylight.  Step-by-step guide: https://wiki.opendaylight.org/view/OVSDB:S3P OVSDB Scale 24 Controller OVS REST OVSDB

25. www.opendaylight.org  Beryllium release:  OVS bridges connected= 1,800 OVSDB SB Scale 25

26. www.opendaylight.org  Test procedure:  OpenDaylight connects to the BGP tool (play.py).  BGP tool sends 1 million updates (single prefix).  Wait until 1 million prefixes are present in ODL.  Disconnect BGP tool and wait until all prefixes are removed from ODL.  Step-by-step guide:  https://wiki.opendaylight.org/view/BGP_LS_PCEP:Specific_Performance_Testing BGP Data Ingest Perf 26 Controller BGP Tool REST BGP (1M prefixes)

27. www.opendaylight.org  Beryllium release:  Prefix Add rate= 16k prefixes/s  Prefix Delete rate= 23k prefixes/s BGP Data Ingest Perf 27

28. www.opendaylight.org  Test procedure:  OpenDaylight connects to the BGP tool (play.py).  Use REST and a Java application in controller to populate the ODL RIB.  Restart the BGP connection and wait until all prefixes are learned by the BGP tool.  Delete all prefixes in ODL RIB and wait until they clear from BGP tool.  Step-by-step guide:  https://wiki.opendaylight.org/view/BGP_LS_PCEP:Specific_Performance_Testing BGP Data Advertise Perf 28 Controller BGP Tool REST BGP (1M prefixes)

29. www.opendaylight.org  Beryllium Release:  REST to populate ODL RIB:  Single prefix/request rate= 200 prefixes/s  100k prefix/request rate= 2k prefixes/s  Java app to populate ODL RIB:  Single prefix/request rate= 6k prefixes/s  100k prefix/request rate= 6k prefixes/s  After reconnect BGP tool:  Prefix add rate= 20k prefixes/s  After remove all prefixes:  Prefix delete rate= 15k prefixes/s BGP Data Advertise Perf 29

30. www.opendaylight.org  Test procedure:  PCC tool (pcc-mock) connects to OpenDaylight (PCE).  PCC tool sends information for 65535 tunnels.  Wait until 65536 tunnels are present in ODL.  Step-by-step guide:  https://wiki.opendaylight.org/view/BGP_LS_PCEP:Specific_Performance_ Testing PCEP LSP Read Performance 30 Controller PCC Tool REST PCEP (65535 tunnels)

31. www.opendaylight.org  Beryllium release:  Read rate (1 PCC)= 4.6k tunnels/s  Read rate (512 PCC)= 4k tunnels/s PCEP LSP Read Performance 31

32. www.opendaylight.org  Test procedure:  PCC tool (pcc-mock) connects to OpenDaylight (PCE).  REST tool (updater.py) is used to send tunnel updates.  Wait until 65536 tunnels are updated in PCC tool.  Step-by-step guide:  https://wiki.opendaylight.org/view/BGP_LS_PCEP:Specific_Performance_ Testing PCEP LSP Update Performance 32 Controller PCC Tool REST PCEP (65535 tunnels) REST Tool

33. www.opendaylight.org  Beryllium release:  Update rate (1 PCC)= 1.8k tunnels/s  Update rate (512 PCC)= 1.9k tunnels/s PCEP LSP Update Performance 33

34. www.opendaylight.org  “Batching” multiple flows/prefixes into a single protocol message improves overall performance:  OpenFlow NB (REST) performance is 5x faster with “batch” requests.  Netconf SB performance is x20 faster than OpenFlow 1.3 as the last does not support “batching” requests.  By default OpenDaylight persists configuration data (other tested controllers do not persist), this has negative impact on performance and depending on the Use Case you may want to:  Use fast disk (e.g. SSD)  Disable persistence if you do not need. Performance Recommendations 34