System Testing SDN - A Case Study on ONOS Suibin Zhang, ON.Lab Jon Hall, Ciena

Known UNKNOWNs Known UNKNOWNs Known KNOWNs Unknown UNKNOWNs Design Considerations Unit Test Considerations System Test Considerations Quality Assurance on Software System Subjective Perception Objective Reality ✔ 2 #ONOSProject

3 Black/Grey Box Testing – Often mimic usage scenarios – Validity not influenced by internal design Product Verification/Qualification Pre-Deployment Verification System Test - Where Used

4 #ONOSProject Testing SDN OS – Nontrivial SDN Network Domain Distributed Controllers SDN CTRL 1 SDN CTRL 1 SDN CTRL 2 SDN CTRL 2 SDN CTRL n SDN CTRL n APP Test Exec. Legacy Network Domain Dependencies

5 #ONOSProject Technical – Accessing heterogeneous, evolving interfaces – Scaling test environment – Facilitate Debugging Arise from Community/Open Source – Run “anywhere” – Handling test dependencies – Contribution and merging tests Challenges in Building System Tests

6 #ONOSProject A Case Study on ONOS

7 #ONOSProject Functionality Suite – NB: Intents and Flow operations – SB: OF 1.0 &1.3, NETCONF, OVSDB, etc. High Availability Suite – Various scenarios of cluster failures Performance and Scale-out Suites – Intent, flow throughputs, latency as scaling-out – Topology discovery latency as scaling-out Longevity – Continuous operation under typical deployment scenarios Selected Use Case Suites – SDNIP App, Segment Routing App ONOS System Test Suites

8 #ONOSProject Establish Test Handles Run-time Configuration Test Logic Reset/Clean Test Env. Inject Run Params to TestStation Start Test Cases Jenkins Plugins Update Codes Publish Results Onos System Test CI

9 #ONOSProject A Deeper Dive into TestON Onos System Test CI

10 #ONOSProject TestON – Key Benefits TestON o SDN-focused Testing Framework o Test case authoring, execution o Python-based Key Benefits: o Extensibility – Simplify case authoring by compartmentalizing concerns o Flexibility - Handle heterogeneous test env. OVS, HW, Quagga, etc. o Debuggability - Facilitate troubleshooting

11 #ONOSProject *.params *.py *.topo Dependency Details on TestON Framework TestON Core Framework: Logging, Utilities, … Mininet Driver ONOS Rest Driver Quagga Driver ONOS CLI Driver Test Files.params File – XML file, used to pass variables into test, such as order of test cases.topo file – XML File, defines the components of the test Dependency Folder – Config files or Mininet topology files used by the test.py file – Combine TestON and driver functions to define your test cases. Test Cases are split into steps, each of which should have an assertion in them Driver Files Can use REST, CLI via SSH, or other custom connection interfaces Defined in a hierarchical structure Creates a library of functions to be used in the test

12 #ONOSProject ONOS Intent Feature Set intent through NB REST interface Validate intent and flows are correctly added Test end-to-end connectivity in Mininet with the specific selectors System Test Procedure Need a library to send packets with test-specified selector Leveraging Python Scapy module System Test Requirement High-level abstraction of setting end to end connectivity Can be defined by “Selectors” of MAC, IP, PORT, MPLS label, etc. Example: ONOS Intent Functionality Test

13 #ONOSProject Building a Scapy Driver (send/recv user-defined packets) class ScapyCliDriver( Emulator ): def sendPacket( self, iface=None, packet=None, timeout=1 ): try: sendCmd = 'srp( ' if packet: sendCmd += packet else:... if iface: sendCmd += ", iface='{}'".format( iface ) sendCmd += ', timeout=' + str( timeout ) + ')' self.handle.sendline( sendCmd ) self.handle.expect( self.scapyPrompt ) except:... TestON |--drivers | |--common | |--api | |--cli | | |--emulator | | |--ScapyCliDriver.py |--tests Example: ONOS’ Intent Functionality Test

14 #ONOSProject for hostName in [ “h1”,”h2” ]: # Create two host components with the scapy driver4 main.Scapy.createHostComponent( hostName ) main.step( ”Login to scapy on these new components" ) for host in [ main.h1, main.h2 ]: host.startHostCli() host.startScapy() Using Scapy Library - in Mininet Host Handle admin ScapyCliDriver 6 TestON |--drivers |--tests |--Example |-- Example.py |--Example.params |-- Example.topo Example: ONOS’ Intent Functionality Test

15 #ONOSProject # Add connectivity from h1 to h2 (unidirectional) addResult = main.ONOS1.addPointIntent( ingress, egress, ethDst, ipDst) main.h2.startFilter() # Start packet filter on host2 main.h1.buildEther( dst=main.h2.hostMac ) # build ethernet frame main.h1.buildIP( dst=main.h2.hostIp ) # Build IP frame main.h1.sendPacket() # Send built packet finished = main.h2.checkFilter() # Check if host2 received any packets Intent Test Case - Using Scapy Driver TestON |--drivers |--tests |--Example |-- Example.py |--Example.params |--Example.topo Example: ONOS’ Intent Functionality Test

16 #ONOSProject Parameterize Test Env & Run Var – Jenkins to inject at run time – Test cases deal with “pure” test logic Extensive Exception Handling in Drivers – Take care of exceptions, timeout – Use “retry” method Test Dependencies – Run-time Activation – “Dockerize” Fixture (Future Work) Debug Utilities – Test pausing – Log collation – More (Future Work) Scale Test Network (Future Work) Lessons Learned and Future Work

17 #ONOSProject For more Information S3 Showcase: “SDN/ONOS System Test” Wiki: Repo: Github: Summary Python-Based Extensibility Debuggability Flexibility ONOS Test Suites: Functionality High Availability Performance Scale-out Capability Longevity Use Cases ONOS Test Suites: Functionality High Availability Performance Scale-out Capability Longevity Use Cases ONOS System Test Highlights Open Source

18 #ONOSProject Join the onosproject.org Software Defined Transformation of Service Provider Networks