A SOFT Way for OpenFlow Interoperability Testing Maciej Kuźniar, Peter Perešini, Marco Canini†, Daniele Venzano, Dejan Kostić‡ EPFL †TU Berlin/T-Labs ‡IMDEA Networks 1

Software-Defined Networking (SDN) Third-party control program 2 Controller

Release Interoperability at Deployment Time 3 OpenFlow program OpenFlow messages One OpenFlow API specification… Are OF switches interoperable? Interop is critical for the success of SDN

Switch III 1.Silently drops packets Switch I 1.Responds with an error message Switch II 1.Trims VLAN value to 12 bits 2.Installs the rule Inconsistency - Example 4 FlowMod message 1.Modify VLAN ID to Forward packet Network in 3 different states Which state is assumed by the controller? Where are packets forwarded?

Interop: How Hard Can It Be? 5 OF Switch Inputs Hardware correctness is formally verified Packets OpenFlow messages “Forwarding” interface OpenFlow interface ASIC switch chip OSOS OpenFlow Agent Likely source of OpenFlow interop issues Flow Table Hardware Abstraction Layer This work: Finding differences between OpenFlow Agent implementations

OpenFlow Software Agent 6 Specifications Rapid flux (3 revisions in ~ 1 year) Ambiguities Specifications  Implementation Implementation freedom Vendors may not follow the specs Testing, testing and testing… Switch software is not provably correct 

Interoperability Event 7 Gather various vendors Hook up switches and controllers Create and run test cases See what breaks and fix it Very high manual effort Test cases are not exhaustive It is not a one time thing [ONF Interop WG, March ’12]

Automating Interop Testing 8 Insight: systematically crosscheck OF implementations

The 10,000 foot view 9 OF Agent 1 Test inputs Input-driven execution Observable behaviors Inconsistency! OF Agent 2 Problem I: What inputs should we use?

Symbolic Execution 10 port < 25? yesno Symbolic message port = ∗ Forward port < 25 port == CTRL? yes no Send to CTRL port ≥ 25 ∧ port = CTRL Send ERROR port ≥ 25 ∧ port ≠ CTRL port CTRL ForwardSend to CTRLSend ERROR Problem II: Path explosion

Challenges Manage test inputs and coverage efficiently Capture behaviors Avoid simultaneous access to all code 11

SOFT (Systematic OpenFlow Testing) 12 ? OF Agent 1OF Agent 2 Automated solution to interop testing Systematic code coverage No simultaneous access to all agents

**** C3 C2 C1 Structuring Inputs 13 ******** FLOW MOD LEN1 STAT REQ LEN2 1.0 Further reductions Some messages are independent Many inputs are entirely concrete Small number of messages Concrete values at cost of coverage

Benefit of Concretizing 14 Fully Symbolic 28h

Capturing Behaviors Externally observable outputs OpenFlow reply messages Data plane packets Normalize harmless nondeterminism (e.g., Buffer IDs) Internal state changes affect successive inputs Use concrete probe packets 15

Example 16 Agent 1Agent 2

Finding Inconsistencies 17 Agent 1Agent 2

Is there an input that causes two distinct behaviors? Finding Inconsistencies 18 No false positives Agent 1: Agent 2: port CTRL

Limitations Short sequences of inputs Unable to find problems with a complex state Is an inconsistency harmless? Can it affect the controller? How to test all initial configurations? Agent’s behavior depends on initial config 19

Prototype & Evaluation SOFT prototype built on top of Cloud9/Klee Compared OpenFlow 1.0 Reference Switch (55k LoC) Open VSwitch 1.0.0(80k LoC) Input Sequences containing messages 20

Does SOFT Work? Found 7 classes of inconsistencies: 21 Packets dropped when action is invalid Different ports considered invalid Lack of error messages Different order of message validation Silently ignored statistics request Missing features Switch terminates with an error Mostly related to message validation Result of underspecification No expected behavior in the specification Inconsistent interpretation of the specification

Summary 22 SOFT automates interoperability testing of OpenFlow Agents Also useful for: regression testing specification improvements