1. Scalable Edge Bridge FDB For Datacenter Networks July-2012

2. Agenda  Problem statement and related work  Protocol properties, concepts and operation  Proposal for data and control planes  Summary & discussion 2 Overlay Network End- Station Edge- Bridge

3. Problem Statement and Related Work

4.  Problem statement  Large # of VMs in datacenters (>1M)  large address table in datacenter bridges  Support for hot VM migration  VM address must not change  address table scaling techniques based on address aggregation limit migration options –For example, IP stations can migrate within the same VLAN  Overlay networks solve address scaling problem in Core Bridges  Core Bridge address table ~= # Edge Bridges << # of VMs in the network  Lot’s of work on overlay protocols: SPB, PBB, VPLS, TRILL, VXLAN, NVGRE  How to scale the address table in Edge Bridges (EB)?  VXLAN/NVGRE – specific solutions for IP overlay  SPB/TRILL – none (July-2012)  Objective: provide a solution to address scaling in SPB Edge Bridges  The solution must complement (not replace) overlay network protocols  Preferably, one solution should fit many overlay network protocols, so it can be easily adapted to work with other overlay protocols 4

5. Bridge FDB Scaling (BFS) Protocol Concepts and Operation

6. Bridge FDB Scaling (BFS) Concepts  BFS defines a handshake between the EB and the End-Station (An End-Station may host 1 or more VMs)  Capabilities exchange use control-plane  Dynamic operation uses the data-plane  EB operation in a nutshell  Learns addresses of local VMs & remote EBs (but not remote VMs)  Uses data-plane signaling to informs the End-Station of the path in the overlay network  Uses the path signaled by the End-Station to forward traffic to remote VMs over the overlay network  End-Station operation in a nutshell  Sends data traffic to EB with path indication  Updates its path database (Path$) using the indications received from the EB 6

7. 7 BFS Databases and Signaling VM1 VM2 B VMPort D S D S B PEB 1A 2B 3C A.1$ VMPath D S S.Path Generated by VM D S T.Path D S Server  EB Overlay Network EB  Server Rx by VM Edge Bridge End-Station Path$ Overlay FDB Local FDB

8. EB Operation  Overlay FDB learning  Control plane triggered as specified by the overlay protocol (e.g. IS-IS for SPB)  Address learning process (Local FDB)  Data-plane learning –Don’t learn on overlay ports –Learn on local ports  Forwarding packets received on local ports  If packet has no T.Path indication Lookup in local FDB using DA if found  forward accordingly, don’t assign S.Path to traffic to local ports else flood to local and overlay ports else // packet has T.Path indication Obtain the overlay path attributes using T.Path Remove T.Path, add ovelay tunnel Send to overlay  Forward packets received on overlay ports  Lookup overlay FDB with the overlay header, obtain S.Path Remove overlay header, assign S.Path Lookup local FDB with DA if found, forward accordingly else flood to local ports 8

9. End-Station Operation  Forwarding packets received from VM  Lookup Path$ with DA If found, assign T.Path to the packet and forward to EB else forward to EB w/o T.Path  Forward packets received from EB  Use DA or 802.1Qbg/802.1BR indication to forward to the VM  Path$ update policy (packets received from EB)  If packet has no S.Path, don’t update Path$ else // packet has S.Path update Path$ if any of the following is met DA indicates a VM hosted by this End-Station, OR DA=BC and L3-DA indicates a VM hosted by this End-Station 9

10. A PEB 1A 2B 3C 10 BFS Operation Example #1  VM1  VM2 flooded Unicast forwarding VM1 VM2 A VMPort C VMPort B VMPort 21 D S 1 A.1 21 D S BCA 21 D S A Dataplane learning  EB table size = # of local VMs + # of EBs in the network C PEB 1A 2B 3C B P 1A 2B 3C A.1$ VMPath B.1$ VMPath 21 D S 1 s.Path 21 D S 1 21 D S 1 21 D S 1 1 1 Learn only in B.1 SPB Overlay

11. A PEB 1A 2B 3C 11 BFS Operation Example #2  VM2  VM1 reply VM1 VM2 A VMPort C VMPort B VMPort 21 S D 1 A.1 BA D S 21 Dataplane learning  EB table size = # of local VMs + # of EBs in the network C PEB 1A 2B 3C B P 1A 2B 3C A.1$ VMPath B.1$ VMPath 11 D T.Path 2 S 1 1 12 D S.Path 2 S 21 S D 2 2 2 B.1 SPB Overlay

12. BFS Data and Control Planes (A Proposal)

13. 13 BFS Data and Control Planes - A Proposal  Control protocol  Capabilities negotiation between the End-Station and the Edge Bridge  Modify 802.1Qaz (DCBx)  Data-plane protocol (2 options)  Add Path-ID Tag (P-Tag) –S-channel/E-Tag is outer –P-Tag is inner: –16b source/target-path-id –Source/target depends on direction  Modify BPE E-Tag –End-Station  EB –Ingress-ECID – identical use to BPE –E-CID – target-path-id –EB  End-Station –Ingress-ECID –Ingress-ECID < 4K local virtual port (identical to BPE) –Ingress-ECID =>4K source-path-id –E-CID – identical use to BPE DA (6B) SA (6B) S-Channel /E-Tag (8/4B) P-Tag (4B)VLAN (4B) Payload + FCC

14. Summary

15. Summary of BFS Properties  Complements SPB towards scaling the EB FDB  A generic solution that can be considered for additional overlay protocols  Small Path$ in End-Station  Holds active sessions only – comparable in size to the ARP$  Easy to implement  Local scope: end-station to edge-bridge protocol  Simple control-plane – only need to negotiate capabilities, no dynamic operation –Extend DCBX 802.1Qaz  Simple extension of existing data-plane protocols –Extends 802.1BR/802.1Qbg with a P-Tag or modifies 802.1BR E-Tag  Easy to deploy  Co-exists with 802.1Qbg/802.1BR protocols  Support for incremental upgrade per EB granularity 15

16. Thank you Contact: Carmi Arad, carmi@marvell.com