1. Trellis Tutorial An Open-Source, White-Box, SDN Based Leaf-Spine Network Fabric Charles Chan Open Networking Foundation

2. Trellis L2/L3 leaf-spine DC network fabric
Open Source
Software
White Box
Hardware
SDN Based

3. Trellis Features Bridging Routing VLAN L3 DHCP Relay vRouter IPv6
Dual Homing
and more…

4. Prerequisite
ONOS
Mininet
DHCP Server
Quagga
&
BGP

5. Demo Topology of:226 of:227 Q of:204 of:205 R rh1 h1 h2 D h3 h4
of:204
of:205 6 R
rh1 5 4 3 3 4
VLAN None
VLAN 10
VLAN 30
VLAN None h1 h2 D h3 h4
Q: Quagga, R: Upstream Router, D: DHCP Server

6. Device Data Path ID (DPID) of the switch { "devices" : {
“segmentrouting” : { "name" : "s204",
"ipv4NodeSid" : 204,
"ipv4Loopback" : " ",
"routerMac" : "00:00:00:00:02:04",
"isEdgeRouter" : true,
"adjacencySids" : [] },
"basic" : {
"name": "s204",
"driver" : "ofdpa-ovs” }
Data Path ID (DPID) of the switch

7. Device Unused. This name doesn’t really matter. Will be deprecated. {
"devices" : {
"of: " : {
“segmentrouting” : { "name" : "s204",
"ipv4NodeSid" : 204,
"ipv4Loopback" : " ",
"routerMac" : "00:00:00:00:02:04",
"isEdgeRouter" : true,
"adjacencySids" : [] },
"basic" : {
"name": "s204",
"driver" : "ofdpa-ovs” }
Unused. This name doesn’t really matter.
Will be deprecated.

8. Device {
"devices" : {
"of: " : {
“segmentrouting” : { "name" : "s204",
"ipv4NodeSid" : 204,
"ipv4Loopback" : " ",
"routerMac" : "00:00:00:00:02:04",
"isEdgeRouter" : true,
"adjacencySids" : [] },
"basic" : {
"name": "s204",
"driver" : "ofdpa-ovs” }
Segment ID.
Can be an arbitrary value but need to be unique in the system. Also used as the MPLS label when doing segment routing.
Do not use reserved MPLS labels (i.e. <=16).

9. Device Loopback address of the switch.{
"devices" : {
"of: " : {
“segmentrouting” : { "name" : "s204",
"ipv4NodeSid" : 204,
"ipv4Loopback" : " ",
"routerMac" : "00:00:00:00:02:04",
"isEdgeRouter" : true,
"adjacencySids" : [] },
"basic" : {
"name": "s204",
"driver" : "ofdpa-ovs” }
Loopback address of the switch.
Can be an arbitrary value (e.g. the management IP) but need to be unique in the system.

10. Device Use for ARP reply when host ARPs its gateway.{
"devices" : {
"of: " : {
“segmentrouting” : { "name" : "s204",
"ipv4NodeSid" : 204,
"ipv4Loopback" : " ",
"routerMac" : "00:00:00:00:02:04",
"isEdgeRouter" : true,
"adjacencySids" : [] },
"basic" : {
"name": "s204",
"driver" : "ofdpa-ovs” }
Use for ARP reply when host ARPs its gateway.
Can be an arbitrary value (e.g. the management MAC) but need to be unique in the system.

11. Device True for leaves. False for spines. { "devices" : {
"of: " : {
“segmentrouting” : { "name" : "s204",
"ipv4NodeSid" : 204,
"ipv4Loopback" : " ",
"routerMac" : "00:00:00:00:02:04",
"isEdgeRouter" : true,
"adjacencySids" : [] },
"basic" : {
"name": "s204",
"driver" : "ofdpa-ovs” }
True for leaves. False for spines.

12. Device Unused. Will be deprecated. { "devices" : {
"of: " : {
“segmentrouting” : { "name" : "s204",
"ipv4NodeSid" : 204,
"ipv4Loopback" : " ",
"routerMac" : "00:00:00:00:02:04",
"isEdgeRouter" : true,
"adjacencySids" : [] },
"basic" : {
"name": "s204",
"driver" : "ofdpa-ovs” }
Unused. Will be deprecated.

13. Device This is the name that does matter. Will be displayed in GUI {
"devices" : {
"of: " : {
“segmentrouting” : { "name" : "s204",
"ipv4NodeSid" : 204,
"ipv4Loopback" : " ",
"routerMac" : "00:00:00:00:02:04",
"isEdgeRouter" : true,
"adjacencySids" : [] },
"basic" : {
"name": "s204",
"driver" : "ofdpa-ovs” }
This is the name that does matter.
Will be displayed in GUI

14. Device {
"devices" : {
"of: " : {
“segmentrouting” : { "name" : "s204",
"ipv4NodeSid" : 204,
"ipv4Loopback" : " ",
"routerMac" : "00:00:00:00:02:04",
"isEdgeRouter" : true,
"adjacencySids" : [] },
"basic" : {
"name": "s204",
"driver" : "ofdpa-ovs” }
Driver. ofdpa-ovs for OpenvSwitch. ofdpa3 for Broadcom XGS (Trident2, Tomahawk)
qmx-ofdpa3 for Broadcom QMX

15. Port - VLAN Untagged DPID/port number Equivalent to access port.
of:204 h1 h2
VLAN None
VLAN 10 3 4 {
"ports" : {
"of: /3" : {
"interfaces" : [
"ips" : [" /24"],
"vlan-untagged": } ] }
DPID/port number
Equivalent to access port.
Expecting untagged packet and will push VLAN 20

16. Port - VLAN Tagged Equivalent to trunk port.
of:204 h1 h2
VLAN None
VLAN 10 3 4 {
"ports" : {
"of: /4" : {
"interfaces" : [
"ips" : [" /24"],
"vlan-tagged": [20] } ] }
Equivalent to trunk port.
Expecting VLAN 20 tagged packet and will keep the tag.
It is possible to have more than one trunk VLAN.

17. Port - VLAN Tagged with NativeQ R D
of:205 h3
VLAN 30 3 4 5 6
VLAN None h4 {
"ports" : {
"of: /3" : {
"interfaces" : [
"ips" : [" /24”, " /24"]
"vlan-tagged": [30], "vlan-native": } ] }
Equivalent to trunk port with native VLAN.
Expecting VLAN 30 tagged packet and will keep the tag. Also expecting untagged packet and will push VLAN 40

18. Port - Subnet Two information can be derived from the subnet config:
The interface IP on the switch is
The subnet on this interface is /24
Currently we don’t map between subnet and VLAN.
VLAN information is solely used for defining bridging domain. Subnet information is solely used for routing.
There are some confusions. Will have 1-to-1 mapping in the future. {
"ports" : {
"of: /3" : {
"interfaces" : [
"ips" : [" /24”, " /24"]
"vlan-tagged": [30], "vlan-native": } ] }

19. L3 DHCP Relay Q R D

20. L3 DHCP Relay The location where the DHCP server is attached to {
"apps" : {
"org.onosproject.dhcprelay" : {
"default": [
"dhcpServerConnectPoint": "of: /6",
"serverIps": [" "] } ]
The location where the DHCP server is attached to

21. L3 DHCP Relay The IP address of DHCP server { "apps" : {
"org.onosproject.dhcprelay" : {
"default": [
"dhcpServerConnectPoint": "of: /6",
"serverIps": [" "] } ]
The IP address of DHCP server

22. dhcpd.conf Subnets we want to assign
subnet netmask {
range ;
option routers ; }
subnet netmask {
range ;
option routers ;
Subnets we want to assign

23. dhcpd.conf IP range that will be used for this subnet
subnet netmask {
range ;
option routers ; }
subnet netmask {
range ;
option routers ;
IP range that will be used for this subnet

24. dhcpd.conf
subnet netmask {
range ;
option routers ; }
subnet netmask {
range ;
option routers ;
Default gateway we want to assign, which is the interface IP of the leaf switch

25. vRouter {
"ports" : {
"of: /8" : {
"interfaces" : [
"ips" : [" /24”]
"vlan-untagged": } ] },
"of: /9" : { }
Quagga and upstream router need to be in the same VLAN and subnet.
We need 3 IPs in the subnet. 1 for Quagga, 1 for upstream router, 1 for leaf switch interface.
/29 is the minimum subnet.

26. Route Map
Quagga announces leaf switch, instead of quagga itself, as the next hop to the internal network
/24 Q
/24
Routed
data traffic
Bridged
BGP traffic
/24 R
/24

27. Quagga - bgpd.conf List of prefixes we want to announce
ip prefix-list 1 seq 10 permit /24
ip prefix-list 1 seq 20 permit /24
route-map NEXTHOP41 permit 10
match ip address prefix-list 1
set ip next-hop
neighbor
remote-as 65001
neighbor ebgp-multihop
neighbor timers connect 5
neighbor advertisement-interval 5
neighbor route-map NEXTHOP41 out
network /24
network /24
List of prefixes we want to announce

28. Quagga - bgpd.conf Define the route map and its name
ip prefix-list 1 seq 10 permit /24
ip prefix-list 1 seq 20 permit /24
route-map NEXTHOP41 permit 10
match ip address prefix-list 1
set ip next-hop
neighbor
remote-as 65001
neighbor ebgp-multihop
neighbor timers connect 5
neighbor advertisement-interval 5
neighbor route-map NEXTHOP41 out
network /24
network /24
Define the route map and its name

29. Quagga - bgpd.conf
ip prefix-list 1 seq 10 permit /24
ip prefix-list 1 seq 20 permit /24
route-map NEXTHOP41 permit 10
match ip address prefix-list 1
set ip next-hop
neighbor
remote-as 65001
neighbor ebgp-multihop
neighbor timers connect 5
neighbor advertisement-interval 5
neighbor route-map NEXTHOP41 out
network /24
network /24
Apply the route map to the prefixes in prefix-list 1

30. Quagga - bgpd.conf
ip prefix-list 1 seq 10 permit /24
ip prefix-list 1 seq 20 permit /24
route-map NEXTHOP41 permit 10
match ip address prefix-list 1
set ip next-hop
neighbor
remote-as 65001
neighbor ebgp-multihop
neighbor timers connect 5
neighbor advertisement-interval 5
neighbor route-map NEXTHOP41 out
network /24
network /24
Set the leaf switch, instead of Quagga itself, as the next hop for these prefixes

31. Quagga - bgpd.conf BGP peer information
ip prefix-list 1 seq 10 permit /24
ip prefix-list 1 seq 20 permit /24
route-map NEXTHOP41 permit 10
match ip address prefix-list 1
set ip next-hop
neighbor
remote-as 65001
neighbor ebgp-multihop
neighbor timers connect 5
neighbor advertisement-interval 5
neighbor route-map NEXTHOP41 out
network /24
network /24
BGP peer information

32. Quagga - bgpd.conf Announce the route map to the neighbor
ip prefix-list 1 seq 10 permit /24
ip prefix-list 1 seq 20 permit /24
route-map NEXTHOP41 permit 10
match ip address prefix-list 1
set ip next-hop
neighbor
remote-as 65001
neighbor ebgp-multihop
neighbor timers connect 5
neighbor advertisement-interval 5
neighbor route-map NEXTHOP41 out
network /24
network /24
Announce the route map to the neighbor

33. Quagga - bgpd.conf Networks that are reachable from Quagga
ip prefix-list 1 seq 10 permit /24
ip prefix-list 1 seq 20 permit /24
route-map NEXTHOP41 permit 10
match ip address prefix-list 1
set ip next-hop
neighbor
remote-as 65001
neighbor ebgp-multihop
neighbor timers connect 5
neighbor advertisement-interval 5
neighbor route-map NEXTHOP41 out
network /24
network /24
Networks that are reachable from Quagga

34. Quagga - zebrad.conf
(Optional) Point default route to management network
ip route /
fpm connection ip port 2620
FPM connection to ONOS IP.
Need to be reachable from Quagga

35. Demo Scripts and configurations are available at https://github

36. Thank you ! Visit http://wiki.opencord.org for more information