Using the WUGS-20 GigE Line Card John DeHart Applied Research Laboratory Computer Science and Engineering Department http://www.arl.wustl.edu/arl/projects/techX

Introduction Really understanding how the WUGS-20 GigE line cards work requires understanding how ethernet, ARP, Routers and ethernet switches all work. This set of instructions can’t possibly teach all of that. If you require all the details you are going to have to do a lot of work/study on your own. These instructions will hopefully get you to the point where you can get the GigE line cards working for some useful experimentation. Questions are always welcome: Please keep in mind that we are continuing to support Kits users without support. The Kits program is officially over. The number of Kits users who have GigE cards is very small (2). If you have questions about the GigE cards, you should probably send them to John DeHart directly (jdd@arl.wustl.edu) and CC the other users: Ron Srodawa (srodawa@oakland.edu) Min Song (msong@odu.edu) Our standard configuration has migrated to use FPXs and SPCs on each port of an NSP (formerly called MSR) Linux for all end hosts used in our router testbeds NetBSD used ONLY for building kernels and plugins We will still try to help you with other configurations but it is becoming increasingly difficult to replicate other configurations

Configuring a GigE Line Card SUNI Config There is a SUNI chip on the GigE card which needs some configuration. We won’t go into the details, just don’t change the SUNI configuration commands in the configuration script. Don’t even change the order. Ethernet Config MAC Address Mask IP Address Next Hop IP Address

Configuring a GigE Line Card Utilities readWriteRegs: low level utility for configuring the GigE card(s) Src Directory: wu_arl/wugs/gige/src/utilities/ReadWriteRegisters Binary Linux/readWriteRegs configGigE.sh: script for configuring the GigE cards uses readWriteRegs wu_arl/wugs/gige/src/utilities/configScripts requires that WUARL environment variable be set to point to the root of your wu_arl tree. for me, I set: > setenv WUARL /d/jdd/wu_arl iperf: IP traffic generator useful for experimentation version 1.6.1 exists in CVS tree wu_arl/utilities/IP/iperf website: http://dast.nlanr.net/Projects/Iperf/ tcpdump: Useful for debugging run as root run it on the gigabit ethernet interface of your host, something like: tcpdump –i eth1 you’ll see things like the ARP messages and ping requests and replies. Just be careful of running it when you are sending LOTS of data

configGigE.sh utility Usage Message Running configGigE.sh with no arguments will give the Usage message: > ./configGigE.sh ./configGigE.sh -vci <VCI> -mac <MAC_H16> <MAC_L32> [-{0,1,2} <MYIP#> <MASK#> <NHIP#>] -vci VCI : VCI on which to send GigE control packets from the CP -mac MAC_H16 MAC_L32: hex MAC address to assign to GigE card, high 16 bits first then low 32 bits -0 MYIP0 MASK0 NHIP0: hex values for MYIP0 MASK0 NHIP0 -1 MYIP1 MASK1 NHIP1: hex values for MYIP1 MASK1 NHIP1 -2 MYIP2 MASK2 NHIP2: hex values for MYIP2 MASK2 NHIP2 Example> ./configGigE.sh -vci 100 -mac 0x4400 0x5E040001 -0 0xC0A82801 0xFFFFFF00 0xC0A82805 -1 0 0 0 -2 0 0 0 this would assign: MAC address: 44 00 5e 04 00 01 MYIP0 : 192.168.50.1 MASK0 : 255.255.255.0 (24 bits) NHIP0 : 192.168.50.5 MYIP1 : 0 MASK1 : 0 NHIP1 : 0 MYIP2 : 0 MASK2 : 0 NHIP2 : 0

Control Connection for GigE Configuration CP VCI=N VCI=N VCI=30 P0:GLink P1:GigE APIC VCI=N+1 VCI=N+1 Commands to read/write registers in GigE card VCI=30 P2:GigE Each GigE card needs a separate bidirectional control connection between it and the CP. At the GigE end it must use VCI=30 In the example above we have two GigE cards and we are using VCI N and VCI N+1 at the CP end. There are two connections on P0 mapping VCI N  P1, VCI 30 VCI N+1  P2 VCI 30 And one connection on each of P1 and P2: P1 VCI 30  P0 VCI N P2 VCI 30  P0 VCI N+1 Remember: You still need at least one ATM interface on your switch so you can connect a CP that can send ATM control cells!

A Simple Configuration Host1 VCI=128 GigE-1 IP: 192.168.40.2 24 bit netmask MAC: <as defined by hw> route: 192.168.50.0/24 gw: 192.168.40.1 GE Eth MYIP0: 192.168.40.1 MYMAC: 00:00:5E:04:00:01 MASK0: * (Don’t care) NHIP0: * (Don’t care) Host2 VCI=128 GigE-2 IP: 192.168.50.2 24 bit netmask MAC: <as defined by hw> route: 192.168.40.0/24 gw: 192.168.50.1 GE Eth MYIP0: 192.168.50.1 MYMAC: 00:00:5E:04:00:02 MASK0: * (Don’t care) NHIP0: * (Don’t care)

Simple Configuration: Ping When Host1 tries to ‘ping’ Host2 for the first time: Host1 has a route that tells it to use 192.168.40.1 as the next hop to get to Host 2 (192.168.50.2) Host1 has no ARP entry for 192.168.40.1 Host1 generates an ARP request asking for MAC address of 192.168.40.1 and sends it out its GE Ethernet interface GigE-1 receives the ARP request which matches its MYIP0 address and responds with its MAC address Host1 receives the ARP reply Host1 then sends out the ping packet. GigE-1 receives the ping packet and formulates an AAL5 frame and sends the cells of the frame out on VPI/VCI 0/128. WUGS-20 is configured to send VPI/VCI 0/128 to port of GigE-2 GigE-2 receives first ping packet destined for 192.168.50.2 GigE-2 has no ARP entry for 192.168.50.2 GigE-2 generates an ARP request asking for the MAC address of 192.168.50.2 and sends it out its ethernet interface. Host2 receives the ARP request which matches its IP address and responds with its MAC address GigE-2 receives the ARP reply and puts an entry in its ARP table.

Simple Configuration: Ping When Host1 tries to ‘ping’ Host2 for the second time: Host1 has a route that tells it to use 192.168.40.1 as the next hop to get to Host 2 (192.168.50.2) Host1 has an ARP entry for 192.168.40.1 Host1 sends out an ethernet frame for the ping packet with a next hop MAC address of GigE-1. GigE-1 receives the ethernet frame and formulates an AAL5 frame and sends the cells of the frame out on VPI/VCI 0/128 WUGS-20 is configured to send VPI/VCI 0/128 to the port of GigE-2 GigE-2 receives first ping packet destined for 192.168.50.2 GigE-2 has an ARP entry for 192.168.50.2 GigE-2 sends the ping packet out its ethernet interface ETC… (similar stuff happens for the ping reply except some ARP information was gleaned on the forward path…)

A Configuration with a Switch IP: 192.163.204.10 MAC: 08:00:20:7C:E3:25 Host1 IP: 192.163.204.2 MAC: 08:00:20:7C:E3:25 IP: 192.163.204.3 MAC: 08:00:20:7C:F2:45 Host2 Host3 P3 Ethernet Switch MSR P1 Port 1: MAC: 00:00:5E:04:00:01 MYIP0: 192.163.204.1 MASK0: * (Don’t care) NHIP0: * MYIP1: * MASK1: NHIP1: * MYIP2: * MASK2: * NHIP2: * Host4 IP: 192.163.204.4 MAC: 00:01:03:7C:23:03

A Configuration with a Switch: Ping When Host1 tries to ‘ping’ Host2: The MSR Port 3 is configured to route packets destined for Host2 to Port 1 on the base VPI/VCI of 0/128. Port 1 has a GigE interface. Port 1 initially does not have an ARP entry for Host2. Port 1 sends an ARP request looking for Host2 and drops this first ping packet. The Ethernet switch sends this packet to Host2 which sends an ARP reply. The Ethernet switch sends the ARP reply back to the MSR Port 3. The next ping from Host1 to Host2 will go through. Similar for when Host1 pings Host3 or Host4. This configuration still does not need to use the Next Hop information in the GigE card.

A Configuration with Routers Host 1 192.168.100.10 192.168.202.0/24 Hosts 192.168.210.0/24 Hosts 192.168.201.0/24 Hosts P3 Ethernet Switch2 P0 Router2 P1 Ethernet Switch1 MSR P0 192.168.200.0/24 Hosts Ethernet Switch3 P0 Router3 P1 192.168.220.0/24 Hosts

A Configuration with Routers In this configuration we have Ethernet Switch1 supporting three subnets: 192.168.200.0/24 192.168.201.0/24 192.168.202.0/24 Each of the Routers (Router1, Router2, MSR) has a presence in each of the subnets (200, 201, 202) on their P0 interfaces MSR P0 (limited to three): 192.168.200.1 192.168.201.1 192.168.202.1 Router2 P0: 192.168.200.2 192.168.201.2 192.168.202.2 Router3 P0: 192.168.200.3 192.168.201.3 192.168.202.3 MSR also supports hosts on the 192.168.100.0/24 subnet such as Host1 192.168.100.10 Router2 also supports the 192.168.210.0/24 subnet Router3 also supports the 192.168.220.0/24 subnet

A Configuration with Routers (con’t) MSR P0 GigE Card Configuration MAC: 44 00 5E 04 00 01 MYIP0: 192.168.200.1 MASK0: 0xffffff00 NHIP0: 192.168.200.2 used for Egress VCI 129 MYIP1: 192.168.201.1 MASK1: 0xffffff00 NHIP1:192.168.201.3 used for Egress VCI 130 MYIP2: 192.168.202.1 MASK2: 0 NHIP2:0 used for Egress VCI 131 (unused in this example)

Example Commands to Configure P0 GigE Assume that VCI 101 is set up in the MSR switch to go from the CP port to P0 VCI 30. cd $WUARL/wugs/gige/src/configScripts ./configGigE.sh -vci 101 -mac 0x4400 0x5E040001 -0 0xc0A8C801 0xffffff00 0xc0A8C802 -1 0xc0A8C901 0xffffff00 0xc0A8C903 -2 0xc0A8CA01 0 0

A Configuration with Routers: Ping When Host1 tries to ‘ping’ any host in the (200, 201, 202) subnets Just like our earlier example where there were no routers.

A Configuration with Routers: Ping When Host1 tries to ‘ping’ a host in the 210 subnet The MSR P3 is configured to route packets destined for the 210 subnet to P0 on the VPI/VCI of 0/129. This will cause P0 to use MYIP0, MASK0 and NHIP0. NHIP0 is the IP address of Router2 A function applied to (MYIP0, MASK0, NHIP0) is used to generate an index into the ARP table in the GE card to store the MAC address of Router2 details are not important at this point. P0 has a GigE interface. P0 GigE initially does not have an ARP entry for Router2. P0 GigE sends an ARP request looking for Router2 and drops this first ping packet. Router2 sends an ARP reply with the MAC address from its P0. The next ping from Host1 to the 210 subnet will go through.

A Configuration with Routers: Ping When Host1 tries to ‘ping’ a host in the 220 subnet The MSR P3 is configured to route packets destined for the 220 subnet to P0 on the VPI/VCI of 0/130. This will cause P0 to use MYIP1, MASK1 and NHIP1. NHIP1 is the IP address of Router3 A function applied to (MYIP1, MASK1, NHIP1) is used to generate an index into the ARP table in the GE card to store the MAC address of Router3 details are not important at this point. P0 has a GigE interface. P0 GigE initially does not have an ARP entry for Router3. P0 GigE sends an ARP request looking for Router3 and drops this first ping packet. Router3 sends an ARP reply with the MAC address from its P0. The next ping from Host1 to the 220 subnet will go through.

Performance In our tests we have seen the following: Using two 2.4 GHz Athlon machines Using iperf UDP: packet size 512 bytes sending rate: ~720 Mb/s receiving rate: ~720 Mb/s with about 0.2% drops at receiver receiving host couldn’t keep up TCP: 700 Mb/s - 770 Mb/s using the iperf defaults 8K buffer size MSS set by TCP based on MTU