1. Using the WUGS-20 GigE Line Card
John DeHart Applied Research Laboratory Computer Science and Engineering Department

2. Introduction
Really understanding how the WUGS-20 GigE line cards work requires understanding how ethernet, ARP, Routers and ethernet switches all work.
This set of instructions can’t possibly teach all of that. If you require all the details you are going to have to do a lot of work/study on your own.
These instructions will hopefully get you to the point where you can get the GigE line cards working for some useful experimentation.
Questions are always welcome:
Please keep in mind that we are continuing to support Kits users without support. The Kits program is officially over.
The number of Kits users who have GigE cards is very small (2).
If you have questions about the GigE cards, you should probably send them to John DeHart directly and CC the other users:
Ron Srodawa
Min Song
Our standard configuration has migrated to use
FPXs and SPCs on each port of an NSP (formerly called MSR)
Linux for all end hosts used in our router testbeds
NetBSD used ONLY for building kernels and plugins
We will still try to help you with other configurations but it is becoming increasingly difficult to replicate other configurations

3. Configuring a GigE Line Card
SUNI Config
There is a SUNI chip on the GigE card which needs some configuration.
We won’t go into the details, just don’t change the SUNI configuration commands in the configuration script. Don’t even change the order.
Ethernet Config
MAC Address
Mask
IP Address
Next Hop IP Address

4. Configuring a GigE Line Card
Utilities
readWriteRegs: low level utility for configuring the GigE card(s)
Src Directory:
wu_arl/wugs/gige/src/utilities/ReadWriteRegisters
Binary
Linux/readWriteRegs
configGigE.sh: script for configuring the GigE cards
uses readWriteRegs
wu_arl/wugs/gige/src/utilities/configScripts
requires that WUARL environment variable be set to point to the root of your wu_arl tree.
for me, I set: > setenv WUARL /d/jdd/wu_arl
iperf: IP traffic generator
useful for experimentation
version exists in CVS tree
wu_arl/utilities/IP/iperf
website:
tcpdump: Useful for debugging
run as root
run it on the gigabit ethernet interface of your host, something like:
tcpdump –i eth1
you’ll see things like the ARP messages and ping requests and replies.
Just be careful of running it when you are sending LOTS of data

5. configGigE.sh utility Usage Message
Running configGigE.sh with no arguments will give the Usage message:
> ./configGigE.sh
./configGigE.sh -vci <VCI> -mac <MAC_H16> <MAC_L32> [-{0,1,2} <MYIP#> <MASK#> <NHIP#>]
-vci VCI : VCI on which to send GigE control packets from the CP
-mac MAC_H16 MAC_L32: hex MAC address to assign to GigE card, high 16 bits first then low 32 bits
-0 MYIP0 MASK0 NHIP0: hex values for MYIP0 MASK0 NHIP0
-1 MYIP1 MASK1 NHIP1: hex values for MYIP1 MASK1 NHIP1
-2 MYIP2 MASK2 NHIP2: hex values for MYIP2 MASK2 NHIP2
Example> ./configGigE.sh -vci 100 -mac 0x4400 0x5E xC0A xFFFFFF00 0xC0A
this would assign:
MAC address: e
MYIP :
MASK : (24 bits)
NHIP :
MYIP : 0
MASK : 0
NHIP : 0
MYIP : 0
MASK : 0
NHIP : 0

6. Control Connection for GigE ConfigurationCP
VCI=N
VCI=N
VCI=30
P0:GLink
P1:GigE
APIC
VCI=N+1
VCI=N+1
Commands to
read/write
registers in
GigE card
VCI=30
P2:GigE
Each GigE card needs a separate bidirectional control connection between it and the CP.
At the GigE end it must use VCI=30
In the example above we have two GigE cards and we are using VCI N and VCI N+1 at the CP end.
There are two connections on P0 mapping
VCI N  P1, VCI 30
VCI N+1  P2 VCI 30
And one connection on each of P1 and P2:
P1 VCI 30  P0 VCI N
P2 VCI 30  P0 VCI N+1
Remember: You still need at least one ATM interface on your switch so you can connect a CP that can send ATM control cells!

7. A Simple Configuration
Host1
VCI=128
GigE-1
IP:
24 bit netmask
MAC: <as defined by hw>
route: /24 gw:
GE Eth
MYIP0:
MYMAC: 00:00:5E:04:00:01
MASK0: * (Don’t care)
NHIP0: * (Don’t care)
Host2
VCI=128
GigE-2
IP:
24 bit netmask
MAC: <as defined by hw>
route: /24 gw:
GE Eth
MYIP0:
MYMAC: 00:00:5E:04:00:02
MASK0: * (Don’t care)
NHIP0: * (Don’t care)

8. Simple Configuration: Ping
When Host1 tries to ‘ping’ Host2 for the first time:
Host1 has a route that tells it to use as the next hop to get to Host 2 ( )
Host1 has no ARP entry for
Host1 generates an ARP request asking for MAC address of and sends it out its GE Ethernet interface
GigE-1 receives the ARP request which matches its MYIP0 address and responds with its MAC address
Host1 receives the ARP reply
Host1 then sends out the ping packet.
GigE-1 receives the ping packet and formulates an AAL5 frame and sends the cells of the frame out on VPI/VCI 0/128.
WUGS-20 is configured to send VPI/VCI 0/128 to port of GigE-2
GigE-2 receives first ping packet destined for
GigE-2 has no ARP entry for
GigE-2 generates an ARP request asking for the MAC address of and sends it out its ethernet interface.
Host2 receives the ARP request which matches its IP address and responds with its MAC address
GigE-2 receives the ARP reply and puts an entry in its ARP table.

9. Simple Configuration: Ping
When Host1 tries to ‘ping’ Host2 for the second time:
Host1 has a route that tells it to use as the next hop to get to Host 2 ( )
Host1 has an ARP entry for
Host1 sends out an ethernet frame for the ping packet with a next hop MAC address of GigE-1.
GigE-1 receives the ethernet frame and formulates an AAL5 frame and sends the cells of the frame out on VPI/VCI 0/128
WUGS-20 is configured to send VPI/VCI 0/128 to the port of GigE-2
GigE-2 receives first ping packet destined for
GigE-2 has an ARP entry for
GigE-2 sends the ping packet out its ethernet interface
ETC… (similar stuff happens for the ping reply except some ARP information was gleaned on the forward path…)

10. A Configuration with a Switch
IP:
MAC: 08:00:20:7C:E3:25
Host1
IP:
MAC: 08:00:20:7C:E3:25
IP:
MAC: 08:00:20:7C:F2:45
Host2
Host3 P3
Ethernet
Switch
MSR P1
Port 1:
MAC: 00:00:5E:04:00:01
MYIP0:
MASK0: * (Don’t care)
NHIP0: *
MYIP1: *
MASK1:
NHIP1: *
MYIP2: *
MASK2: *
NHIP2: *
Host4
IP:
MAC: 00:01:03:7C:23:03

11. A Configuration with a Switch: Ping
When Host1 tries to ‘ping’ Host2:
The MSR Port 3 is configured to route packets destined for Host2 to Port 1 on the base VPI/VCI of 0/128.
Port 1 has a GigE interface.
Port 1 initially does not have an ARP entry for Host2.
Port 1 sends an ARP request looking for Host2 and drops this first ping packet.
The Ethernet switch sends this packet to Host2 which sends an ARP reply.
The Ethernet switch sends the ARP reply back to the MSR Port 3.
The next ping from Host1 to Host2 will go through.
Similar for when Host1 pings Host3 or Host4.
This configuration still does not need to use the Next Hop information in the GigE card.

12. A Configuration with Routers
Host 1
/24 Hosts
/24 Hosts
/24 Hosts P3
Ethernet
Switch2 P0
Router2 P1
Ethernet
Switch1
MSR P0
/24 Hosts
Ethernet
Switch3 P0
Router3 P1
/24 Hosts

13. A Configuration with Routers
In this configuration we have Ethernet Switch1 supporting three subnets:
/24
/24
/24
Each of the Routers (Router1, Router2, MSR) has a presence in each of the subnets (200, 201, 202) on their P0 interfaces
MSR P0 (limited to three):
Router2 P0:
Router3 P0:
MSR also supports hosts on the /24 subnet
such as Host
Router2 also supports the /24 subnet
Router3 also supports the /24 subnet

14. A Configuration with Routers (con’t)
MSR P0 GigE Card Configuration
MAC: E
MYIP0:
MASK0: 0xffffff00
NHIP0:
used for Egress VCI 129
MYIP1:
MASK1: 0xffffff00
NHIP1:
used for Egress VCI 130
MYIP2:
MASK2: 0
NHIP2:0
used for Egress VCI 131 (unused in this example)

15. Example Commands to Configure P0 GigE
Assume that VCI 101 is set up in the MSR switch to go from the CP port to P0 VCI 30.
cd $WUARL/wugs/gige/src/configScripts
./configGigE.sh -vci 101 -mac 0x4400 0x5E040001
-0 0xc0A8C801 0xffffff00 0xc0A8C802
-1 0xc0A8C901 0xffffff00 0xc0A8C903
-2 0xc0A8CA

16. A Configuration with Routers: Ping
When Host1 tries to ‘ping’ any host in the (200, 201, 202) subnets
Just like our earlier example where there were no routers.

17. A Configuration with Routers: Ping
When Host1 tries to ‘ping’ a host in the 210 subnet
The MSR P3 is configured to route packets destined for the 210 subnet to P0 on the VPI/VCI of 0/129. This will cause P0 to use MYIP0, MASK0 and NHIP0.
NHIP0 is the IP address of Router2
A function applied to (MYIP0, MASK0, NHIP0) is used to generate an index into the ARP table in the GE card to store the MAC address of Router2
details are not important at this point.
P0 has a GigE interface.
P0 GigE initially does not have an ARP entry for Router2.
P0 GigE sends an ARP request looking for Router2 and drops this first ping packet.
Router2 sends an ARP reply with the MAC address from its P0.
The next ping from Host1 to the 210 subnet will go through.

18. A Configuration with Routers: Ping
When Host1 tries to ‘ping’ a host in the 220 subnet
The MSR P3 is configured to route packets destined for the 220 subnet to P0 on the VPI/VCI of 0/130. This will cause P0 to use MYIP1, MASK1 and NHIP1.
NHIP1 is the IP address of Router3
A function applied to (MYIP1, MASK1, NHIP1) is used to generate an index into the ARP table in the GE card to store the MAC address of Router3
details are not important at this point.
P0 has a GigE interface.
P0 GigE initially does not have an ARP entry for Router3.
P0 GigE sends an ARP request looking for Router3 and drops this first ping packet.
Router3 sends an ARP reply with the MAC address from its P0.
The next ping from Host1 to the 220 subnet will go through.

19. Performance In our tests we have seen the following:
Using two 2.4 GHz Athlon machines
Using iperf
UDP:
packet size 512 bytes
sending rate: ~720 Mb/s
receiving rate: ~720 Mb/s with about 0.2% drops at receiver
receiving host couldn’t keep up
TCP: 700 Mb/s Mb/s
using the iperf defaults
8K buffer size
MSS set by TCP based on MTU