MB – NG SuperJANET4 Development Network Production Network Leeds RAL / UKERNA SJ4 Dev C-PoP Reading 12416 Warrington Manc MCC ULCC UCL London Gigabit Ethernet 2.5 Gbit POS 2.5 Gbit POS core WorldCom Dark Fibre (SSE) MB-NG Meeting Dec 2001 R. Hughes-Jones Manchester

Some Edge (Bbone) Router requirements Connect to the test systems in the IP domain. Accept marked packets and be able to re-mark them Mark the packets on input according to at least: Source – dest IP address IP port Layer3 and Layer4 headers TCP/UDP Support Diffserve code points Perform policing and admission control Provide a range of Queue type options – Several queues with different priorities Real-time queue Provide flexible Queue Scheduling Weighted round robin Weighted fair queuing Provide Congestion control such as RED ECN Be able to assign MPLS labels to the packets in a flexible manner. MB-NG Meeting Dec 2001 R. Hughes-Jones Manchester

Equipment for the External / Edge Domains Functionality Load PC Test PC Difserve router OR MPLS Edge Router 4 * 1 Gigabit Ethernet blade QoS Marking Gigabit Ethernet 2.5 Gbit POS 2.5 Gbit POS core SJDN Core MPLS Backbone Router 12416 3 * 1 Gigabit Ethernet blade 10 Gbit POS MPLS MPLS Backbone Router OSR IP or Supervisor Classify packets QoS Difserv QoS – WRED MPLS Generate MPLS frames Multi-loop MB-NG Meeting Dec 2001 R. Hughes-Jones Manchester

Equipment for the External / Edge Domains Connectivity Gigabit Ethernet 2.5 Gbit POS 2.5 GbitPOS core Spy 3508 Monitor PC Load PC Test PC Ethernet Switch 3512 Local site Ethernet Test PC 4 * 1 Gigabit Ethernet blade MPLS 4 * 1 Gigabit Ethernet blade QoS Marking Test PC Test PC Load PC MPLS to core 2.5 Gbit POS MPLS Load PC Supervisor Supervisor Load PC Difserve router OR MPLS Edge Router MPLS Backbone Router OSR MB-NG Meeting Dec 2001 R. Hughes-Jones Manchester

IP Access to MB-NG Systems A suggestion only – we all must decide ! End/Test systems routing: The default route from the PC is set to the Gigabit link i.e. into the development network. Static routes for researchers only are set for the 100 Mbit interface, e.g. to Manchester HEP, MCC, RAL, UCL, ULCC, UKERNA. The link between the Cisco 3512 and the Campus is set to 10 Mbit – this would minimise any unintentional traffic. IP access to core routers for researchers only Similar approach? via the test PCs at the CPoP MB-NG Meeting Dec 2001 R. Hughes-Jones Manchester

R. Hughes-Jones Manchester Some Questions ??? Can one Gigabit Ethernet blade perform the IP and MPLS functions we need or do we require two cards – one for IP input functions, the other for MPLS output?  What is the performance of the 4-port Gigabit Ethernet interfaces? Confirm that an interface can support both IP and MPLS frames at same time. Can the GSRs use MPLS to loop the frames round n times? Can all our “requirements be” met? MB-NG Meeting Dec 2001 R. Hughes-Jones Manchester

Focus on the routers - Hardware Cisco OSR based on 6500 Are the details correct ? More memory MB-NG Meeting Dec 2001 R. Hughes-Jones Manchester

Focus on the routers - Summary   Focus on the routers - Summary MB-NG Meeting Dec 2001 R. Hughes-Jones Manchester  

Hardware & Software Tools High Bandwidth High Throughput Protocol Investigations (mainly) 4 * 1 Gigabit Ethernet blade Test PC 2.5 Gbit POS Monitor PC NIC EthernetSwitch 3508 Monitor port 4 * 1 Gigabit Ethernet blade Test PC 2.5 Gbit POS Monitor PC NIC Optical splitter or opto-electrical tap opto-electrical tap constructed for logic analyser Software use of : tcpdump tcptrace Web100 MB-NG Meeting Dec 2001 R. Hughes-Jones Manchester

Latency Tests with 64-bit PCI Alteon Gigabit NIC + RAW SuperMicro 370DLE ServerWorks III LE Chipset 64 bit PCI @ 33 MHz Request – Response Transfer rates: observe expect us/B 32+32 bit 0.0265 0.0232 -> 108Mbyte/s 32+64 bit 0.0209 0.0211 64+64 bit 0.0149 0.0156 64 byte Request 1400 byte Response MB-NG Meeting Dec 2001 R. Hughes-Jones Manchester

Throughput Tests with 64-bit PCI RAW Stream Not using IP stack Vary time between packets 1.6 factor improvement over 32-32 bit transfers MB-NG Meeting Dec 2001 R. Hughes-Jones Manchester

Throughput Tests UDP SysKonnect UDP Stream Vary time between packets 660-690 Mbit/s @ 17us No interrupt coalescing 15 μs between packets 1400 byte sent 1400 byte received Pci-pci latency 28.3 μs MB-NG Meeting Dec 2001 R. Hughes-Jones Manchester

Latency Tests UDP SysKonnect UDP request - response Latency and transfer rates very similar to raw frames observe us/B offset us 64+64 bit 0.0225 46.88 64+64 bit 0.0149 54. (raw) 1400 byte sent 10.66 μs pci time 1400 byte received 10.96 μs pci time Pci-pci latency 28.3 μs MB-NG Meeting Dec 2001 R. Hughes-Jones Manchester

Throughput Tests UDP Alteon UDP Stream Vary time between packets 880-930 Mbit/s @ 11us interrupt coalescing 15 μs between packets MB-NG Meeting Dec 2001 R. Hughes-Jones Manchester

Latency Tests UDP Alteon UDP request – response Due to interrupt coalescing MB-NG Meeting Dec 2001 R. Hughes-Jones Manchester