1. NEGST2008 @ Toulouse 2008/6/24 Network monitoring on 10GbE Y. Kodama ITRI (Information Technology Research Institute) AIST (National Institute of Advanced Industrial Science and Technology)

2. NEGST2008 @ Toulouse 2008/6/24 ITRI, AIST GTRC (Grid Technology Research Center) was finished March 2008. Research activities of GTRC were merged to ITRI (Information Technology Research Institute), and continues research of Grid middleware and infrastructure. The director of ITRI is Sekiguchi who was the director of GTRC.

3. NEGST2008 @ Toulouse 2008/6/24 GtrcNET-10 GtrcNET is a fully programmable hardware network box. GtrcNET-1, which has 4 ports of GbE, was produced in 2003. GtrcNET-10, which has 3 ports of 10GbE, was produced in 2005.

4. NEGST2008 @ Toulouse 2008/6/24 Architecture of GtrcNET-10p3 FPGA (XC2VP100) 10GbE XENPAK. 10GbE XENPAK 10GbE XENPAK 10GbE MAC 10GbE MAC 10GbE MAC 4bit x 3.125GHz SO-DIMM DDR333 SO-DIMM DDR333 SO-DIMM DDR333 MICTOR USB2.0GPS System ACE/CF 64bit x 156.25MHz 64bit x 162MHz x 2 1GBytes

5. NEGST2008 @ Toulouse 2008/6/24 Functions Delay emulation: Up to 800 ms delay without packet losses by 1 us step. Random frame loss: The rate of frame losses is controlled by 2 -31. Output bandwidth Control: pacing (154Mbps-10Gbps) or shaping / policing by token bucket (1Mbps-10Gbps) Buffer size control: It controls the buffer size storing input frames up to 1 GByte by 1 KByte step. Frame generation: It generates UDP frames with several parameters, such as frame length, bandwidth, burstiness, VLAN tag, etc. Bandwidth measurement: It measures all of input and output bandwidth every 10ms. Two of ports are measured more precisely every 100 us. Per-stream bandwidth measurement: using stream ID specified by any 8 bit field in header, per-stream bandwidth is measured every 1 ms. Frame capture: It captures frames in 1 GBytes of memory. It is available to capture any combination of all frames or filtered frames, and whole fields or selected header fields. Capture Forwarding: Captured header fields are transmitted by frames. The number of captured data in a frame is controlled. Destinations are distributed up to 16 nodes.

6. NEGST2008 @ Toulouse 2008/6/24 Per-stream bandwidth GtrcNET-10p3 (S/N:4) grmpi01grmpi02 DELL 6248 vlan1 DELL 6248 vlan100 1/ge011/xg1 CH0CH1 grmpi03 1/ge02 Str 1 Str 2 1/xg21/ge25 grmpi04 1/ge26 Stream ID Field

7. NEGST2008 @ Toulouse 2008/6/24 Capture frames GtrcNET-10p3 (S/N:4) grmpi01grmpi02 DELL 6248 vlan1 DELL 6248 vlan100 1/ge011/xg1 CH0CH1 1/xg21/ge25 CH2 Capture only one-way traffic Speed of reading captured data is slow via USB. Duration of capture is very short full frames: 800ms for wire-rate traffic select header: 17sec for wire-rate traffic, when first 64bytes in 1500 bytes frame are captured, duplicate Store off-chip memory 1GB CH0.Rx or CH1.Rx gnetctrl3 USB

8. NEGST2008 @ Toulouse 2008/6/24 Capture Forwarding GtrcNET-10p3 (S/N:4) grmpi01grmpi02 DELL 6248 vlan1 DELL 6248 vlan100 1/ge231/xg1 CH0CH1 CH2 1/xg4 grmpi05 1/ge19 eth1: 192.168.2.101 eth1: 192.168.2.102 eth1: 192.168.2.105 (00:07:E9:5C:34:9C) gnetctrl3 USB 150.29.149.156 Application data Captured data 1/xg21/ge25 CH0.Rx or CH1.Rx capture forwarding traffic: 550Mbps if 64 bytes of header in 1500 bytes IP length are captured in wire-rate traffic, and 10 date are packed in a forwarding frame. Store off-chip memory 1GB duplicate

9. NEGST2008 @ Toulouse 2008/6/24 Capture bidirectional traffic (1) GtrcNET-10p3 (S/N:4) grmpi01grmpi02 DELL 6248 vlan1 DELL 6248 vlan100 1/ge231/xg1 CH0CH1 CH2 1/xg4 grmpi05 1/ge19 eth1: 192.168.2.101 eth1: 192.168.2.102 eth1: 192.168.2.105 (00:07:E9:5C:34:9C) gnetctrl3 USB 150.29.149.156 Application data Captured data 1/xg21/ge25 CH0.Rx GtrcNET-10p3 (S/N:2) CH0CH1 CH2 USB grmpi04 eth1: 192.168.2.104 (00:07:E9:3A:C5:9F) CH1.Rx1/ge211/xg3 two GtrcNET are required.

10. NEGST2008 @ Toulouse 2008/6/24 Capture bidirectional traffic (2) GtrcNET-10p3 (S/N:4) grmpi01grmpi02 C4948 vlan2 DELL 6248 vlan100 Gi1/23Te1/49 CH0CH1 CH2 grmpi05 Gi1/19 eth1: 192.168.2.101 eth1: 192.168.2.102 eth1: 192.168.2.105 (00:07:E9:5C:34:9C) gnetctrl3 USB 150.29.149.156 1/xg21/ge25 CH2.Rx Application data Mirrored data Captured data Mirror: Rx/Tx/Both Config:192.168.2.232 Gi1/47 Te1/50 sum of bidirectional traffic should be less than 10Gbps. mirrored port should be able to receive packets.

11. NEGST2008 @ Toulouse 2008/6/24 Capture bidirectional traffic (3) Application data Mirrored data Captured data grmpi01grmpi02 GtrcNET-10p3 (S/N:2) Gi1/23Gi1/25Gi1/18Gi1/32 grmpi04 grmpi05 Gi1/19 Gi1/21 eth1: 192.168.2.101eth1: 192.168.2.102 eth1: 192.168.2.105 (00:07:E9:5C:34:9C) eth1: 192.168.2.104 (00:07:E9:3A:C5:9F) gnetctrl3 USB 150.29.149.156 Config:192.168.2.232 Gi1/47 Mirror :Rx Mirror: Tx Te1/50 CH1 Te1/49 CH0 C4948 vlan1 C4948 vlan2

12. NEGST2008 @ Toulouse 2008/6/24 Capture bidirectional traffic (4) grmpi01grmpi02 DELL 6248 vlan1 DELL 6248 vlan100 GtrcNET-10p3 (S/N:2) 1/ge23 CH0 1/xg2 1/ge251/xg31/xg4 grmpi05 1/ge19 eth1: 192.168.2.101 eth1: 192.168.2.102 eth1: 192.168.2.105 (00:07:E9:5C:34:9C) gnetctrl3 USB 150.29.149.156 CH1 Mirror: Rx, Tx Application data Mirrored data Captured data 1/xg1

13. NEGST2008 @ Toulouse 2008/6/24 Capture bidirectional traffic (5) GtrcNET-10p3 (S/N:4) grmpi01grmpi02 DELL6248 vlan1 DELL 6248 vlan100 1/ge231/xg3 CH0CH1 grmpi05 1/ge19 eth1: 192.168.2.101 eth1: 192.168.2.102 eth1: 192.168.2.105 (00:07:E9:5C:34:9C) gnetctrl3 USB 150.29.149.156 1/xg41/ge25 CH2.Rx Application data Mirrored data Captured data Mirror: Rx, Tx 1/xg2 1/xg1 CH2

14. NEGST2008 @ Toulouse 2008/6/24 Distributed destination Destination are changed in round-robin up to 16 nodes. Traffic and stored data are divided to the destination nodes. grmpi01grmpi02 DELL 6248 vlan1 DELL 6248 vlan100 GtrcNET-10p3 (S/N:2) 1/ge23 1/xg1CH0 1/xg2 1/ge251/xg31/xg4 grmpi05 1/ge19 eth1: 192.168.2.101 eth1: 192.168.2.102 eth1: 192.168.2.105 (00:07:E9:5C:34:9C) gnetctrl3 USB 150.29.149.156 CH1 Mirror: Rx, Tx Application data Mirrored data Captured data grmpi04 Gi1/21 eth1: 192.168.2.104 (00:07:E9:3A:C5:9F)

15. NEGST2008 @ Toulouse 2008/6/24 Packet filtering in capture grmpi01grmpi02 DELL 6248 Cisco 4948 GtrcNET-10p3 (S/N:2) 1/ge1 CH0 1/xg4 1/ge11/xg1Te1/49 grmpi05 1/ge26 eth1: 192.168.2.101 eth1.101:192.168.101.101 eth1.102:192.168.102.101 eth1: 192.168.2.102 eth1.101:192.168.101.102 eth1.102:192.168.102.102 eth1: 192.168.2.105 (00:07:E9:5C:34:9C) gnetctrl3 USB 150.29.149.156 CH1 Mirror: Rx, Tx Application data Mirrored data Captured data 1/xg3 mark = ((pos0 & mask0)== val0) and/or ((pos1 & mask1) == val1)

16. NEGST2008 @ Toulouse 2008/6/24 Example of packet filtering capmark 0 0f 8000 8000: –TTL >= 128 capmark 1 0a ffff 8100 and 0b ffff 0065 –VLAN tag is 101(0x65) capmark 1 15 ffff 1389 or 16 ffff 1389 –Either srcport or dstport is 5001 capmark 2 02 00ff 0001 –Random 1/256  VLAN frame detect mode is supported.

17. NEGST2008 @ Toulouse 2008/6/24 Summary Many monitoring functions are available on GtrcNET-10. Requirements for these functions are given by co-research with AIST and INRIA Lyon. Experimental results and analysis on real network monitoring will be presented by Paulo INRIA Lyon. Please visit our web page http://www.gtrc.aist.go.jp/gnet/