The Network Affandi Singaren, Singapore Doug Pearson Indiana University Internet2 Commons Site Coordinator Training December 3, 2003 National University of Singapore

2 A Sample University Network  LAN  Building network  Campus backbone  WAN intranet  Internet  Internet2

3 Sample Network

4 Wiring and Station Cables  Common practice is unshielded twisted-pair (UTP) according to the specs: CAT 3 [old] supports 10 Mbps Ethernet (10base-T) CAT 5 [modern] supports 10base-T, 100 Mbps (100base-TX) and 1000 Mbps (1000base-T) Ethernet CAT 6 [new] supports CAT 5 applications +

5 Wiring and Station Cables  Actual wire used and quality of installation may vary widely – know your wiring!  Important to consider the station cables Don’t use sub-CAT 5 station cables for 100 Mbps connections.

6 Ethernet LAN

7  10 / 100 / 1000 Mbps  Full- and Half-Duplex Half-duplex: send or receive, one at a time. Full-duplex: send and receive simultaneously. 10 Mbps Ethernet supports half-duplex; full-duplex is not consistently implemented. 100 Mbps supports half- and full-duplex.

8 LAN: Switches vs. Repeaters  Repeaters (hubs) are old technology.  A repeater sends (repeats) packets that are incoming on one port, out all other ports (I know you’re out there somewhere!).  Can only operate in half-duplex mode.  Bandwidth and jitter provided to any single device is highly dependent on the LAN traffic.

9 LAN: Switches vs. Repeaters  A switch learns the MAC addresses of the devices connected to it, and sends packets directly and only to the target end-point.  Provides much more consistent bandwidth and latency (low jitter).  A well-designed switched LAN is important for videoconferencing. Repeater-based LANs should be upgraded to switched for videoconferencing!

10 LAN: Ethernet Duplex Mismatch  “One of the most common causes of performance issues on 10/100Mb Ethernet links is when one port on the link is operating at half-duplex while the other port is operating at full- duplex.”

11 LAN: Ethernet Duplex Mismatch  “There is a silent performance-killer out there, one so inconspicuous that it is hardly ever looked for or even suspected. You could suffer from it and never know it, as it robs a site of performance but not connectivity. This performance-killer has a name: Ethernet duplex mismatch.”

12 LAN: Ethernet Duplex Mismatch  If one end of a connection (device or Ethernet switch) is set for auto- negotiation, and fails to see auto- negotiation at the other end, the former sets itself to the default, half-duplex.  Auto-negotiation can sometimes fail, even when both sides are set to auto (although this isn’t as prevalent as in the past).

13 LAN: Duplex Mismatch – Detection  Microsoft Windows doesn’t display the auto-negotiated duplex setting.  Some routers re-negotiate Auto- duplex, which introduces jitter.

14 LAN: Ethernet Duplex Mismatch

15 LAN: Duplex Mismatch – Detection  Show port statistics on the Ethernet switch. When mismatched, the full-duplex end will report a high level of CRC or alignment errors; the half-duplex end will report a high number of late collisions.  PortAlignFCSXmitRcvUnderSize  ErrErrErrErr  2/   Port Single-Multi-Late-Excess-CarriSenRuntsGiants  CollCollCollColl  2/

16 LAN: Duplex Mismatch- Prevention  Always configure switches and devices according to your local policy. An example policy is: If building wiring is sub-CAT 5, then set switch ports to 10/half If building wiring is CAT 5 or better, then set switch ports and devices to Auto.  Monitor switch port stats and logs

17 Router  Segments LANs into distinct networks and subnetworks, e.g., the distinct red, green, and blue LANs with distinct network numbers.  Segments LANs into broadcast domains

18 Router  Provides interface to the WAN. Intranet, commercial Internet, and Internet2 connections. Typically, every networked device at an Internet2-connected institution has connectivity to Internet2.

19 VLAN  A single, physical LAN can be logically segmented into multiple logical LANs; and,  Physically separate LANs can be made to behave and appear as a single LAN.

20 VLAN  Packets are tagged according to LAN membership, e.g., green LAN, red LAN, and blue LAN.  Ethernet switches establish broadcast domains according to the defined VLAN boundaries.  Routers establish multiple VLANs on a single interface.

21 VLAN

22 VLAN  Modern campus network architectures are tending to move away from traditional router-for-a-building design, to VLAN designs.

23 Old Design Included a lot of Routers routers

24 New Design Includes VLAN Router router

25 WAN Segments ServiceSpeedSample Uses T11.5 Mbpsremote building; extension center DS345 Mbpsinter-campus; Internet (I1) connection OC3155 Mbpsinter-campus; I1 & Internet2 connection OC12622 MbpsI1 backbones; Internet2 connection OC482.4 GbpsI1 and Internet2 backbones Gigabit Ether net 1 Gbpsadvanced inter-campus connections when have access to dark fiber

26 Indiana University Abilene NOC Weathermap 

27 High Performance Research and Education Networks  Internet2 / Abilene  STARTAP and International Networks  US Government-Sponsored Networks

28 Traffic on the Network  Typical university today: IP – TCP – UDP IPX [diminishing] Appletalk [diminishing]

29 Traffic on the LAN  Unicast:one-to-one  Multicast:one-to-many  Broadcast:one-to-every

30 Unicast  Most common traffic  Common applications: mail, Web browsing, file transfer, etc.

31 IP Multicast  A one-to-many mode of transmission  Network numbers through are reserved for multicast.  Examples of multicast applications: Vic/rat videoconferencing Centralized PC software administration tools such as Symantec Ghost

32 IP Multicast – Leak Problems  Beware: high rates of unpruned multicast can adversely affect videoconference performance.  Use a network traffic and protocol analyzer to identify this problem.

33 Broadcast  A one-to-every mode of transmission  Used by network protocols including ARP and IPX, NetBIOS system discovery, and name resolution.  All devices on the network must process every broadcast packet; high broadcast rates can divert processing capacity.  If the broadcast domain is too large or unusually active, the activity required at the end-point to deal with the broadcasts could diminish performance.

34 Broadcast  A healthy network should have less than 100 broadcast packets per second.  Check using a network traffic and protocol analyzer tool.

35 Firewalls  A firewall is a network node that acts to enforce an access control policy between two networks, e.g., between a university intranet and the commercial Internet.  Used to secure IT resources against external attacks and break-ins.  Network-layer firewalls typically make their decisions based upon port numbers and source/destination addresses.  Application-layer firewalls act as proxies.

36 Firewalls  H.323 uses the IP ports: Statically-assigned TCP ports 1718 – 1720 and 1731 for call setup and control. Dynamically-assigned UDP ports in the range of 1024 – for video and audio data streams.  Firewalls don’t allow unrestricted ports. Typical modern firewalls and H.323 don’t get along so well.

37 Firewalls – Solutions for H.323  [bad; non-scaleable] Allow unrestricted ports for specific, known, external IP- addresses.  [better, but still not so good] Use feature of some videoconferencing clients to confine dynamic ports to a specific, narrow range.  [OK, but extra admin work and cost] Use an H.323 application proxy.  [best] Use a firewall that snoops on the H.323 call set-up channels (static ports) and opens ports for the audio/video (dynamic ports) as needed.

38 NATs  Allows multiple computers behind the NAT to share one external network address.  Uses: Alleviate shortage of IP addresses Security – obscures view of the network from outside Flexible network administration  Not commonly used at universities on the campus level. Used somewhat in corporations. Common in small offices and at home – behind DSL, cable modem, or ISDN network service.

39 NATs  Difficult to use H.323 behind NATs.  Some videoconferencing terminals provide features to work with NAT – refer to videoconferencing terminal documentation.

40 Latency  Latency is the time required for a packet to traverse a network from source to destination.  Components of latency include: Propagation delay: the time it takes to traverse the distance of the transmission line; controlled by the speed of light in the media; rule-of-thumb: 20ms San Francisco to New York.

41 Latency  Transmission delay: the time it takes for the source to put a packet on the network. Rule- of-thumb: < 1ms.  Store-and-forward delay: the cumulative length of time it takes the internetworking devices along the path to receive, process, and resend the packets. Rule-of-thumb: variable, and depends upon network load.

42 Latency  Rule of thumb: A one-way delay of: 0 – 150 ms provides excellent interactivity 150 – 300 ms is OK 300 – 400 ms is bad 400+ ms is unacceptable

43 Jitter  Jitter is variation in latency over time.  If the endpoints are on switched LANs, then the primary source of jitter is variation in the store-and-forward time, resulting from network load.  H.323, particularly audio, is adversely affected by high levels of jitter.  What is high? Rule of thumb?

44 Packet Loss  Packet loss is typically due to congested links and routers. 1% is noticeable 5% becomes intolerable

45 QoS  Not currently feasible on commercial Internet and Internet2 networks for production, regular use. Internet2 is working on QoS plans, but the current over-provisioned Internet2 network doesn’t dictate need.  Is useful on over-utilized intranet WAN links.

46 QoS  How: Some videoconferencing terminals can set the IP precedence bits. Use that for marking and priority queuing on the WAN.  Or: Use a H.323 Proxy for consolidation of traffic to a single address, router access list for marking, and priority queuing on the WAN.

47 QoS  Caution! The wrong implementation could result in unwanted tradeoffs, e.g., packet loss improves but jitter gets worse.

48 The End-to-End Performance Problem  Scenario Users on two different campuses of a university are experiencing poor video and audio in a conference. Each user is supported by a different group of videoconferencing engineers. Each campus is supported by a different group of network engineers. The wide-area network is supported by a third group of network engineers.

49 The End-to-End Performance Problem  Problem How do the users get timely, useful assistance? How is network problem resolution coordinated?

50 The End-to-End Performance Problem  Obstacles Different groups, schedules, and priorities. No one engineer has a complete understanding of the entire network path. No one engineer can gain access to all the network nodes (routers, switches) along the path to inspect for trouble. Communications are inconsistent from engineer to engineer.

51 The End-to-End Performance Problem  Solutions Articulate the E-2-E problem to network management and engineers on all campuses. Establish reliable communication tools, and insist that engineers utilize the tools. Hold regular meetings; bring all engineers together in one place and time to share information. Have good network documentation for all networks.

52 H.323 is Network Sensitive!  The big problems are: Half/Full-duplex mismatches Packet loss Jitter Substandard horizontal wiring or station cables Multicast leaks High broadcast rates

53 Tools  Ping – availability, loss, roundtrip time  Traceroute – path discovery  Pingplot – graphical traceroute/ping  MRTG – graph link/port utilization & errors  Iperf – bandwidth, loss and jitter  Gnuplotping – visualize jitter  Sniffer – inspect traffic on the LAN  VideNet Scout – bandwidth, loss and jitter  Internet2 Detective – detect I2 connection  H.323. Beacon – protocol-specific tests

54 Tools: Ping  Test for availability, loss, and roundtrip time  ICMP Echo Request Plus optional dummy payload – only in the direction of the ping, i.e., source  destination

55 Tools: Sample Ping from Windows  C:\WINDOWS>ping  Pinging with 32 bytes of data:  Reply from : bytes=32 time=88ms TTL=112  Reply from : bytes=32 time=72ms TTL=112  Reply from : bytes=32 time=69ms TTL=112  Ping statistics for :  Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),  Approximate round trip times in milli-seconds:  Minimum = 69ms, Maximum = 88ms, Average = 74ms  C:\WINDOWS>

56 Tools: Sample Ping from Windows  C:\WINDOWS>ping -l  Pinging with bytes of data:  Reply from : bytes=40000 time=2412ms TTL=112  Reply from : bytes=40000 time=2721ms TTL=112  Reply from : bytes=40000 time=2761ms TTL=112  Reply from : bytes=40000 time=2714ms TTL=112  Ping statistics for :  Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),  Approximate round trip times in milli-seconds:  Minimum = 2412ms, Maximum = 2761ms, Average = 2652ms  C:\WINDOWS>

57 Tools: Ping Big and Fast on Linux  ping -c2000 -i.03 -s1470 -q [destination] Count of 2000 packets Interval of.03 seconds between packet starts Packet size of 1470 bytes (2000)(.03) = 60 second long test (1/(.03 sec/packet))(1470 bytes/packet)(8 bits/byte) = 392 Kbps

58 Tools: Ping Big and Fast on Linux  $ ping -c2000 -i.03 -s1470 -q [hostname]  PING [hostname] ([hostaddr]) from [hostaddr2] : 1470(1498) bytes of data.  --- [hostname] ping statistics ---  2000 packets transmitted, 2000 packets received, 0% packet loss  round-trip min/avg/max = 4.8/5.1/13.2 ms

59 Tools: Traceroute  Used to discover the layer-3 network path (routers) between the two endpoints  Doesn’t identify layer-2 devices (switches)  Must run from one of the discovery endpoints – it can’t act as a third party.  Take baselines – know what your path should be in advance of trouble

60 Tools: Traceroute  Microsoft tracert uses ICMP packets, whereas Unix traceroute uses UDP; may be of importance in networks where routers are configured to not respond to ICMP; or if ICMP is blocked.

61 Tools: Traceroute; Sample Output  dodpears]$ traceroute  traceroute to ( ), 30 hops max, 38 byte packets  1 wcc-sub5-hp1 ( ) ms ms ms  2 iub-gw ( ) ms ms ms  ( ) ms ms ms  4 abilene-iupui.abilene.ucaid.edu ( ) ms ms ms  5 clev-ipls.abilene.ucaid.edu ( ) ms ms ms  6 nycm-clev.abilene.ucaid.edu ( ) ms ms ms  7 border-abilene-oc3.advanced.org ( ) ms ms ms  8 ( ) ms ms ms

62 Tools: Traceroute  8 ( ) ms ms ms  Hop  Router/host name  Router/host address  Round-trip times of each of three probes

63 Tools: Traceroute – Example Uses  Identify the path, and then perform pings along the path segments to isolate troublesome segments.  Insure that Internet2 is being used for a connection, rather than commercial Internet.

64 Tools: Ping Plotter  Shareware tool; $15  Performs a visual traceroute and ping tests along the entire path  Permits identification of bottlenecks along a path

65 Tools: Ping Plotter

66 Tools: MRTG  Multi Router Traffic Grapher  Collect and graph scalar, time-based data, e.g., router and link performance data.

67 Tools: MRTG

68 Tools: MRTG

69 Tools: Iperf   Client/server application that Measures maximum TCP bandwidth Facilitates tuning of TCP and UDP parameters Reports bandwidth, jitter, and packet loss

70 Tools: Iperf Example on Intercampus DS3  At server, invoke:  iperf -fk -i30 -u -s (f)ormat reports in kbps (i)nterval for reporting = 30 seconds (u)dp (s)erver mode

71 Tools: Iperf Example on Intercampus DS3  At client, invoke:  iperf -u -b800k -t3600 -c [hostname- server] (u)dp (b)andwidth = 800kbps (t)ime of run = 3600 seconds (c)lient mode [hostname-server] = server to target

72 Tools: Iperf Example on Intercampus DS3  iperf]$ iperf -fk -i30 -u -s   Server listening on UDP port 5001  Receiving 1470 byte datagrams  UDP buffer size: 64.0 KByte (default)   [ 3] local port 5001 connected with port 1031  [ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams  [ 3] sec 3000 KBytes 819 Kbits/sec ms 0/ 2090 (0%)  [ 3] sec 3000 KBytes 819 Kbits/sec ms 0/ 2090 (0%)  [ 3] sec 3000 KBytes 819 Kbits/sec ms 0/ 2090 (0%)  [...]  [ 3] sec 9000 KBytes 819 Kbits/sec ms 0/ 6393 (0%)

73 Iperf- Example on Intercampus DS3  MRTG utilization graph showed bandwidth peaking at capacity ~ 10:00a – 2:00p  As utilization peaked on the DS3, jitter measured by Iperf rose to unacceptable level  Iperf also reported periodic high packet loss, with no apparent correlation to the low-resolution MRTG utilization reports

74 Iperf- Example on Intercampus DS3  Second day, utilization as reported by MRTG is staying reasonable.  Jitter measured by Iperf is staying low.  The periodic high packet loss remains, until noon when network engineer adjusted the QoS settings.

75  Third day, utilization peaking at 3:00p.  As utilization peaked jitter measured by Iperf also rising.  Still no packet loss.  QoS fixed the packet loss problem, but still not certain about the jitter – more analysis needed. Iperf- Example on Intercampus DS3

76 Tools: Gnuplotping  Pings multiple hosts in parallel with graphical display (gnuplot) of the delay distribution.  Runs on Unix/X-Windows

77 Tools: gnuplotping

78 Tools: Network Traffic Analyzer  Reveals the traffic on a LAN  Protocol analysis  Reports such as utilization, protocols, conversations, nodes, etc.  Network General Sniffer  WildPackets EtherPeek

79 Tools: ViDeNet Scout  Scout is a web-based, distributed network performance analysis tool developed at the University of North Carolina at Chapel Hill.  Scout makes use of the Chariot performance testing engine developed by NetIQ.

80 Tools: Internet2 Detective  I2 Detective is a small application.  Detects Internet2 connection.  Measures connection bandwidth (using Iperf).  Detects multicast connection.

81 Tools: H.323 Beacon  Used to measure, monitor and qualify the performance of an H.323 Videoconference session.  Provides H.323-protocol specific evidence and other information necessary to troubleshoot H.323 application performance problems in the network and at the host (end-to-end)

82 Recommendations  Develop a close relationship with the network engineers and NOC. Make sure they understand what’s being done with videoconferencing and the network sensitivity of IP-based video.  Articulate the End-to-End Performance Problem to network engineering and operations management. Champion ways to reduce the problem.  Be sure to open trouble tickets with your NOC so that a problem history is maintained.

83 Recommendations  Use switched Ethernet.  Watch out for duplex mismatches.  Keep an eye on utilization of WAN links, packet loss, and jitter.  Make sure you don’t have broadcast or multicast leaking problems.  Make sure wiring is up to the task.

84 Recommendations  Have engineers in the videoconferencing support group trained to understand networking issues and tools.