1. The Network Affandi Singaren, Singapore Doug Pearson Indiana University Internet2 Commons Site Coordinator Training December 3, 2003 National University of Singapore

2. 2 A Sample University Network  LAN  Building network  Campus backbone  WAN intranet  Internet  Internet2

3. 3 Sample Network

4. 4 Wiring and Station Cables  Common practice is unshielded twisted-pair (UTP) according to the specs: CAT 3 [old] supports 10 Mbps Ethernet (10base-T) CAT 5 [modern] supports 10base-T, 100 Mbps (100base-TX) and 1000 Mbps (1000base-T) Ethernet CAT 6 [new] supports CAT 5 applications +

5. 5 Wiring and Station Cables  Actual wire used and quality of installation may vary widely – know your wiring!  Important to consider the station cables Don’t use sub-CAT 5 station cables for 100 Mbps connections.

6. 6 Ethernet LAN

7. 7  10 / 100 / 1000 Mbps  Full- and Half-Duplex Half-duplex: send or receive, one at a time. Full-duplex: send and receive simultaneously. 10 Mbps Ethernet supports half-duplex; full-duplex is not consistently implemented. 100 Mbps supports half- and full-duplex.

8. 8 LAN: Switches vs. Repeaters  Repeaters (hubs) are old technology.  A repeater sends (repeats) packets that are incoming on one port, out all other ports (I know you’re out there somewhere!).  Can only operate in half-duplex mode.  Bandwidth and jitter provided to any single device is highly dependent on the LAN traffic.

9. 9 LAN: Switches vs. Repeaters  A switch learns the MAC addresses of the devices connected to it, and sends packets directly and only to the target end-point.  Provides much more consistent bandwidth and latency (low jitter).  A well-designed switched LAN is important for videoconferencing. Repeater-based LANs should be upgraded to switched for videoconferencing!

10. 10 LAN: Ethernet Duplex Mismatch  “One of the most common causes of performance issues on 10/100Mb Ethernet links is when one port on the link is operating at half-duplex while the other port is operating at full- duplex.” http://www.cisco.com/warp/public/473/3.html

11. 11 LAN: Ethernet Duplex Mismatch  “There is a silent performance-killer out there, one so inconspicuous that it is hardly ever looked for or even suspected. You could suffer from it and never know it, as it robs a site of performance but not connectivity. This performance-killer has a name: Ethernet duplex mismatch.” http://www.hostingtech.com/nm/01_01_mismatch.html

12. 12 LAN: Ethernet Duplex Mismatch  If one end of a connection (device or Ethernet switch) is set for auto- negotiation, and fails to see auto- negotiation at the other end, the former sets itself to the default, half-duplex.  Auto-negotiation can sometimes fail, even when both sides are set to auto (although this isn’t as prevalent as in the past).

13. 13 LAN: Duplex Mismatch – Detection  Microsoft Windows doesn’t display the auto-negotiated duplex setting.  Some routers re-negotiate Auto- duplex, which introduces jitter.

14. 14 LAN: Ethernet Duplex Mismatch

15. 15 LAN: Duplex Mismatch – Detection  Show port statistics on the Ethernet switch. When mismatched, the full-duplex end will report a high level of CRC or alignment errors; the half-duplex end will report a high number of late collisions.  PortAlignFCSXmitRcvUnderSize  ErrErrErrErr  2/11-0030770   Port Single-Multi-Late-Excess-CarriSenRuntsGiants  CollCollCollColl  2/11 3233025880024890

16. 16 LAN: Duplex Mismatch- Prevention  Always configure switches and devices according to your local policy. An example policy is: If building wiring is sub-CAT 5, then set switch ports to 10/half If building wiring is CAT 5 or better, then set switch ports and devices to Auto.  Monitor switch port stats and logs

17. 17 Router  Segments LANs into distinct networks and subnetworks, e.g., the distinct red, green, and blue LANs with distinct network numbers.  Segments LANs into broadcast domains

18. 18 Router  Provides interface to the WAN. Intranet, commercial Internet, and Internet2 connections. Typically, every networked device at an Internet2-connected institution has connectivity to Internet2.

19. 19 VLAN  A single, physical LAN can be logically segmented into multiple logical LANs; and,  Physically separate LANs can be made to behave and appear as a single LAN.

20. 20 VLAN  Packets are tagged according to LAN membership, e.g., green LAN, red LAN, and blue LAN.  Ethernet switches establish broadcast domains according to the defined VLAN boundaries.  Routers establish multiple VLANs on a single interface.

21. 21 VLAN

22. 22 VLAN  Modern campus network architectures are tending to move away from traditional router-for-a-building design, to VLAN designs.

23. 23 Old Design Included a lot of Routers routers

24. 24 New Design Includes VLAN Router router

25. 25 WAN Segments ServiceSpeedSample Uses T11.5 Mbpsremote building; extension center DS345 Mbpsinter-campus; Internet (I1) connection OC3155 Mbpsinter-campus; I1 & Internet2 connection OC12622 MbpsI1 backbones; Internet2 connection OC482.4 GbpsI1 and Internet2 backbones Gigabit Ether net 1 Gbpsadvanced inter-campus connections when have access to dark fiber

26. 26 Indiana University Abilene NOC Weathermap 

27. 27 High Performance Research and Education Networks  Internet2 / Abilene http://www.internet2.edu http://www.abilene.iu.edu  STARTAP and International Networks http://www.startap.net  US Government-Sponsored Networks http://www.startap.net/NETWORKS

28. 28 Traffic on the Network  Typical university today: IP – TCP – UDP IPX [diminishing] Appletalk [diminishing]

29. 29 Traffic on the LAN  Unicast:one-to-one  Multicast:one-to-many  Broadcast:one-to-every

30. 30 Unicast  Most common traffic  Common applications: mail, Web browsing, file transfer, etc.

31. 31 IP Multicast  A one-to-many mode of transmission  Network numbers 224.0.0.0 through 239.255.255.255 are reserved for multicast.  Examples of multicast applications: Vic/rat videoconferencing Centralized PC software administration tools such as Symantec Ghost

32. 32 IP Multicast – Leak Problems  Beware: high rates of unpruned multicast can adversely affect videoconference performance.  Use a network traffic and protocol analyzer to identify this problem.

33. 33 Broadcast  A one-to-every mode of transmission  Used by network protocols including ARP and IPX, NetBIOS system discovery, and name resolution.  All devices on the network must process every broadcast packet; high broadcast rates can divert processing capacity.  If the broadcast domain is too large or unusually active, the activity required at the end-point to deal with the broadcasts could diminish performance.

34. 34 Broadcast  A healthy network should have less than 100 broadcast packets per second.  Check using a network traffic and protocol analyzer tool.

35. 35 Firewalls  A firewall is a network node that acts to enforce an access control policy between two networks, e.g., between a university intranet and the commercial Internet.  Used to secure IT resources against external attacks and break-ins.  Network-layer firewalls typically make their decisions based upon port numbers and source/destination addresses.  Application-layer firewalls act as proxies.

36. 36 Firewalls  H.323 uses the IP ports: Statically-assigned TCP ports 1718 – 1720 and 1731 for call setup and control. Dynamically-assigned UDP ports in the range of 1024 – 65535 for video and audio data streams.  Firewalls don’t allow unrestricted ports. Typical modern firewalls and H.323 don’t get along so well.

37. 37 Firewalls – Solutions for H.323  [bad; non-scaleable] Allow unrestricted ports for specific, known, external IP- addresses.  [better, but still not so good] Use feature of some videoconferencing clients to confine dynamic ports to a specific, narrow range.  [OK, but extra admin work and cost] Use an H.323 application proxy.  [best] Use a firewall that snoops on the H.323 call set-up channels (static ports) and opens ports for the audio/video (dynamic ports) as needed.

38. 38 NATs  Allows multiple computers behind the NAT to share one external network address.  Uses: Alleviate shortage of IP addresses Security – obscures view of the network from outside Flexible network administration  Not commonly used at universities on the campus level. Used somewhat in corporations. Common in small offices and at home – behind DSL, cable modem, or ISDN network service.

39. 39 NATs  Difficult to use H.323 behind NATs.  Some videoconferencing terminals provide features to work with NAT – refer to videoconferencing terminal documentation.

40. 40 Latency  Latency is the time required for a packet to traverse a network from source to destination.  Components of latency include: Propagation delay: the time it takes to traverse the distance of the transmission line; controlled by the speed of light in the media; rule-of-thumb: 20ms San Francisco to New York.

41. 41 Latency  Transmission delay: the time it takes for the source to put a packet on the network. Rule- of-thumb: < 1ms.  Store-and-forward delay: the cumulative length of time it takes the internetworking devices along the path to receive, process, and resend the packets. Rule-of-thumb: variable, and depends upon network load.

42. 42 Latency  Rule of thumb: A one-way delay of: 0 – 150 ms provides excellent interactivity 150 – 300 ms is OK 300 – 400 ms is bad 400+ ms is unacceptable

43. 43 Jitter  Jitter is variation in latency over time.  If the endpoints are on switched LANs, then the primary source of jitter is variation in the store-and-forward time, resulting from network load.  H.323, particularly audio, is adversely affected by high levels of jitter.  What is high? Rule of thumb?

44. 44 Packet Loss  Packet loss is typically due to congested links and routers. 1% is noticeable 5% becomes intolerable

45. 45 QoS  Not currently feasible on commercial Internet and Internet2 networks for production, regular use. Internet2 is working on QoS plans, but the current over-provisioned Internet2 network doesn’t dictate need.  Is useful on over-utilized intranet WAN links.

46. 46 QoS  How: Some videoconferencing terminals can set the IP precedence bits. Use that for marking and priority queuing on the WAN.  Or: Use a H.323 Proxy for consolidation of traffic to a single address, router access list for marking, and priority queuing on the WAN.

47. 47 QoS  Caution! The wrong implementation could result in unwanted tradeoffs, e.g., packet loss improves but jitter gets worse.

48. 48 The End-to-End Performance Problem  Scenario Users on two different campuses of a university are experiencing poor video and audio in a conference. Each user is supported by a different group of videoconferencing engineers. Each campus is supported by a different group of network engineers. The wide-area network is supported by a third group of network engineers.

49. 49 The End-to-End Performance Problem  Problem How do the users get timely, useful assistance? How is network problem resolution coordinated?

50. 50 The End-to-End Performance Problem  Obstacles Different groups, schedules, and priorities. No one engineer has a complete understanding of the entire network path. No one engineer can gain access to all the network nodes (routers, switches) along the path to inspect for trouble. Communications are inconsistent from engineer to engineer.

51. 51 The End-to-End Performance Problem  Solutions Articulate the E-2-E problem to network management and engineers on all campuses. Establish reliable communication tools, and insist that engineers utilize the tools. Hold regular meetings; bring all engineers together in one place and time to share information. Have good network documentation for all networks.

52. 52 H.323 is Network Sensitive!  The big problems are: Half/Full-duplex mismatches Packet loss Jitter Substandard horizontal wiring or station cables Multicast leaks High broadcast rates

53. 53 Tools  Ping – availability, loss, roundtrip time  Traceroute – path discovery  Pingplot – graphical traceroute/ping  MRTG – graph link/port utilization & errors  Iperf – bandwidth, loss and jitter  Gnuplotping – visualize jitter  Sniffer – inspect traffic on the LAN  VideNet Scout – bandwidth, loss and jitter  Internet2 Detective – detect I2 connection  H.323. Beacon – protocol-specific tests

54. 54 Tools: Ping  Test for availability, loss, and roundtrip time  ICMP Echo Request Plus optional dummy payload – only in the direction of the ping, i.e., source  destination

55. 55 Tools: Sample Ping from Windows  C:\WINDOWS>ping 10.1.1.1  Pinging 10.1.1.1 with 32 bytes of data:  Reply from 10.1.1.1: bytes=32 time=88ms TTL=112  Reply from 10.1.1.1: bytes=32 time=72ms TTL=112  Reply from 10.1.1.1: bytes=32 time=69ms TTL=112  Ping statistics for 10.1.1.1:  Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),  Approximate round trip times in milli-seconds:  Minimum = 69ms, Maximum = 88ms, Average = 74ms  C:\WINDOWS>

56. 56 Tools: Sample Ping from Windows  C:\WINDOWS>ping -l 40000 10.1.1.1  Pinging 10.1.1.1 with 40000 bytes of data:  Reply from 10.1.1.1: bytes=40000 time=2412ms TTL=112  Reply from 10.1.1.1: bytes=40000 time=2721ms TTL=112  Reply from 10.1.1.1: bytes=40000 time=2761ms TTL=112  Reply from 10.1.1.1: bytes=40000 time=2714ms TTL=112  Ping statistics for 10.1.1.1:  Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),  Approximate round trip times in milli-seconds:  Minimum = 2412ms, Maximum = 2761ms, Average = 2652ms  C:\WINDOWS>

57. 57 Tools: Ping Big and Fast on Linux  ping -c2000 -i.03 -s1470 -q [destination] Count of 2000 packets Interval of.03 seconds between packet starts Packet size of 1470 bytes (2000)(.03) = 60 second long test (1/(.03 sec/packet))(1470 bytes/packet)(8 bits/byte) = 392 Kbps

58. 58 Tools: Ping Big and Fast on Linux  $ ping -c2000 -i.03 -s1470 -q [hostname]  PING [hostname] ([hostaddr]) from [hostaddr2] : 1470(1498) bytes of data.  --- [hostname] ping statistics ---  2000 packets transmitted, 2000 packets received, 0% packet loss  round-trip min/avg/max = 4.8/5.1/13.2 ms

59. 59 Tools: Traceroute  Used to discover the layer-3 network path (routers) between the two endpoints  Doesn’t identify layer-2 devices (switches)  Must run from one of the discovery endpoints – it can’t act as a third party.  Take baselines – know what your path should be in advance of trouble

60. 60 Tools: Traceroute  Microsoft tracert uses ICMP packets, whereas Unix traceroute uses UDP; may be of importance in networks where routers are configured to not respond to ICMP; or if ICMP is blocked.

61. 61 Tools: Traceroute; Sample Output  [dodpears@huck dodpears]$ traceroute www.internet2.edu  traceroute to www.internet2.edu (209.211.239.208), 30 hops max, 38 byte packets  1 wcc-sub5-hp1 (129.79.5.253) 11.726 ms 0.627 ms 0.571 ms  2 iub-gw (129.79.8.10) 3.133 ms 0.717 ms 0.651 ms  3 156.56.249.22 (156.56.249.22) 2.544 ms 3.138 ms 2.538 ms  4 abilene-iupui.abilene.ucaid.edu (198.32.11.13) 5.245 ms 3.402 ms 3.493 ms  5 clev-ipls.abilene.ucaid.edu (198.32.8.26) 9.381 ms 9.586 ms 9.244 ms  6 nycm-clev.abilene.ucaid.edu (198.32.8.30) 23.198 ms 21.963 ms 21.775 ms  7 border-abilene-oc3.advanced.org (209.211.237.97) 23.448 ms 23.268 ms 23.052 ms  8 www.internet2.edu (209.211.239.208) 23.559 ms 23.478 ms 23.234 ms

62. 62 Tools: Traceroute  8 www.internet2.edu (209.211.239.208) 23.559 ms 23.478 ms 23.234 ms  Hop  Router/host name  Router/host address  Round-trip times of each of three probes

63. 63 Tools: Traceroute – Example Uses  Identify the path, and then perform pings along the path segments to isolate troublesome segments.  Insure that Internet2 is being used for a connection, rather than commercial Internet.

64. 64 Tools: Ping Plotter  Shareware tool; $15 http://www.pingplotter.com  Performs a visual traceroute and ping tests along the entire path  Permits identification of bottlenecks along a path

65. 65 Tools: Ping Plotter

66. 66 Tools: MRTG  Multi Router Traffic Grapher  Collect and graph scalar, time-based data, e.g., router and link performance data.

67. 67 Tools: MRTG

68. 68 Tools: MRTG

69. 69 Tools: Iperf  http://dast.nlanr.net/Projects/Iperf/  Client/server application that Measures maximum TCP bandwidth Facilitates tuning of TCP and UDP parameters Reports bandwidth, jitter, and packet loss

70. 70 Tools: Iperf Example on Intercampus DS3  At server, invoke:  iperf -fk -i30 -u -s (f)ormat reports in kbps (i)nterval for reporting = 30 seconds (u)dp (s)erver mode

71. 71 Tools: Iperf Example on Intercampus DS3  At client, invoke:  iperf -u -b800k -t3600 -c [hostname- server] (u)dp (b)andwidth = 800kbps (t)ime of run = 3600 seconds (c)lient mode [hostname-server] = server to target

72. 72 Tools: Iperf Example on Intercampus DS3  [dodpears@vc-iperf iperf]$ iperf -fk -i30 -u -s  ------------------------------------------------------------  Server listening on UDP port 5001  Receiving 1470 byte datagrams  UDP buffer size: 64.0 KByte (default)  ------------------------------------------------------------  [ 3] local 149.166.197.80 port 5001 connected with 129.79.92.230 port 1031  [ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams  [ 3] 0.0-30.0 sec 3000 KBytes 819 Kbits/sec 0.300 ms 0/ 2090 (0%)  [ 3] 30.0-60.0 sec 3000 KBytes 819 Kbits/sec 0.242 ms 0/ 2090 (0%)  [ 3] 60.0-90.0 sec 3000 KBytes 819 Kbits/sec 0.338 ms 0/ 2090 (0%)  [...]  [ 3] 0.0-90.0 sec 9000 KBytes 819 Kbits/sec 0.263 ms 0/ 6393 (0%)

73. 73 Iperf- Example on Intercampus DS3  MRTG utilization graph showed bandwidth peaking at capacity ~ 10:00a – 2:00p  As utilization peaked on the DS3, jitter measured by Iperf rose to unacceptable level  Iperf also reported periodic high packet loss, with no apparent correlation to the low-resolution MRTG utilization reports

74. 74 Iperf- Example on Intercampus DS3  Second day, utilization as reported by MRTG is staying reasonable.  Jitter measured by Iperf is staying low.  The periodic high packet loss remains, until noon when network engineer adjusted the QoS settings.

75. 75  Third day, utilization peaking at 3:00p.  As utilization peaked jitter measured by Iperf also rising.  Still no packet loss.  QoS fixed the packet loss problem, but still not certain about the jitter – more analysis needed. Iperf- Example on Intercampus DS3

76. 76 Tools: Gnuplotping  Pings multiple hosts in parallel with graphical display (gnuplot) of the delay distribution.  Runs on Unix/X-Windows

77. 77 Tools: gnuplotping

78. 78 Tools: Network Traffic Analyzer  Reveals the traffic on a LAN  Protocol analysis  Reports such as utilization, protocols, conversations, nodes, etc.  Network General Sniffer  WildPackets EtherPeek

79. 79 Tools: ViDeNet Scout  Scout is a web-based, distributed network performance analysis tool developed at the University of North Carolina at Chapel Hill.  Scout makes use of the Chariot performance testing engine developed by NetIQ. http://scout.video.unc.edu/

80. 80 Tools: Internet2 Detective  I2 Detective is a small application.  Detects Internet2 connection.  Measures connection bandwidth (using Iperf).  Detects multicast connection. http://detective.internet2.edu/

81. 81 Tools: H.323 Beacon  Used to measure, monitor and qualify the performance of an H.323 Videoconference session.  Provides H.323-protocol specific evidence and other information necessary to troubleshoot H.323 application performance problems in the network and at the host (end-to-end) http://www.itecohio.org/beacon/

82. 82 Recommendations  Develop a close relationship with the network engineers and NOC. Make sure they understand what’s being done with videoconferencing and the network sensitivity of IP-based video.  Articulate the End-to-End Performance Problem to network engineering and operations management. Champion ways to reduce the problem.  Be sure to open trouble tickets with your NOC so that a problem history is maintained.

83. 83 Recommendations  Use switched Ethernet.  Watch out for duplex mismatches.  Keep an eye on utilization of WAN links, packet loss, and jitter.  Make sure you don’t have broadcast or multicast leaking problems.  Make sure wiring is up to the task.

84. 84 Recommendations  Have engineers in the videoconferencing support group trained to understand networking issues and tools.