1. Network Measurements Session Introduction Joe Metzger Network Engineering Group ESnet Eric Boyd Deputy Technology Officer Internet2 July 16 2007 Joint Techs at FERMI

2. Why is Network Measurement Important? Users dependence on the network is increasing –Distributed Applications –Moving Larger Data Sets –The network is becoming a critical part of large science experiments The network is growing much more complex –ESnet had 6 core devices in 05’, 25+ in 08’ –ESnet had 6 core links in 05’, 40+ in 08’, 80+ by 2010? –Dynamic Circuits –Network Security Issues The community needs to better understand the network –Users must know what performance levels to expect. –Network Operators need to be able to demonstrate that the network meets or exceeds those expectations. –Application Developers must understand the ‘wizards gap’ and have access to tools that differentiate between network problems and application problems.

3. Data Transfer times over R&E Networks 10PB300,240.0 Gbps25,020.0 Gbps 3,127.5 Gbps1,042.5 Gbps148.9 Gbps34.7 Gbps 1PB30,024.0 Gbps2,502.0 Gbps312.7 Gbps104.2 Gbps14.9 Gbps3.5 Gbps 100TB2,932.0 Gbps244.3 Gbps30.5 Gbps10.2 Gbps1.5 Gbps339.4 Mbps 10TB293.2 Gbps24.4 Gbps3.1 Gbps1.0 Gbps145.4 Mbps33.9 Mbps 1TB29.3 Gbps2.4 Gbps305.4 Mbps101.8 Mbps14.5 Mbps3.4 Mbps 100GB2.9 Gbps238.6 Mbps29.8 Mbps9.9 Mbps1.4 Mbps331.4 Kbps 10GB286.3 Mbps23.9 Mbps3.0 Mbps994.2 Kbps142.0 Kbps33.1 Kbps 1GB28.6 Mbps2.4 Mbps298.3 Kbps99.4 Kbps14.2 Kbps3.3 Kbps 100M B2.8 Mbps233.0 Kbps29.1 Kbps9.7 Kbps1.4 Kbps0.3 Kbps 5 Minutes1 Hour8 Hours24 Hours7 Days30 Days RED: Something is broken! Usually TCP tuning or HW problems within 100 feet of end points. GREEN: Supported by R&E Backbones today (may have local campus challenges) WHITE: Requires special engineering.

4. TCP Throughput Limits

5. Gbps on Campus with any window size Need 1 MB Windows to Get 100 Mbps Cross country Default OS Window sizes. Is this enough For you?

6. Scale of the Integration Challenge Measurement infrastructure needs to: –Obey agreed-upon protocols (schema and semantics) –Be interoperable across administrative boundaries –Integrate with middleware (federated trust) infrastructure –Integrate with circuit provisioning software

7. Scale of the Deployment Challenge Universities, national labs, regionals, and national backbones are all autonomous Measurement infrastructure needs to: –Be deployed widely (Metcalf’s Law) –Be locally controlled –Work well with existing local infrastructure –Integrate easily into local processes

8. Internet2 Connectors 8 MAGPI 3ROX CalREN-2 South Great Plains Network Indiana GigaPoP MREN Merit LONI Internet2 ESnet NoX NYSERNet OARnet OmniPoP SoX Oregon GigaPoP Pacific Northwest GigaPoP

9. ESnet Connects 9 Lawrence Livermore National Lab (T3) Brookhaven National Lab (T1) SLAC (T2) Fermi National Accelerator Lab (T1) Lawrence Berkeley National Lab (T3) ESnet Argonne National Lab (T3)

10. Nine Universities Connect through CalREN-2 South 10 University of Arizona (T3) UC Irvine (T3) UC Santa Cruz (T3) UC Davis (T2) UCLA (T3) UC Riverside (T3) UC San Diego(T3) UC Santa Barbara (T3) California Institute of Technology (T2) CENIC

11. Universities Connecting through Oregon GigaPoP and Pacific NW GigaPoP 11 University of Oregon (T3) University of Washington (T3) Oregon GigaPOP Pacific Northwest GigaPOP

12. Four Universities Connect through LONI 12 University of Texas, Arlington (T2) LONI University of Texas, Dallas (T3) Southern Methodist University (T3) University of Mississippi (T3)

13. Seven Universities Connect through Great Plains Network 13 Oklahoma State University (T3) Great Plains Network University of Nebraska-Lincoln (T2) University of Kansas (T3) University of Oklahoma (T2) Kansas State University (T3) University of Iowa (T3) Iowa State University (T3)

14. Two Universities Connect through OmniPoP 14 University of Wisconsin, Madison (T3) OmniPoP University of Wisconsin, Milwaukee (T2)

15. Five Universities Connect through MREN 15 University of Notre Dame (T3) University of Illinois at Chicago (T3) University of Chicago (T2) Northwestern University (T3) Univ of Illinois, Urbana-Champaign (T3) MREN

16. Universities that Connect through Indiana GigaPoP and OARnet 16 Indiana University (T2) Indiana GigaPoP Purdue University (T2) OARnet Ohio State University (T3)

17. Two Universities Connect through Merit 17 Michigan State University (T2) Merit University of Michigan (T2)

18. Eight Universities Connect through SoX 18 Florida International University (T3) Duke University (T3) University of Florida (T2) Vanderbilt University (T3) University of Puerto Rico (T3) Florida State University (T3) University of Tennessee (T3) University of South Carolina (T3) SoX

19. Two Universities Connect through 3ROX 19 Carnegie Mellon University (T3) 3ROX University of Pittsburgh (T3)

20. Three Universities Connect through MAGPI 20 Princeton University (T3) MAGPI University of Pennsylvania (T3) Rutgers University (T3)

21. Seven Universities Connect through NYSERNet 21 SUNY Albany (T3) Columbia University (T3) New York University (T3) University of Rochester (T3) SUNY Stony Brook (T3) Cornell University (T3) SUNY Buffalo (T3) NYSERNet

22. Nine Universities Connect through NoX 22 MIT (T2 and T3) Brandeis University (T3) Harvard University (T2) Boston University (T2 and T3) Brown University (T3) Yale University (T3) Northeastern University (T3) Tufts University (T3) U Mass, Amherst (T3) NoX

23. LHC Measurement Requirements 1 1.Monitor up/down status of cross domain circuits A.Publish status via a web services interface B.Provide tools to visualize state C.Generate NOC alarms when circuits change states 2.Monitor Link/Circuit Capacity, Errors & Utilization A.Publish statistics via a web services interface B.Provide tools to visualize the data C.Generate NOC alarms when thresholds are crossed

24. LHC Measurement Requirements 2 3.Continuously measure delay between participants A.Manage multiple sparse meshs of continuous tests and store results in an MA B.Publish results via a standardized web service interface C.Provide a tool to visualize the data D.Provide tools to automatically analyze data and generate NOC alarms 4.Make scheduled bandwidth measurements across paths of interest A.Manage multiple regularly scheduled sparse meshes of tests and store results in an MA B.Publish results via a standardized web service interface C.Provide a tool to visualize the data D.Provide tools to automatically analyze data and generate NOC alarms

25. LHC Measurement Requirements 3 5.Measure & Publish Topology of both primary and backup paths A.Publish statistics via a web services interface B.Provide tools to visualize the data over time

26. Directions Forward Deploy measurement tools –To quantify the service your receiving/delivering Set User Expectations –100 to 300 Mbps per stream Educate your user base –So they know what is possible

27. Questions? Joe Metzger (metzger@es.net)metzger@es.net Eric Boyd (eboyd@internet2.edu)eboyd@internet2.edu