1. ESnet Network Measurements ESCC Feb 15 2007 Joe Metzger metzger@es.net

2. Measurement Motivations
Users dependence on the network is increasing
Distributed Applications
Moving Larger Data Sets
The network is becoming a critical part of large science experiments
The network is growing more complex
6 core devices in 05’, in 08’
6 core links in 05’, in 08’, 80+ by 2010?
Users continue to report performance problems
‘wizards gap’ issues
The community needs to better understand the network
We need to be able to demonstrate that the network is good.
We need to be able to detect and fix subtle network problems.

3. perfSONAR
perfSONAR is a global collaboration to design, implement and deploy a network measurement framework.
Web Services based Framework
Measurement Archives (MA)
Measurement Points (MP)
Lookup Service (LS)
Topology Service (TS)
Authentication Service (AS)
Some of the currently Deployed Services
Utilization MA
Circuit Status MA & MP
Latency MA & MP
Bandwidth MA & MP
Looking Glass MP
Topology MA
This is an Active Collaboration
The basic framework is complete
Protocols are being documented
New Services are being developed and deployed.

4. perfSONAR Collaborators
ARNES
Belnet
CARnet
CESnet
Dante
University of Delaware
DFN
ESnet
FCCN
FNAL
GARR
GEANT2
* Plus others who are contributing, but haven’t added their names to the list on the WIKI.
Georga Tech
GRNET
Internet2
IST
POZNAN Supercomputing Center
Red IRIS
Renater
RNP
SLAC
SURFnet
SWITCH
Uninett

5. perfSONAR Deployments
16+ different networks have deployed at least 1 perfSONAR service (Jan 2007)

6. ESnet perfSONAR Progress
ESnet Deployed Services
Link Utilization Measurement Archive
Virtual Circuit Status
In Development
Active Latency and Bandwidth Tests
Topology Service
Additional Visualization capabilities
perfSONAR visualization tools showing ESnet data
Link Utilization
perfSONARUI
VisualPerfSONAR
Traceroute Visualizer
E2EMon (for LHCOPN Circuits)

7. LHCOPN E2Emon E2ECU LHCOPN Monitoring
An Optical Private Network connecting LHC Teir1 centers around the world to CERN.
The circuits to two of the largest Tier1 centers, FERMI & BNL cross ESnet
E2Emon
An application developed by DFN for monitoring circuits using perfSONAR protocols
E2ECU
End to End Coordination Unit that uses E2Emon to monitor LHCOPN Circuits
Run by the GEANT2 NOC

8. Central Monitoring Software
E2Emon Components
Central Monitoring Software
Uses perfSONAR protocols to retrieve current circuit status every minute or so from MAs and MPs in all the different domains supporting the circuits.
Provides a web site showing current end-to-end circuit status
Generates SNMP traps that can be sent to other management systems when circuits go down
MA & MP Software
Manages the perfSONAR communications with the central monitoring software
Requires an XML file describing current circuit status as input.
Domain Specific Component
Generates the XML input file for the MA or MP
Multiple development efforts in progress, but no universal solutions
CERN developed one that interfaces to their abstraction of the Spectrum NMS DB
DANTE developed one that interfaces with the Acatel NMS
ESnet developed one that uses SNMP to directly poll router interfaces
FERMI developed one that uses SNMP to directly poll router interfaces
Others under development

9. E2Emon Central Monitoring Software

10. ESnet4 Hub Measurement Hardware
Latency
1U Server with one of:
EndRun Praecis CT CDMA Clock
Meinberg TCR167PCI IRIG Clock
Symmetricom bc637PCI-U IRIG Clock
Bandwidth
4U dual Opteron server with one of:
Myricom 10GE NIC
9.9 Gbps UDP streams
~6 Gbps TCP streams
Consumes 100% of 1 CPU
Chelsio S320 10GE NIC
Should do 10G TCP & UDP with low CPU Utilization
Has interesting shaping possibilities
Still under testing…

11. Network Measurements ESnet is Collecting
SNMP Interface Utilization
Collected every minute
For MRTG & Monthly Reporting
Circuit Availability
Currently based on SNMP Interface up/down status
Limited to LHCOPN and Service Trial circuits for now
NetFlow Data
Sampled on our boundaries
Latency
OWAMP

12. ESnet Performance Center
Web Interface to run Network Measurements
Available to ESnet sites
Supported Tests
Ping
Traceroute
IPERF
Pathload, Pathrate, Pipechar
(Only on GE systems)
Test Hardware
GE testers in Qwest hubs
TCP iperf tests max at ~600 Mbps.
10GE testers are being deployed in ESnet4 hubs
Deployed in locations where we have Cisco GE Interfaces
Available via Performance Center when not being used for other tests
TCP iperf tests max at 6 Gbps.

13. ESnet Measurement Summary
Standards / Collaborations
PerfSONAR
LHCOPN
Circuit Status Monitoring
Monitoring Hardware in ESnet 4 Hubs
Bandwidth
Latency
Measurements
SNMP Interface Counters
Circuit Availability
Flow Data
One Way Delay
Achievable Bandwidth
Visualizations
PerfSONARUI
VisualPerfSONAR
NetInfo