US LHCNet Update Dan Nae California Institute of Technology LHC OPN Meeting Munich, April 2007

LHC OPN Meeting, Munich What is the US LHCNet u A transatlantic network designed to support the LHC and the U.S. HEP community u Funded by the US DoE and CERN and managed by Caltech in collaboration with CERN u Evolved from a network between US and CERN which dates back to 1985 u Our mission is to deliver a reliable network service to support the upcoming LHC experiments at CERN u Designed to support the LHC three-tiered model and to deliver data directly to/from the US Tier 1’s (FNAL and BNL)

LHC OPN Meeting, Munich Atlantic Ocean Circuit Status NYC 111 8th Pottington (UK) VSNL South NY 60 Hudson Highbridge VSNL North AMS-SARA AC-2 Bude GVA-CERN FrankfurtVSNL Wal, NJ London Global Crossing Qwest Colt GEANT NYC-MANLAN CHI-Starlight Paris Bellport Whitesands  Unprotected circuits (lower cost)  Service availability from provider’s offers:  Colt Target Service Availability is 99.5%  Global Crossing guarantees Wave Availability at 98%  Canarie and GEANT: No Service Level Agreement (SLA) LCG Availability requirement: 99.95%

LHC OPN Meeting, Munich Circuit Status (cont.) - 6 x 10 Gbps SONET/10GbE WAN-PHY. - Amsterdam PoP installed in January. - Last circuit (Geneva-Amsterdam) delivered at the end of last month. - Not as diverse as one might expect. - We’ve never had an isolated node so far. - It happened more than once that the circuits went down in pairs (GC NYC-CHI and NYC-AMS, Qwest NYC-CHI and CHI- GVA). - Rumors of a new RFP in May-June?

LHC OPN Meeting, Munich Circuit Availability (March) Proposal to produce a monthly report

LHC OPN Meeting, Munich “Old” Network Map (March ’07)

LHC OPN Meeting, Munich Equipment Status - Next generation network based on Ciena CoreDirector/CI – VCAT/LCAS/GFP capabilities. - Four nodes, one at each PoP to complement the existing Force10 boxes. - Number of links on each box expected to increase each year; the upgrade plan has been discussed with Ciena up to ~ Deployment has already started (the New York and Chicago nodes were installed, tested and commissioned by the end of last month). - The Geneva node is physically installed, will be tested and most likely accepted next week. - The fourth node (Amsterdam) expected to be operational in May.

LHC OPN Meeting, Munich Past and Present Issues - At least one Stratum 1 PRS is recommended for normal operation (still to be installed) - Second DC power feed in Chicago is still to be installed (today in non-redundant mode) - One ESLM card (10GbE) locked up during the migration of the the NYC-CHI GC circuit (fixed by hardware reset) - DC rectifier at CERN not fully populated (but enough for the moment) - CERN commissioning has been delayed due to complicated site access procedures - We need more operational experience with the boxes - Currently only one circuit (GC NYC-CHI) runs over the new equipment (migrated on March 29 th )

LHC OPN Meeting, Munich Equipment Status Today - Three nodes delivered (GVA, CHI, NYC) - Two nodes installed (NYC, CHI) - One circuit migrated (GC NYC-CHI)

LHC OPN Meeting, Munich Proposed Migration (May) Two complementing networks: - first network is essentially last year’s US LHCNet – stable, reliable, dedicated 10Gbps T0-T1 (and T1-T1) - the second is based on the new equipment – dynamic or static point-to- point VCs between any of the four nodes - the “new network” can be reconfigured to carry T0-T1 traffic in case of problems Equipment diversity (more robust)

LHC OPN Meeting, Munich GEANT-ESNET Peering over US LHCNet u The idea came up last year during the meeting at FNAL u US LHCNet would transparently carry a L1/L2 Virtual Circuit between NYC and Amsterdam u GEANT and ESNET peer over this VC u The bandwidth of the circuit is still to be discussed u An MOU between ESNET and US LHCNet was signed (with DOE’s blessing) u ESNET will install a new 10 GbE connection to the US LHCNet CD/CI in NYC for this purpose u A new 10 GbE connection in Amsterdam between US LHCNet and GEANT is required

LHC OPN Meeting, Munich Proposed Migration (cont.) Additional links are needed to make full use of the new infrastructure: -New BNL/ESNET link in NYC - New FNAL/ESNET link in Chicago (or existing to be moved to the CD/CI) - New LCG – CD/CI link(s) in Geneva - New GEANT & SARA links in Amsterdam (T1-T1 and peerings)

LHC OPN Meeting, Munich Network Forecast Geneva Chicago New York 4 x 10GBE 1 x 10G SDH 2 x 10G SDH 1 x 10G SDH ESnet SDN Amsterdam 1 x 10G SDH 2 x 10GBE 1 x 10G SDH Ports at Each PoP 10 GbE OC-192 /STM-64 AMS22 GVA43 NYC45 CHI44 Mostly sunny, scattered clouds

LHC OPN Meeting, Munich Network Forecast Geneva Chicago New York 4 x 10GBE 6 x 10GBE 2 x 10G SDH 1 x 10G SDH ESnet SDN 1 x 10G SDH 2 x 10G SDH Amsterdam 1 x 10G SDH 2 x 10GBE 2 x 10G SDH 1 x 10G SDH I2 NewNet Ports at Each PoP 10 GbE OC-192 /STM-64 AMS23 GVA65 NYC47 CHI45 Clouds gathering…

LHC OPN Meeting, Munich Network Forecast Geneva Chicago New York 4 x 10GBE 8 x 10GBE 3 x 10G SDH 2 x 10G SDH ESnet SDN 1 x 10G SDH 2 x 10G SDH Amsterdam 2 x 10G SDH 2 x 10GBE 2 x 10G SDH 1 x 10G SDH I2 NewNet Hybrid Network Ports at Each PoP 10 GbE OC-192 /STM-64 AMS25 GVA87 NYC412 CHI410 Overcast ;-)

LHC OPN Meeting, Munich Network Forecast Geneva Chicago 4 x 10GBE 10 x 10GBE 4 x 10G SDH 2 x 10G SDH ESnet SDN 1 x 10G SDH 3 x 10G SDH New York 4 x 10G SDH Amsterdam 3 x 10G SDH 2 x 10GBE 3 x 10G SDH 1x 10G SDH 3 x 10G SDH I2 NewNet Hybrid Network Ports at Each PoP 10 GbE OC-192 /STM-64 AMS27 GVA109 NYC416 CHI412 Severe chances of high-speed data transfers

LHC OPN Meeting, Munich Ciena CD/CI - VCAT/LCAS/GFP support – allows for provisioning of dynamic circuits - Difficult to manage with current network tools (no SNMP for example) - Integrated control plane functionality (still to be explored) - Need to be integrated with our provisioning/monitoring tools (MonALISA) - Also selected by Internet2 for their NewNet installation (dynamic provisioning with DRAGON software) - Inter-domain integration for dynamic circuits is desirable in the future

LHC OPN Meeting, Munich Ciena CD/CI (cont.) - Bandwidth granularity of STS-1 (51Mbps) - Bandwidth adjustments without interrupting the circuit (very important for long-lived transatlantic data transfers) - OSRP + Mesh restoration allows for easy provisioning and re-routing of circuits over alternate paths - Fast circuit restoration (Ciena claims 50ms) - Associated circuit priorities (lower priority circuits can be preempted during restoration) - Need to be integrated with perfSONAR monitoring (help needed)

LHC OPN Meeting, Munich Other Technical Issues u The CD/CI management/monitoring is done only via TL1 or GUI (NodeManager) u Circuit status is not reflected in the terminating 10 GbE interface status (workaround – UDLD, FAFD) u We need to understand how to interact with the CD/CI control plane u We need an API to work with for automated operations; apparently there is a CORBA interface but we don’t have any specifications for it yet.

LHC OPN Meeting, Munich US LHCNet Network Monitoring Various monitoring tools are deployed - Spectrum – used by the CERN operators (first line of support) - MonALISA – used by the US LHCNet team to get a global view of all the services - Various open-source toos (Nagios, Cricket, rancid, RRD) - used by the US LHCNet NOC for configuration tracking, logs and alarms - perfSONAR – used by the E2ECU Current perfSONAR installation extracts data from the Spectrum monitoring service at CERN; we have plans to migrate this to a stand-alone setup (at least for the CD/CIs)

LHC OPN Meeting, Munich US LHCNet Team - Harvey Newman – Project PI - Dan Nae – Network Engineer – tech team lead - Ramiro Voicu – Software Engineer - Two people left – Sylvain Ravot (October, 2006) and Yang Xia (March, 2007) - Artur Barczyk - new Network Engineer, started on March 1 st, 2007, based at CERN - Tony Cheng – new Network Engineer, started April 15 th, based in Pasadena, CA

LHC OPN Meeting, Munich Questions? Thank You!