1. Use of Alternate Path Circuits at Fermilab {A Site Perspective of E2E Circuits} Phil DeMar I2/JointTechs Meeting Monday, Feb. 12, 2007

2. Why end-to-end circuits? Convergence of need, capability, & strategic direction  And sometimes just because our stakeholders ask for them… Need:  Emerging high impact data movement requirements  Predictable network performance requirements: Distributed DAQ function Distributed analysis model  Data movement thru CMS Tier structure is flexible, not geographic With significant trans-oceanic traffic  LHC traffic projections call for rapid increase in traffic levels

3. Why end-to-end circuits? - Capability - An optical network infrastructure deployed between FNAL & StarLight  Dark fiber lease procured in summer, 2004  DWDM gear (Ciena Metro) procured & installed in Sept, 2004 Initial configuration: 1x10GE & 2x1GE channels 2 nd 10GE channel added in 2006 Direct fiber connectivity at StarLight opened up a plethora of opportunities for enhanced network connectivity  Broad spectrum of possible peering partners available  L2 technology options (L1 someday?) Optical network infrastructure offered flexible & (relatively…) economic growth options

4. Why end-to-end circuits? - Strategic Direction 2002 DOE High Performance Network Planning Workshop:  Established a strategic model for ESnet to follow: High bandwidth, scalable, reliable production IP network service Very high-bandwidth network for large scale science data flows  (ESnet Science Data Network) MANs for National Labs local access  Enabled us to : Pursue high impact data movement circuits as a strategic direction Feel comfortable about convergence of our local & wide area network directions with our service provider (ESnet)

5. Fermilab end-to-end circuits today Fermilab has been implementing circuit-based services for ~2 yrs  Circuits have been established for all major collaborations at the Lab  Remote sites are scattered across the globe  Technical implementations have evolved on multiple tracks  Circuits have had varying degrees of usefulness: LHCOPN circuit is an essential component of the Lab’s CMS Tier-1 support CMS Tier-2 circuits are very heavily used for experiment rampup activities Opposite extreme:  WestGrid circuit decommissioned last fall as no longer being necessary  Apache Pt circuit discontinued; IP path worked fine for tape storage access

6. Topology of circuit connections Circuits are L2 (vLAN-based) Circuits share physical & network device infrastructure with IP service  One 10GE channel (purple) supports production network (ESnet) traffic  Other 10GE channel (red) supports circuits to most collaboration sites LHCOPN & newest (IN2P3) circuits are early ESnet ChiMan users Complexity of “circuits” is higher than IP service:  Multiple service domains is the norm  Deployed technology within service domains vary

7. Making the E2E circuit routing work Define high impact traffic flows:  Minimal-size source/dest. netblock pairs US-CMS Tier-1 / CERN T0 address pairs to follow LHCOPN E2E circuit path Other FNAL-CERN traffic follows routed path Deploy alternate path border router for E2E`circuits  BGP peer across vLAN-based circuits, advertising only source netblock Implement alternate forwarding:  Outbound: policy route on source/dest pairs:  Inbound: depends on policies of remote end Prefer comparable PBR for symmetry But implement local inbound PBR

8. Usefulness of E2E Circuits Monthly FNAL outbound traffic Recent spikes exclusively due to CMS ramp-up testing  Supports CMS traffic projections  Tests indicate current performance capabilities, not traffic trends SC = service challenge, internally set testing goal of collaboration CSA = computing & software assessment ; externally set standards the project is supposed to meet Relative ratio of circuit-based traffic to routed traffic is also more an indication of performance capability  But US Tier-2s (circuit-based) currently sustaining 2-3 Gb/s  In SC4 European Tier-2s (routed) were sustaining 100Mb/s-900Mb/s

9. Issues with E2E circuits Circuit coordination & establishment can be complex  Varies with # of administrative domains and mix of underlying technology Monitoring certainly becomes more difficult  Varies with # … (II) Troubleshooting problems are more difficult too  Varies with # … (III)  Superficial observation: likely to be needed more frequently as well Failure modes need to be understood and failover scenarios tested Proper documentation can be a lot of work  Or doesn’t get adequately done (guilty…)

10. End-to-End circuit complexity IN2P3 - FNAL Service Trial

11. Monitoring E2E circuits Utilizing perfSONAR for LHCOPN E2E monitoring & others (IN2P3)  At this point, pretty minimal level of monitoring E2E Measurement Point (MP)  physical interfaces monitoring  BGP session monitoring  VLAN state monitoring SQL Measurement Archive (MA) for history  Requires a lot of custom effort Its been difficult to get all the pieces in place  PerfSonar is a work-in-progress analysis & troubleshooting tools needed

12. Failover Testing & Monitoring End-to-end circuit failure modes are different than for IP service links  From our experience, they are more complex  The impact of the failure may be severely felt elsewhere Service Challenge-2 failover (11/04)  High rate (~7Gb/s) testing between Tier-0 & CMS FNAL Tier-1 under way: FNAL routed IP path still an OC12  Trawler cuts trans-Atlantic cable  Production IP OC12 is buried  Test traffic is manually rerouted  Trans-Atlantic circuit finally rerouted to different under-seas cable

13. Static versus dynamic E2E circuits Currently all FNAL E2E circuits are static circuits One problem with dynamic circuits is adapting local routing to use them Lambda Station (LS) project:  Based on PBR mechanisms used for static E2E`circuits  LS is called by apps or wrapper scripts Application awareness is a difficult issue for any differential service…  It schedules reservable network paths  It configures selective forwarding within site to utilize alternate path  Only configures local site infrastructure & ACL protections Coordinates with LS on remote end

14. Future directions: Still a function of need, capability, and strategic direction… ESnet Chicago MAN (right) will provide additional bandwidth capability for FNAL:  When completed (2-3 months): 2 x 10GE routed IP connections 4 x 10GE SDN channels In light of CSA07 tests, US-CMS requested upgrade to 2x10GE for T1/T2 circuit E2E circuit testing with a European CMS Tier-1 site in near future Planning to evaluate impact of mid-path channelized (vcat/lcas) service on 10GE links

15. Winding It Up… End-to-end circuits have proven to be useful at FNAL  At least for LHC/CMS high impact data movement  Not so clear how useful for other experiments & projects There is certainly additional management & support cost involved  Complexity is an obvious concern  Scalability too… I predict suspect that we will see a natural selection process play out  What works & is worth the effort will remain and grow  What doesn’t prove to be worth the effort will disappear Dynamic end-to-end circuits?  My crystal ball is too cloudy on that one…