The extension of optical networks into the campus Wade Hong Office of the Dean of Science Carleton University

Outline Motivation CA*net 4 IGT From the Carleton U Perspective Issues Lessons learned

Motivation Large scale distributed scientific experiments (LHC - ATLAS, SNOLab, Polaris, NEES Grid... ) Access to regional distributed HPC resources (HPCVL, SharcNet, WestGrid, TRIUMF Tier 1.5,...) Federating growing research-based computing resources on campus Allowing the end users to access these resources in an unencumbered way CA*net 4 customer empowered networking last mile

CA*net 4 IGT CANARIE funded directed research project build a testbed to experiment with customer empowered networking, pt2pt optical networks, network performance, long haul 10 GbE, UCLP, last mile issues, etc. participants from the HEP community from across Canada, the provincial ORANs, CERN, StarLight, SURFnet, and potentially others setup end to end GbE and 10 GbE lightpaths between institutions in Canada and CERN

CA*net 4 Network

CA*net 4 IGT Sites

CA*net 4 IGT interoperability testing with 10 GbE WAN PHY and OC-192 used IXIA traffic generators to characterize the trams-atlantic link transferred real experimental data from ATLAS FCAL beam tests (GbE and 10 GbE) demonstrated native end-to-end 10 GbE between CERN and Ottawa for the ITU Telecom World 2003

Planned CA*net 4 IGT Activities complete the last mile connectivity for most of the participating Canadian sites third OC-192 across Canada being brought up using Nortel OME 6500s continuing long haul native 10 GbE experiments (Foundry MG8s) TRIUMF to CERN, TRIUMF to Carleton, Carleton to CERN CERN to Tokyo via Canada HEPix Robust Transfer Challenge - sustained disk to disk transfers between TRIUMF and CERN

Planned CA*net 4 IGT Activities Real-time remote farms for ATLAS CERN to U of Alberta Data transfer of End Cap Calorimeter data from the combined beam tests to several Canadian sites one beam test just completed (~1TB) second test to start late August (significantly more data) Transfer of CDF MC data from the Big Mac Cluster establish a GbE lightpath between UofT and FermiLab

Planned CA*net 4 IGT Activities Experimentation with bulk data transfer investigating RDMA/IP (sourcing NICs) establish GbE lightpaths between Canadian sites

Carleton University located in Ottawa, the nation’s capital at the southern end of the world’s longest outdoor skating rink Canada’s Capital University student population of 22,000 students, 1700 faculty and staff over $100M in research funding in the past year CFI contribution significant about half to Physics Bill St. Arnaud’s alma mater

Carleton University

External Network Connectivity commodity Internet Telecom Ottawa - was the largest metro 10 GbE deployment R&E traffic finally connected to ORION (Dec 2003), the new ORAN, just prior to the decommissioning of ONET EduNet non profit, OCRI managed dial-up and High Speed Internet for higher education institutions in Ottawa dial-up ISP has a dedicated link back to campus

Carleton U Network Upgrade campus has been in the process of planning a campus network upgrade for the past 3 to 4 years several false starts application to funding agencies based on requirements of research activities may have missed the window of opportunity finally proceeding with the network upgrade RFPs currently being evaluated

Network Upgrade Proposal original proposal phase one (Year 1) build the campus core network phase two (Year 2) build the distribution layer phase three (Year 3) rewire the buildings for access not my preferred ordering!

Proposed Topology

Differing Viewpoints debate over how to handle high capacity research traffic flows necessity of routing traffic through the proposed high capacity campus core on the other hand optical bypasses would simplify and reduce the complexity and cost of the campus network 4 fibre pairs between Herzberg Laboratories and Robertson Hall cost about $4K CDN - we prevailed reality check current campus network cannot handle the high volume and high speed flows

Motivations Revisited Large scale distributed scientific experiments

Motivations Revisited Access to regional distributed HPC resources other HPCVL sites (Queens, UofO, RMC, Ryerson U) TRIUMF ATLAS Canada computing centre SNOLab shared ORION and CA*net 4 connectivity is only at GbE high capacity flows probably dictate pt2pt optical bypass interconnectivity can be static or dynamic fully statically meshed or scheduled dynamic connectivity on demand - probably the latter

Motivations Revisited Federating growing research-based computing resources into a campus grid HPCVL Linux cluster upgrade ( CPUs) Physics research cluster upgrade ( CPUs) Civil Engineering (~128 CPUs) Architecture/Psychology visualization cluster (>128 CPUs) Systems and Computer Engineering ( 64 CPUs) debating a condominium or distributed model most likely a hybrid with optical fibre as the interconnecting fabric probably static pt2pt optical bypass for ease of use and user control

Motivations Revisited federated the Physics research computing cluster with part of the HPCVL Linux cluster last summer for about 2 months clusters located on different floors pt2pt link established - much easier than routing through the campus network completed half of the MC regeneration for the third SNO paper similar arrangement this summer to add part of the HPCVL cluster to the Carleton U Physics contribution to the LHC Computing Grid till the end of the year

Issues control central management and control vs end user empowerment disruptive network complexity using pt2pt ethernet links for high capacity flows should simplify campus networks (reduce costs?) security disruptive - bypassing DMZ for the uses considered here, the pt2pt links are inherently secure - non routed private subnets

Issues why not copper? it could be but with fibre greater distances requires less active devices along the path management and control - device at each end under the control of the end users is ideal consistent device characteristics - jumbo frames, port speed, duplex, etc. inter-building connectivity is fibre and planned vertical cabling will be fibre

Issues last mile connectivity demarcation point end user device (NIC) or an edge device (switch, CWDM mux) location of the demarc at the end user or a common shared location technology used to extend the end to end lightpath into the campus pt2pt GbE optical GbE NIC - patched thru to GbE interface on ONS media converter - copper to optical

Issues pt2pt 10GbE LAN PHY to WAN PHY conversion to OC192c on ONS 15454/OME 6500 wavelength conversion CWDM media converters - copper to colored wavelength colored GBICS for GbE switch optical link charateristics padding (attenuation), proper power budget, etc. end user shouldn’t need to be an optical networking expert

Lessons Learned good to be rich in fibre provides greater flexibility support of ORANs, national R&E network, and international partners is essential - all have been very supportive need to convince local campus networking folks that this is not really too disruptive will simplify and not burden the campus production network need a more coherent way of dealing with optical access in the last mile still lots to learn!

Thank You! Wade Hong