Copyright AARNet International connectivity & “Big Science” WA September 2005 George McLaughlin AARNet

Copyright AARNet About AARNet AARNet grew from an initiative of the AVCC to build a TCP/IP (internet) network in 1989 In 1995 the AVCC sold the commercial customer base of AARNet to Telstra, spawning the commercial Internet in Australia In 2000 AARNet obtained a carrier licence and IRU’s on international capacity Shareholders are 38 Australian Universities and the CSIRO A not-for-profit company that builds, manages and operates the AARNet3 Network Based on lighting fibre across Australia, from NextGen, power utilities and fibre suppliers 12 circuits from Australia. 8 global points-of-presence

Copyright AARNet Southern Cross Australia Japan Cable SEAMEWE3 APCN

Copyright AARNet Going East & North – Southern Cross Currently 6 circuits –2 x 10Gbps (SXTransPORT, R&E only) –2 X 622Mbps (commodity PAIX and LA) –2 X 155Mbps (mixed, Fiji, Hawaii to Japan) About to add 2 more 622Mbps circuits –For commodity expansion This will also extend rights to use SXTransPORT to end 2013 Participation in the National Science Foundation’s (NSF) International Research Connections Program (IRNC) Strategy defined and largely implemented Where it gets us to: Fiji Hawaii US West Coast and US West Coast to: North America Central America South America Europe Japan (and other Asia Pacific)

Copyright AARNet TransLight Pacific Wave Partners: AARNet, CENIC, Pacific Wave, University of Hawaii Distributed International Peering Exchange along US West Coast Hybrid Optical Packet Infrastructure Seed Global Astronomy Initiative based around the international telescopes at Mauna Kea, Hawaii GLIF infrastructure between US, Hawaii and Australia An initiative of the US National Science Foundation’s International Research Network Connections Program

Copyright AARNet AARNet, Pacific Wave, NLR,.……..

Copyright AARNet Going West and North (via Singapore) This strategy has been developed together with the European Commission’s TEIN2 project It will provide a massive increase in connectivity within the region and importantly provides to geographically diverse routes from the region going west to Europe Will significantly enhance the infrastructure for engagement between AARNet’s members and others in the region Future extensions to South Asia and Africa likely Key element of AARNet global connectivity strategy, soon to be implemented Where it gets us to:| Singapore From Singapore to: Indonesia, Malaysia Thailand, Hong Kong Japan, Korea From Hong Kong to: Vietnam, Beijing (then via Russia to Europe)

Copyright AARNet Trans Eurasian Information Network (TEIN2) Partners An initiative of the European Commission with the objective of improving connectivity in certain developing countries of the Asia Pacific region Beneficiaries: China (CERNET) Indonesia (ITB) Malaysia (MDC) Philippines (ASTI) Thailand (ThaiREN) Vietnam (MOST) Non-beneficiaries: Korea (KISDI) Singapore (SingAREN) Australia (AARNet) France (RENATER) Netherlands (SURFnet) UK (UKERNA)

Copyright AARNet TEIN2 Topology Singapore to Perth (4 x 155Mbps, 2x2) Frankfurt (west) (3 x 622Mbps) Japan (at least 2 x 622Mbps) Korea (622Mbps) Hong Kong (622Mbps) Thailand (155Mbps) Malaysia (45Mbps) Indonesia (45Mbps) Hong Kong to: Vietnam (45Mbps) Beijing (622Mbps) Beijing to Europe via Russia (622Mbps) 2

Copyright AARNet How we currently get to these places

Copyright AARNet AARNet and TEIN2 circuits

Copyright AARNet TEIN2 – AARNet’s role (i) TEIN2, though funded by EuropeAID, falls under the auspices of ASEM (Asia Europe Meeting) Australia isn’t an ASEM member Australian Govt can’t participate in ASEM meetings AARNet acting as proxy for Australia in TEIN2 (firstly as project advisor, then as project partner) Building on AARNet's leading role in APAN

Copyright AARNet TEIN2 – AARNet’s role (ii) AARNet will establish PoPs in Singapore and Frankfurt The 4 x 155Mbps (2 on SEAMEWE3 and 2 on APCN) from Perth to Singapore and one of the 622Mbps circuits to Frankfurt will be under AARNet’s control Up to 50% of the capacity may be used for commodity traffic Application deployment already in planning AARNet will be intimately involved in the engineering task force and NOC implementation Current project runs to Dec 2007

Copyright AARNet TEIN2, Taiwan, AARNet and LHC Taiwan is not a direct partner in TEIN2 Taiwan is the TEIR1 Large Hadron Collider site in the region Taiwan will deploy a 2.5Gbps link going west to Amsterdam via Singapore AARNet will interconnect with this at Singapore Taiwan plans to upgrade this circuit going west to 10Gbps by end 2007

Copyright AARNet Combined Strategy elements 1 plus 2 2 AARNet International Gateways: Sydney Perth 7 AARNet Global PoPs: Seattle Palo Alto Los Angeles Hawaii (Oahu) Hawaii (Big Island) Suva Singapore Frankfurt AARNet owned and operated circuits Circuit sizepath 1x10GbpsSydney - Oahu-Seattle (R&E) 1x10GbpsSydney - Big Island - Los Angeles (R&E) 2x622MbpsSydney - Palo Alto (Commodity) 2x622MbpsSydney - Los Angeles (Commodity) 1x155MbpsSydney - Suva - Oahu - Seattle (both) 1x155MbpsSydney - Seattle (both) 2x155MbpsPerth - Singapore (both) SEAMEWE3 path 2x155MbpsPerth - Singapore (both) APCN path 1x622MbpsSingapore – Frankfurt (both) westerly path

Copyright AARNet Combined Strategy elements 1 plus 2 2 AARNet International Gateways: Sydney Perth 7 AARNet Global PoPs: Seattle Palo Alto Los Angeles Hawaii (Oahu) Hawaii (Big Island) Suva Singapore Frankfurt Other Circuits accessible from Singapore PoP Circuit sizeCity/country/economy 2x622MbpsFrankfurt nx622MbpsJapan 1x622MbpsKorea 1x622MbpsHong Kong 1 x 622MbpsBeijing (via Hong Kong) 1x2.5GbpsTaiwan 1x155MbpsThailand 1x155MbpsPhilippines (via Japan) 1x45MbpsMalaysia 1x45MbpsIndonesia 1 x 45MbpsVietnam (via Hong Kong)

Copyright AARNet Facilitating recent S&T agreements Minister Nelson’s recent trip to Asia and the signing or extension of various agreements –China –Malaysia –Singapore –Indonesia All are both TEIN2 and APAN partners New links will facilitate new collaborations

Copyright AARNet Going North (Australia Japan Cable) We have attractive pricing on this cable system Need to avoid biting off more than we can chew Logistics dictate that TEIN2 and additional commodity are higher priorities As soon as TEIN2 is bedded down, should progress with AJC options Part of strategy yet to be implemented Where it gets us to: Japan (optionally Guam) From Japan to: North Asia South East Asia US From Guam: Cable interconnect opportunities

Copyright AARNet “Big Science” projects driving networks Large Hadron Collider –Coming on-stream in 2007 –Particle collisions generating terabytes/second of “raw” data at a single, central, well-connected site –Need to transfer data to global “tier 1” sites. A tier 1 site must have a 10Gbps path to CERN –Tier 1 sites need to ensure gigabit capacity to the Tier2 sites they serve Square Kilometre Array –Coming on-stream in 2010? –Greater data generator than LHC –Up to 125 sites at remote locations, data need to be brought together for correlation –Can’t determine “noise” prior to correlation –Many logistic issues to be addressed Billion dollar globally funded projects Massive data transfer needs

Copyright AARNet From very small to very big

Copyright AARNet Scientists and Network Engineers coming together HEP community and R&E network community have figured out mechanisms for interaction – probably because HEP is pushing network boundaries eg the ICFA workshops on HEP, Grid and the Global Digital Divide bring together scientists, network engineers and decision makers – and achieve results

Copyright AARNet What’s been achieved so far  A new generation of real-time Grid systems is emerging - support worldwide data analysis by the physics community  Leading role of HEP in developing new systems and paradigms for data intensive science  Transformed view and theoretical understanding of TCP as an efficient, scalable protocol with a wide field of use  Efficient standalone and shared use of 10 Gbps paths of virtually unlimited length; progress towards 100 Gbps networking  Emergence of a new generation of “hybrid” packet- and circuit- switched networks

Copyright AARNet LHC data (simplified) Per experiment 40 million collisions per second After filtering, 100 collisions of interest per second A Megabyte of digitised information for each collision = recording rate of 100 Megabytes/sec 1 billion collisions recorded = 1 Petabyte/year CMSLHCbATLASALICE 1 Megabyte (1MB) A digital photo 1 Gigabyte (1GB) = 1000MB A DVD movie 1 Terabyte (1TB) = 1000GB World annual book production 1 Petabyte (1PB) = 1000TB 10% of the annual production by LHC experiments 1 Exabyte (1EB) = 1000 PB World annual information production

Copyright AARNet LHC Computing Hierarchy Tier 1 Tier2 Center Online System CERN Center PBs of Disk; Tape Robot FNAL Center IN2P3 Center INFN Center RAL Center Institute Workstations ~ MBytes/sec Gbps Tens of Petabytes by An Exabyte ~5-7 Years later. ~PByte/sec ~ Gbps Tier2 Center ~ Gbps Tier 0 +1 Tier 3 Tier 4 Tier2 Center Tier 2 Experiment CERN/Outside Resource Ratio ~1:2 Tier0/(  Tier1)/(  Tier2) ~1:1:1 0.1 to 10 Gbps Physics data cache

Copyright AARNet Lightpaths for Massive data transfers From CANARIE A small number of users with large data transfer needs can use more bandwidth than all other users

Copyright AARNet Why? Cees de Laat classifies network users into 3 broad groups. 1.Lightweight users, browsing, mailing, home use. Who need full Internet routing, one to many; 2.Business applications, multicast, streaming, VPN’s, mostly LAN. Who need VPN services and full Internet routing, several to several + uplink; and 3.Scientific applications, distributed data processing, all sorts of grids. Need for very fat pipes, limited multiple Virtual Organizations, few to few, peer to peer. Type 3 users: High Energy Physics Astronomers, eVLBI, High Definition multimedia over IP Massive data transfers from experiments running 24x7

Copyright AARNet What is the GLIF? Global Lambda Infrastructure Facility - International virtual organization that supports persistent data-intensive scientific research and middleware development Provides ability to create dedicated international point to point Gigabit Ethernet circuits for “fixed term” experiments

Copyright AARNet Huygens Space Probe – a practical example Cassini spacecraft left Earth in October 1997 to travel to Saturn On Christmas Day 2004, the Huygens probe separated from Cassini Started it’s descent through the dense atmosphere of Titan on 14 Jan 2005 Using this technique 17 telescopes in Australia, China, Japan and the US were able to accurately position the probe to within a kilometre (Titan is ~1.5 billion kilometres from Earth) Need to transfer Terabytes of data between Australia and the Netherlands Very Long Baseline Interferometry (VLBI) is a technique where widely separated radio- telescopes observe the same region of the sky simultaneously to generate images of cosmic radio sources

Copyright AARNet AARNet - CSIRO ATNF contribution Created “dedicated” circuit The data from two of the Australian telescopes (Parkes [The Dish] & Mopra) was transferred via light plane to CSIRO Marsfield (Sydney) CeNTIE based fibre from CSIRO Marsfield to AARNet3 GigaPOP SXTransPORT 10G to Seattle “Lightpath” to Joint Institute for VLBI in Europe (JIVE) across CA*net4 and SURFnet optical infrastructure

Copyright AARNet But……….. 9 organisations in 4 countries involved in “making it happen” Required extensive human-human interaction (mainly s…….lots of them) Although a 1Gbps path was available, maximum throughput was around 400Gbps Issues with protocols, stack tuning, disk-to- disk transfer, firewalls, different formats, etc Currently scientists and engineers need to test thoroughly before important experiments, not yet “turn up and use” Ultimate goal is for the control plane issues to be transparent to the end-user who simply presses the “make it happen” icon Although time from concept to undertaking the scientific experiment was only 3 weeks……..

Copyright AARNet International path for Huygens transfer

Copyright AARNet EXPReS and Square Kilometre Array SKA bigger data generator than LHC But in a remote location Australia one of countries bidding for SKA – significant infrastructure challenges Also, Eu Commision funded EXPReS project to link 16 radio telescopes around the world at gigabit speeds

Copyright AARNet In Conclusion scientists and network engineers working together can exploit the new opportunities that high capacity networking opens up for “big science” Need to solve issues associated with scalability, control plane, ease of use QUESTIONS?