1. Janet Update GridPP Collaboration meeting Sept 2012 David Salmon

2. Topics Janet6 Janet & e-Infrastructure Issues & futures....

3. Janet6 Neil Shewry Rob Evans

4. SuperJANET4/5 Getting us out of the cycle of networks that were bursting at the seams at the end of their life Dedicated fibre backbone, allowed us to grow. – Subject to contract SJ5 was originally due to finish October 2011 – Extended to October 2013 – Still no congestion – Originally 10Gbit/s, but have upgraded through 40Gbit/s to 100Gbit/s

5. SuperJANET5 Greater visibility of the optical layer than previous networks – Managed by Verizon Business – Janet NOC had read-only access to (most) transmission equipment – Regular meetings with Verizon on implications of capacity requests – Good communications between Janet’s senior engineers and Verizon’s. – 40G and 100G trials before deployment However, still some distance away from the ‘laser-face’ – Deployment issues that were contractual rather than technical or operational

6. Janet6: Building up ideas Thinking & reflection as SJ5 progressed – Greater visibility of optical layer – New features of transmission systems ROADMs: Colourless; Directionless; Contentionless (more later) Direct experience of transmission equipment – TVN Developments in Janet and other R&E networks – Use of point-to-point circuits – Dynamic provisioning

7. Requirements Gathering

8. Bandwidth, bandwidth, bandwidth... Flexibility Agility Cost control Service delivery Delivery of third party services Cloud services Reliability and resilience Partnerships Public/public and public/private Information assurance Off-net Anytime, anywhere access Internationalisation of education Management of costs Funding environment Costs of change

9. Requirements Gathering Functionality Responsiveness to additional network service requirements Flexibility Protection of interests of teaching and learning and research sectors Separability Ability to increase bandwidth at controllable cost Scalability High reliability by minimising single points of failure; flexibly coping with major breaks; assisting in resilience to customer organisations Reliability With agility End-to end Across JANET Support for the provision of wide range of services: e.g. High information assurance; Unified comms

10. Procurement Strategy

11. Options appraisal Bespoke arrangement Procure dark fibre infrastructure Procure optical transmission equipment Management in-house by the Janet NOC competitive dialogue procedure

12. Procurement Strategy Fibre Procurement Lot 1: Fibre Infrastructure in the UK Options: Scotland, Wales, Ireland, Aurora, London, Research Lot 2: Fibre Infrastructure in Ireland Framework Agreement(s) for 4 years Initial order from 5+2 years to 10+5 years

13. Procurement Strategy Transmission equipment procurement Supply, design, delivery, installation, commissioning, maintenance, training, specialist technical support Framework Agreement for 7 years Initial order 3 + 2 + 2 years

14. Project Timetable

16. Procurement Update:

17. Fibre Procurement Prequalification phase Launched October 2011 13 responses Shortlist 6 bidders

18. Fibre Procurement UK Government approval to proceed – Cabinet Office July 2012 Contract Signed July 2012

19. Transmission Equipment Procurement Prequalification phase Launched November 2011 17 responses Shortlist 6 bidders

20. Transmission Equipment Procurement Contract signed last week September 2012

21. Current Status

23. From requirements to services Capacity – Exponential growth continues ‘Commodity’ IP networking R&E requirements – Must be able to scale the network – Not just more channels, but faster channels over the network’s life 100Gbit/s to start 400Gbit/s? 1Tbit/s?

24. From requirements to services Ubiquitous connectivity – Access to Janet from anywhere – Not all in the scope of Janet6 Other activities in the company – Feeds the capacity requirements – Feeds the external connection requirements – Feeds the reliability requirements

25. From requirements to services Distributed campuses – Inter-site, intra-organisation connections – VPNs Separacy Centralised firewall

26. From requirements to services Outsourced services – Compute, data storage, applications – Low latency for interactive services – High bandwidth for bulk data services – Reliability, reliability, reliability.

27. From requirements to services Research – National and global collaboration Interoperation with GEANT and other R&E networks worldwide LHC, SKA, ITER, Bioinformatics, Climate… – ‘Data deluge’ – Services Dynamic provisioning of layer 2 circuits

28. Similar topology to SJ5. Two north-south paths New east-west path from Birmingham to Nottingham – Erdington to Lowdham No backbone PoP at Reading – Still have a regional PoP – STFC RAL as a backbone PoP New PoP in Acton, West London Janet6: Core Fibre and Backbone Resilience

29. Illustrative only – The details are subject to change Similar to SJ5 – ‘Collector arcs’ – Spurs Regional networks connect to two different routers Over 70 optical nodes – Amplifier sites, add/drop sites, and router sites Janet6: Regional fibre

30. Lighting the fibre DWDM transmission equipment Still some way to go in the procurement Theoretically 80 channels of 100Gbit/s per channel Roadmap for 400Gbit/s and 1Tbit/s with better spectral density ROADMs ‘Native’ capacity on the core to be 100Gbit/s – 10Gbit/s multiplexed over that where needed Muxponders or OTN switching – Coherent technology required for 100Gbit/s doesn’t sit well with dispersion compensation required for 10Gbit/s – RNEP fibre may still carry 10G native

31. 10G Circuits: muxponders or switched? Muxponders Switching

32. ROADMs Will be more reconfigurable than SJ5 – One of the collector arcs in Scotland required 13 engineers to be in dotted along it at the same time to commission Get away from the ‘air gap’ between RNEP fibre and backbone fibre that exists on SJ5. – Different transmission systems on collector arcs and backbone – Requires two client cards and a patch cable to bridge a circuit from one to the other

33. Building the network and the NOC What will this give us? – Janet NOC will have a view from the fibre up the stack to the routers – We will see when fibres/lasers/receivers degrade Lower signal levels Increase in error rate – Forward Error Correction (FEC) working harder – We will see where faults are – Fewer administrative boundaries to cross – Directly translate the community’s requirements into engineering

34. IP service As you were Finish moving away from SDH access towards ethernet – 802.1ad bonding for 20Gbit/s, 30Gbit/s etc. – 100GE if you can afford the optics (have seen a dramatic fall) Still IPv4, IPv6, unicast, multicast In London, retaining Telehouse and Telecity Harbour Exchange as major points of external connection Continuing to expand Telecity Manchester – Content providers and content delivery networks – ‘IX Manchester’ operated by LINX

35. VPNs VPNs over primary access – Inter-site connectivity – Low bandwidth layer 2 circuits – Other services? – Statically configured VLANs (site/RN boundary, perhaps RN/core) – BGP signalled VPNs (RN/core, perhaps some sites) – Bandwidth management between IP and VPNs These are layer 2/3 VPNs – Not IPsec – If you want to be sure it is encrypted, best to do that yourself Would you trust this man with your encryption keys?

36. Lightpath service Good question Ethernet over MPLS network will remain – Upgrade the MX960s to 100GE interconnects – Move 10Gbit/s circuits from dedicated wavelengths to EoMPLS Investigating OTN switching as part of transmission procurement – May be too early to do much – …especially as we’ll also have MX960 platform Don’t need guaranteed bandwidth? – See VPN service

37. Migration 66.666666666666667% (ish) chance we’ll be moving core PoPs Moving routers from PoP to PoP overnight would be ‘challenging’ – Requires all regional networks connected to a PoP to move simultaneously (no link between old PoP & new PoP) – Can’t move anything until all the regional network fibres to a PoP are ready One delay holds up everything Remember the newts! Need to upgrade the switching fabric anyway – T640 to T1600 – T1600 to T4000 Investigating options & costs

38. Migration Aiming to have the network installed and ready by the end of March 2013 Migration between March and August Post-Clearing until end of October is contingency only End of October is a hard cut-off.

39. Janet6 summary Extensive fibre footprint Direct management of optical transmission On-schedule Very capable strategic foundation – 10 years+

40. e-Infrastructure Networking David Salmon

41. Topics e-Infrastructure funding – What has Janet been doing ? Emerging Issues

42. Janet – e-Infrastructure networking £26MWith HEFCE Breakdown £10MContribution to Janet6 backbone £12M Network provision within “classic” Janet community, including key locations which may need additional fibre access £4M “Industry connection”

43. e-Infrastructure programme support CTO Bob Day on ELC Paul Lewis secondment to BIS programme Office – 8 months STFC Michael Wilson secondment – just started – funded from Janet’s e-Infrastructure allocation

44. e-Infrastructure Strategic Sites Potential fibre / capacity requirements Work with organisations to understand requirements Build cases for provision If approved, integrate provision with deployment of Janet6

45. Patterns of use Data movement – working patterns – How much – data volume ? Beware Bytes vs. bits ! – How quickly ? – How often ? – Where to ? – National & International Network engineers need bit-rates ! Working with research communities to understand emerging/evolving requirements 45

46. Norwich Bioinformatics Cluster The Genome Analysis Centre - TGAC The Sainsbury Laboratory John Innes Centre Institute for Food Research University of East Anglia Norwich Research Park

47. Hinxton Genome Campus European Molecular Biology Laboratory - European Bioinformatics Institute The Sanger Centre Babraham Institute

48. Exeter & locale Met Office University of Exeter Weather & Climate research Monsoon HPC – NERC & M.O.

49. Broaden scope of Janet’s fibre infrastructure Complementary & integrated with Janet6   

50. e-Infrastructure projects funded HPC – 5 regional consortia – National facility – Hector / Archer – Dirac Particle physics NERC STFC – RAL – Daresbury will engage to discuss requirements & evolution

51. Outcome Outline proposals made Approvals discussions with Research Councils Must converge soon to optimise Janet6 fibre layout Some budget remains Likely to be a further proposals Academic & Industry locations ?

52. Some implications 52

53. Throughput challenges Infrastructures can deliver very high capacities Realising high-throughput end-to-end will take some effort Can’t just take systems “out of the box” and expect to get full line-rate data-transfers This needs to be communicated & understood more widely Janet & NREN community needs to work more closely with research communities to understand and solve issues. 53

54. Achievable Throughput What is realistic ? – Challenge: 1Gbit/s, 24hr ~= 10TBytes Holistic approach – End systems - Hardware & Software - Both ends ! – Local networks – Wide area networks Highlight successful examples – Technical case studies Janet role ? – Co-ordination/collaboration – Advice & troubleshooting support ?

55. Network structure implications ? Large research communities International/global overlays or private networks Particle physics - LHC OPN (LHCONE) Radio Astronomy – eVLBI, LOFAR, (SKA...) Future – Bioinformatics ?? ELIXIR Campus implications ? point to point services (lightpaths) more complex multi-homing – routing/switching issues

56. Industry ? Existing Janet policy – Supporting collaborative R&D – Business and Community Engagement (BCE) Evolution - more needed ? e-Infrastructure – “Open and accessible Janet”

57. Connecting Industry – meaning ? Network connections ? Access to community systems & resources connected to Janet ? Access to Janet community knowledge & expertise ? – Mix of all three... BIS & HPC Consortia views ?

58. Issues & futures Good planning & dialogue between Janet & GridPP – continue ! – understand evolving requirements – address these as they emerge Involved in LHCOPN & LHCONE meetings – Rob Evans (Janet) – Technical input – Balancing UK requirements, GEANT & other NRENs in Europe, and US & global NRENs

59. GidPP networking archaeology....(2000)

60. Conclusion Janet6 on-track – Very capable strategic platform supporting UK Research and Education for 10 years+ e-Infrastructure – Approvals soon hopefully – Challenge of delivery & deployment – Industry ? Using the network – Achieving higher data throughput

61. ?

62. Janet Aurora

63. Last presented at CEF in 2010 Very good results produced – High profile papers at conferences But, funding issues have held back full development

64. Context - Funding History e-Science and JISC Funding – £4.35M – 2007-2012 Fibre infrastructure only – Fibre access (single pair) – ILA rental – space for researchers equipment – 1 rack – basic telecomms for management access – Tax – lit fibres are subject to business rates in UK NOT – Research equipment – Staff effort

65. JANET Aurora Intermediate equipment co-location point University / JANET access point Fibre spans International Lightpath access to other NRENs via JANET & GEANT JANET Lightpath access to other UK Locations University of Cambridge University of Essex UCL University of Southampton Telehouse London 550km of fibre-pairs VM Brentford PoP VM Camberley PoP VM Crawley Court PoP VM Chelmsford PoP VM Enfield PoP VM Ipswich PoP VM Bury St Edmonds PoP 31km 52km 58km 43km CAL483331-1 (28km) 30km 55km 57km 54km 50km 72km

66. Current position Janet will maintain Aurora in current form until Sept/Oct 2013 Aurora / Dark Fibre facility WILL need to evolve over next 12-18 months – Can not extend contracts for current infrastructure any further (procurement law) – The Essex research group is moving to Bristol (Prof. Dimitra Simeonidou) Research Council (EPSRC) funding for a “Dark-Fibre Facility” is still being considered – Equipment and staff effort to exploit dark-fibre as a test platform

67. e-Infrastructure New funding form UK government £100M+ into HPC – National – Regional £26M to Janet to continue and extend capability – Extend Janet6 to key locations – Biosciences – Climate and Weather Research – HPC Dark-Fibre facility / Aurora MAY also be possible – Exploring with Research Councils

68. Recent Results ECOC – next week – Dimitra Simeonidou & colleagues will present interesting new results Collaborations with NPL continue – Ultra-stable lasers – NPL and UCL – Optical comb transfer – NPL and Southampton Janet in discussion with NPL about participation in Aurora NPL has independent funding, and plans to deploy fibre for comb- transfers Following slides from Radan Slavik – UFE, Prague – ORC, Southampton

69. JANET Aurora Dark Fiber Link Intermediate equipment co-location point University / JANET access point Fibre spans International Lightpath access to other NRENs via JANET & GEANT JANET Lightpath access to other UK Locations University of Cambridge University of Essex Aston University UCL University of Southampton Telehouse London 850km of fibre-pairs University of Southampton Telehouse London 40 km 60 km 40 km Southampton to London part of the link: 200 km, dispersion-compensated, 6 double in-line flat-gain CLA EDFAs with maximum input/output powers of -5/15 dBm. Experiments: (A) Southampton-London roundtrip for phase regeneration; (B) Southampton-Essex roundtrip for optical wavelength conversion.

70. Phase and amplitude all-optical regenerator All-optical wavelength conversion Taken from University of Southampton, U.K. (V. Rancano; results under progress) CLA amplifiers used Taken from University of Southampton, U.K. (R. Slavík et al, Nature Photonics 2010 and JLT 2011)