1 NZNOG 2007 – Inspire.Net / Massey University, Palmerston North, February 1st, 2007 Design and Implementation of the REANNZ KAREN network Jörg Micheel

2 NZNOG 2007 – Inspire.Net / Massey University, Palmerston North, February 1st, 2007 Outline for this talk Network design goals for KAREN Layer network architecture Network services and implementation International transit network design Network performance Checklist for KAREN connectors (REANNZ members) Summary and references

3 NZNOG 2007 – Inspire.Net / Massey University, Palmerston North, February 1st, 2007 MoRST/REANNZ/KAREN design goals A high performance network for the NZ R&E community! 10 Gbps capable backbone interconnecting all major cities in NZ Access speeds at 1 Gbps and 10 Gbps Unconstrained end-to-end performance at (multi- )Gigabits/second Tailored on-demand performance for specific applications or experiments (bandwidth, delay, jitter) International connectivity at 155 Mbps to AU, 622 Mbps to US New services: multicast, IPv6, Jumbo frames (9000 Bytes MTU) Virtual Private Network functionality for members Telco-grade implementation and network management Security, redundancy, high availability Range of network measurement facilities (wire tap, NetFlow, SNMP data collection, active measurement) and development environment Most importantly: stick to budget and timelines!!!

4 NZNOG 2007 – Inspire.Net / Massey University, Palmerston North, February 1st, 2007 L1/L2/L3 Network Design L1 Network core as rings on TCL OPTera DWDM L1 Dark fiber spur to neutral POP and AAP L2 Nationwide network based on Extreme X450a and BD10K L3 using Juniper M320 in Auckland and Wellington Note: 10GigE WANPHY is Gbps! ANOPS management network based on TCL PIP service and CISCO 2801

5 NZNOG 2007 – Inspire.Net / Massey University, Palmerston North, February 1st, 2007 Extreme Networks Black Diamond – metro core switch Black Diamond (BD10K) 22 rack mount units 1280 Gbps capacity (blocking) Up to Gigabit ports Up to /100/1000 ports Powerful VLAN, Virtual router Layer2 and Layer3 capabilities Proprietary EAPS link-protection protocol provides continuity in case of fiber cut L2/L3 Quality-of-Service L2/L3 hardware filtering and priority Jumbo frames at 9212 High availability, hardware redundancy

6 NZNOG 2007 – Inspire.Net / Massey University, Palmerston North, February 1st, 2007 Extreme Networks Summit X450a – edge switch X450a-24t with 24 ports 10/100/1000 copper, four combined SFP GigE ports X450a-24x with 24 ports 1-GigE SFP, four combined 10/100/1000 ports Optional dual 10-Gigabit Ethernet uplinks 1 RU form factor 160 Gigabits-per-second capacity 65 million packets-per-second forwarding performance Stacking capability with XOS 11.7 (April 2007) Other features similar to Black Diamond series

7 NZNOG 2007 – Inspire.Net / Massey University, Palmerston North, February 1st, 2007 Juniper Networks M320 Multi-service Edge Router 320 Gbps switching capacity 8 FPC slots with 20 Gbps FD capacity ½ rack size 32 PICs per chassis 10 GigE capable

8 NZNOG 2007 – Inspire.Net / Massey University, Palmerston North, February 1st, 2007 L2/L3 design Connector joins KAREN via dark fiber Switch access into one or more VLANs BGP peering with core L2 packet switched data nationally L3 routing overseas

9 NZNOG 2007 – Inspire.Net / Massey University, Palmerston North, February 1st, 2007 KAREN Service Matrix

10 NZNOG 2007 – Inspire.Net / Massey University, Palmerston North, February 1st, 2007 Internet Exchange model (L2 switching + BGP Route Reflector) “Switch – don’t route” “Peer with two – route with many (others)”  scalable Internet Exchange model

11 NZNOG 2007 – Inspire.Net / Massey University, Palmerston North, February 1st, 2007 Challenges of a L2 network Redundant links will be pruned (Spanning Tree, etc), creating a star topology Only difference between L2 resilience protocols is speed Issue: capacity not utilised Issue: shortest path Issue: protect all VLANs Solution: VLANs EAST/WEST for public IP services Solution: Extreme EAPS for protection

12 NZNOG 2007 – Inspire.Net / Massey University, Palmerston North, February 1st, 2007 KAREN Multicast – two options for connectors Bootstrap as part of KAREN multicast cloud (quick start for small sites, no MSDP, but doesn’t scale) Create your own multicast domain (requires MSDP, scales well)

13 NZNOG 2007 – Inspire.Net / Massey University, Palmerston North, February 1st, 2007 Multi-Protocol BGP and routing tables

14 NZNOG 2007 – Inspire.Net / Massey University, Palmerston North, February 1st, 2007 KAREN International Separate to domestic KAREN, but co-joined As a static 3-point transit network has to implement all services (IPv4/IPv6 uni- and multicast, jumbo frames) Routing policy ensures traffic flows between NZ and overseas peers (but not between other peers) Pacific Wave landing point in Seattle poised for peering expansion Bulk of 9K routes from Internet2 ITN

15 NZNOG 2007 – Inspire.Net / Massey University, Palmerston North, February 1st, 2007 REANNZ POP

16 NZNOG 2007 – Inspire.Net / Massey University, Palmerston North, February 1st, 2007 KAREN performance tests Network commissioning in October and November successfully demonstrated capacity, delay and jitter parameters Bandwidth tests carried out as 1 Gbps VLANs POP-to-POP All L2 components stressed at or near capacity limits (see next slide for example) Delay and jitter tests carried out as RTT measurements using hardware loopbacks

17 NZNOG 2007 – Inspire.Net / Massey University, Palmerston North, February 1st, 2007 KAREN performance tests (as shown on WAND weathermap) See recorded animations at for other tests carried out during November and December.

18 NZNOG 2007 – Inspire.Net / Massey University, Palmerston North, February 1st, 2007 KAREN delay and jitter tests

19 NZNOG 2007 – Inspire.Net / Massey University, Palmerston North, February 1st, 2007 KAREN delay and jitter results

20 NZNOG 2007 – Inspire.Net / Massey University, Palmerston North, February 1st, 2007 KAREN Connectors 101 (and shopping list) KAREN is a Tier 1 network – you need to behave like a Tier 2 – control your own routing (policy). Consider the services you want: IPv4 unicast, multicast, IPv6, Jumbo Are you a heavy hitter ? Thinking of 10 Gbps ?  Router that speaks BGP, holds 20K+ routes and does 1 Gbps If you are a heavy hitter, you need VLAN support and 40K+ routes If you want jumbo frames, you need VLAN support and 40K+ routes If you want multicast, you need PIM-SM, preferably MSDP and M-BGP If you want IPv6, you need M-BGP and space for even more routes

21 NZNOG 2007 – Inspire.Net / Massey University, Palmerston North, February 1st, 2007 KAREN Connectors to date About a dozen connectors at around 15 sites, wide range of equipment CISCO 6500 series edge routers Allied Telesyn AT-9924Ts Juniper M and J series – J6350 Linux PC and Quagga Routing Suite Check Point Firewall on Nokia platform

22 NZNOG 2007 – Inspire.Net / Massey University, Palmerston North, February 1st, outlook Connectors, connectors, connectors! Access solutions for schools and other non-BGP speakers International IPv6 peering Peering with US FedNets; China, Japan R&D networks IPv6 DNS Better solution to the 2/4/8 peering problem for complex sites Pushing traffic around EAST/WEST for dual attached sites L2 PIM-SM snooping (on top of IGMP snooping) More security, core hardening Stacking support in Napier and 10 Gbps services to Havelock North Measurement infrastructure (active and passive) – capability build fund

23 NZNOG 2007 – Inspire.Net / Massey University, Palmerston North, February 1st, 2007 Summary KAREN creates a fast lane for the R&E community, inside New Zealand and with overseas R&D networks. It offers a range of new services previously unavailable or inaccessible in New Zealand, such as multicast, IPv6. It offers a test bed for novel tools and applications. Performance is stunning – go and use it!

24 NZNOG 2007 – Inspire.Net / Massey University, Palmerston North, February 1st, 2007 Acknowledgements and references REANNZ KAREN WAND weather map A cast of dozens of hands at TelstraClear and JazzTech Questions: please contact myself or David Brownlie and Clayton Ejiofor at REANNZ. Thank you!