Agnès Pouélé Jan Novak Inter-domain Multicast in European Research Networking TEN-155 Operational Experience and Deployment on GÉANT NANOG24 Miami 12th February 2002 Agnes Pouélé, DANTE Network Engineer Jan Novak, Cisco Systems Inc. Network Consulting Engineer 1

Agnès Pouélé Jan Novak Agenda DANTE TEN-155 Operational Experience –Evolution of the TEN-155 multicast topology from 1998 to 2000 –Operational Experience GÉANT deployment –Overview of GÉANT Network –GÉANT’s Multicast design –GÉANT Multicast and Unicast Coverage –GÉANT Multicast Service and Monitoring Conclusion 2

Agnès Pouélé Jan Novak DANTE, TEN-155, GÉANT DANTE DANTE is a not-for-profit company set up in 1993 by European National Research Network organizations. TEN-155 was an ATM based network built initially on OC3 links and then upgraded in 2000 GÉANT successors of TEN-155 launched in December Gbps Pan-European Network Is partially funded by the EC 3

Agnès Pouélé Jan Novak Part I TEN-155 Operational Experience 4

Agnès Pouélé Jan Novak Typical mess of DVMRP tunnels on Sun WSs with usual tunnel routing problems. 5 Starting point MBONE TUNNELED TOPOLOGY

Agnès Pouélé Jan Novak 6 OSLO IETF transmitted over both native STM-1 ATM based network and DVMRP tunnelled infrastructure 1999: PIM-SM and DVMRP set-up PIM-SM domain

Agnès Pouélé Jan Novak Multicast deployment in TEN-155 Based on this first set-up and successful operation, we concluded to enable multicast on all production routers in TEN-155. Multicast code stable, CPU usage ok, M-BGP ok PIM-SM “only” for production service PIM-SM to DVMRP border works, but not possible to operate routinely (NOC) From 1999 to 2000 each country was migrated from the DVMRP cloud to a interconnection with TEN-155 using PIM-SM/MBGP/MSDP 7

Agnès Pouélé Jan Novak 2000: TEN-155 final topology NL United Kingdom France Italy Greece Belgium Spain CH FR UK SE IT DE Slovenia Czech Rep Germany AT Switzerland Portugal AS8933 OSPF + internal MBGP Poland Israel Netherlands Nordics e x t e r n a l B G P AT & T UUNET Luxembourg Hungary US ABILENE 8 tunnel unicast/mcast line dedicated mcast

Agnès Pouélé Jan Novak TEN-155 Operational Experience Performance Impact Parameters to be considered for the exploitation of Multicast –CPU - parameters about 1600 forwarding (mroute) entries max 8 outgoing interfaces (average 2-3) max 20 Mbit/s of data forwarded by one router resulting in 5-10% of CPU increase (mainly PIM) 9

Agnès Pouélé Jan Novak TEN-155 Operational Experience Performance Impact Parameters to be considered for the exploitation of Multicast –Memory about 3000 SA messages in the cache about 1600 forwarding entries (mroute) about routes in the MBGP table These parameters didn’t represent a significant memory usage 10

Agnès Pouélé Jan Novak Usage of MSDP – new component – new storms Default peer in redundant topology “Redundant” mesh-groups MSDP peers default peer Group A Group C Group B Impact of a MSDP storm on the CPU load 11

Agnès Pouélé Jan Novak Usage of MSDP – early implementation problems Cisco IOS S and lower –ghost SA entries in the MSDP cache SA messages recreated by the incoming (S,G) joins FIXED –Origination of SA messages only when source registered to the originating RP –“ip msdp redistribute” command – without arguments re-originated all known SAs caused huge increase of the SA counts worldwide FIXED 12

Agnès Pouélé Jan Novak TEN-155 MSDP monitoring MSDP usage monitoring number of RPs, average 90 (40 EU) number of groups number of pairs source, group (S,G) number of SA messages per minute Monitoring set-up WS - DANTE DE.TEN-155 router Software – adapted C++ MSDP implementation of Steve Rubin 13 MSDP peering

Agnès Pouélé Jan Novak TEN-155 MSDP monitoring Number of RPs originators announced to TEN-155 with and without multicast US connectivity number of RPs 14

Agnès Pouélé Jan Novak TEN-155 MBGP monitoring MBGP monitoring stability/updates number routes, max about 9000 (760 EU) number of ASNs, average 240 (80 EU) Monitoring set-up WS - DE PoP DE.TEN-155 router Software - Merit’s MRTD - modified SAFI definition for MBGP 15 MBGP peering

Agnès Pouélé Jan Novak TEN-155 MBGP monitoring In blue, the number of routes originated from one AS In green, the number of updates originated by the same AS stability/updates 16

Agnès Pouélé Jan Novak TEN-155 Multicast Monitoring tools The graph values from mbgp and msdp monitoring are historical values. 17

Agnès Pouélé Jan Novak TEN-155 Operational Experience Conclusion The adoption of PIM-SM domain (without DVMRP) encouraged the NRENs to do the same. Unicast and multicast non congruent MSDP peer doesn’t need to be RP. Concept of two BGP tables and “multi-protocol” RPF check often still misunderstood. Inter-domain Multicast debugging –Almost impossible to fix problems in just one week TEN-155 work provided valuable inputs for GEANT multicast services and design 18

Agnès Pouélé Jan Novak PART II NEXT GENERATION GÉANT DEPLOYMENT 19

Agnès Pouélé Jan Novak 20

Agnès Pouélé Jan Novak GÉANT Services GÉANT Standard IP Service –IP traffic from NREN to NREN and Research peerings. Multicast Service (rolling out now) Replacement of TEN-155 Managed Bandwidth Service –GÉANT Premium IP Service –Layer-2 VPNs (forthcoming) Upcoming Security and Dos attack detection, IPV6 21

Agnès Pouélé Jan Novak GÉANT’s Customers and Other Peerings GÉANT Unicast Customers –27 countries in Europe GÉANT Multicast Customers –24 countries in Europe GÉANT Unicast and Multicast research and commercial peerings –Abilene, Canarie and ESnet via GTREN –Infonet, UUNET –KPNQwest, Global Crossing 22

Agnès Pouélé Jan Novak GÉANT Multicast Customers 23

Agnès Pouélé Jan Novak GÉANT Multicast Implementation Current design built on –The experience gained from TEN-155 –Guidelines from multicast experts –Juniper Laboratory tests –GÉANT Multicast services –Multicast transit domain NRENs to NRENs –Multicast transit domain NRENs to other PEERS –Beacon monitored backbone GÉANT Multicast routing policy at: – MULTICAST/routing-policy.html 24

Agnès Pouélé Jan Novak HostX Mangan ese Iridium HeliumGallium Sulfur HostY Xenon Network 10.2/16 –AS 102 Network 10.0/16 -AS 100 Network 10.1/16 -AS 101 HostZ Tin Network 10.3/16 –AS 103 Group Source Source Group Group Source Group Source Group Source TEST-BED LAB 26/27th Nov 2001 M-BGP peering MSDP peering I-MBGP full mesh Junos 5.0R3.3 mesh group 25 I-MSDP full mesh

Agnès Pouélé Jan Novak PIM SMv2 GÉANT domain GÉANT single PIM-SMv2 –Version 2, enabled on all interfaces. Three Rendez-vous Points with private anycast address. –backup for internal sources and receivers. –Private Anycast address (filtered out) –closest RP based on the OSPF cost All other interconnected administrative domain have to be PIM-SM v2 enabled with their dedicated RP. 26

Agnès Pouélé Jan Novak PHYSICAL TOPOLOGY: RP uk fr at ch cz pl hu sk gr ie si be nl 40 lu it es de se IUCC Janet NY4-1 Nordunet Eenet Latnet Litnet Posnan DFN NY4-2 Infonet Multicast access Rendez-vous Point Cesnet Sanet Hungarnet RoEduNet Arnes Grnet Switch&Cern Renater Rediris FCCN Surfnet Belnet Restena Heanet Carnet INFN Aconet Unicom-bCynet STM64/OC192 STM16/OC48 STM4/OC12 STM1/OC3 27 PoP

Agnès Pouélé Jan Novak GÉANT Design: MBGP and MSDP MBGP Separate multicast routing table (inet.2) Currently congruent BGP and MBGP topology in Europe iMSDP MSDP is fully meshed between 19 PoPs Use of mesh group i-MSDP Peering with loopback addresses (Not the anycast address !!) eMSDP NREN GÉANT Access Router 28

Agnès Pouélé Jan Novak MSDP peering Logical view iMSDP Mesh Group iMBGP Full Mesh uk se de1 cz hu at es fr Iucc JanetNy4-1 Renater Switch&Cern Rediris FCCN INFN Arnes RoEdunet Sanet Posnan Cesnet DFN NY4-2 Infonet GRnet Nordunet Latnet Litnet Eenet Carnet gr Cynet Unicom-b ch lu Restena be Belnet it si Hungarnet sk de2 pl ie Heanet pt Surfnet nl Aconet External MSDP peering NRN GÉANT router 29 RP GTREN Abilene

Agnès Pouélé Jan Novak MSDP SA Filtering A list of filtered SA is defined at: – MULTICAST/deployment-msdp.html We filter the recommended list. We authorise /16 from the IPV4 Organisation Local Scope through GÉANT. 30

Agnès Pouélé Jan Novak GÉANT AS20965 CAnet Third party provider European Distributed Access Commodity Internet Access DFN NREN2 JANET NREN4 GTREN RESEARCH Peerings STM-16 UKDE Esnet... Abilene 31 STM-4 GÉANT Multicast and Unicast Coverage

Agnès Pouélé Jan Novak GÉANT Multicast Service Access to the service –Via the primary access to GÉANT –Via a GRE tunnel (currently nobody) –Support of PIM-SM v2 only Operational procedures (rolling out now) –Goal: fully supported service as Unicast. Troubleshooting –Extension of the trouble ticket systems to multicast incidents 32

Agnès Pouélé Jan Novak GÉANT multicast monitoring Beacon –Tool initially developed by Kai Chen from NLANR dast.nlanr.net/projects/Beacon/ Relies on a number of Agents spread over the network which simultaneously send and receive multicast packets carrying a packet sequence number and a timestamp. Communicates with a central server which displays matrices of Agents via web pages. 33

Agnès Pouélé Jan Novak GÉANT multicast monitoring Recommended by TF-NGN group –( To use from day 1 –monitoring of multicast inside and outside of GÉANT. Server code enhanced with historical functionality  ( Menu item “Applications”)   Multicast Beacon Agent written in C  34

Agnès Pouélé Jan Novak GÉANT multicast monitoring Beacon’s matrices –One for the internal sources of GÉANT Each GÉANT POP has a beacon agent installed –One for the external sources of GÉANT –We have assigned two multicast groups from GLOP range [RFC2770] for each matrix Parameters monitored Loss Delay Jitter 35

Agnès Pouélé Jan Novak Beacon internal/external matrix 36

Agnès Pouélé Jan Novak Other Monitoring Tools Per group monitoring from TEN-155 –Based on the IETF IP-MROUTE MIB –Shows traffic per multicast group per interface –Under installation on ws1.se.geant.net MSDP and MBGP monitoring tools –Not yet available 37

Agnès Pouélé Jan Novak CONCLUSION Deployment status –beaconserver.geant.net:19999/ Conclusion –From TEN-155 to GÉANT Unicast and multicast moves to a congruent topologyacross Europe and towards research peerings. Links – – – 38

Agnès Pouélé Jan Novak We would like to thank and acknowledge the help of the people who worked and are working with us on these projects, mainly from all EU and US research networks 39 THANKS

Agnès Pouélé Jan Novak Questions ? 40

Agnès Pouélé Jan Novak SA Filter list 41