1. Challenges and experiences with IPTV from a network point of view Challenges and experiences with IPTV from a network point of view by Jesper Dangaard Brouer Master of Computer Science ComX Networks A/S Bifrost Workshop 2009 d.18/3-2009 ComX Networks A/S

2. 2 /26 IPTV multicast from a network point of view Who am I ● Name: Jesper Dangaard Brouer – Edu: Computer Science for Uni. Copenhagen ● Focus on Network, Dist. sys and OS – Linux user since 1996, professional since 1998 ● Sysadm, Developer, Embedded – OpenSource projects ● Author of – ADSL-optimizer – CPAN IPTables::libiptc ● Patches accepted into – Linux kernel, iproute2, iptables and Wireshark

3. 3 /26 IPTV multicast from a network point of view ComX Networks A/S ● I work for ComX Networks A/S – Danish Fiber Broadband Provider ● variety of services (TV, IPTV, VoIP, Internet) ● This talk is about network experiences with IPTV – Only shortly, challenges to get multicast to work. – Main focus network: ● Pitfalls when handing large amounts of multicast traffic. ● Bonus slides will cover – Our 10GbE Linux router project. – Short status from the Netfilter Workshop 2008

4. 4 /26 IPTV multicast from a network point of view ComX: Physical surroundings ● ComX delivers fiber based solutions – Our primary customers are 1. Apartment buildings with end-user relationship 2. OEM customers, like “Syd Energi” – Apartment building have ethernet switches ● Port separation, “private” VLAN – CPE box in apartment performs ● service separation into VLANs

5. 5 /26 IPTV multicast from a network point of view What is this talk NOT about ● Lot of challenges to implement multicast routing – Its not straight forward, BUT its not the focus of this talk. ● Shortly, pitfalls to watch out for: – It can be difficult to : ● get different vendors multicast to work together ● get even the same vendor – Watch out for firmware upgrades... – Fx: New firmware Foundry FWSX 424 drop packets, MC in HW. – Also Remember: – Apartment buildings Ethernet switches, proper config. – CPE equipment needs IGMP snooping, more Settop boxes

6. 6 /26 IPTV multicast from a network point of view The focus: Scalability ● What happened when more channels were added – High-Definition channels – More than 120 IPTV channels – Approx 600 Mbit/s multicast traffic. ● What happens to a realtime service – on a loaded ethernet switch based network?

7. 7 /26 IPTV multicast from a network point of view History: What was the problem? ● Customers were experiencing bad quality, – due to packet drops ● Traffic had reached 600Mbit/s – Drops were worse on loaded links. – But dedicated links also showed significant drops ● 60 percent loading link should not have drops ● Finding the cause of packet drops – Turned out to be a harder task than expected...

8. 8 /26 IPTV multicast from a network point of view History: Measure the problem ● First objective: Be able to measure the problem – Used "IP-probe" from BridgeTech, to detect/see drops. ● Which is an IPTV analyser appliance – internally runs Linux and has a special FPGA chip – Really nice tool, but rather expensive. ● Work under pressure: OEM Customer mad!!! – Network Core team was getting desperate ● Foundry support – Found issues with their PIM implementation ● Bad distribution over trunked links ● Desperate clear PIM cache drills

9. 9 /26 IPTV multicast from a network point of view What caused the drops? ● Bursty traffic from some of the IPTV streamers. – Some of the "cheap" (100k DKK) streamers, bursting – Worst: Linux boxes with VLC and old 2.6.18 kernel ● Linux VLC machines easily fixed – by newer kernel with high resolution timer support ● The commercial streamers were harder to "fix" – Closed source – Slow and expensive support – Kickoff IPTV shaper project based on Linux

10. 10 /26 IPTV multicast from a network point of view ● Important to enable – high resolution timers and tickless OS ● to avoid VLC multicast bursts ● Kernel config options – CONFIG_HIGH_RES_TIMERS=y – CONFIG_NO_HZ=y ● Watch the kernel log messages for: “Switched to high resolution mode on CPU 0” or “Could not switch to high resolution mode on CPU 0” Hint slide: Enable Linux Highres

11. 11 /26 IPTV multicast from a network point of view How did we locate the problem? ● One of the Linux based streamers, – Alternating between 6 Mbit/s and 15 Mbit/s streams – All channels experienced more drops during 15 Mbit/s streams ● IP-probe tool, very helpful, showing the drops – but could not show the burstiness directly ● Detailed analysis of traffic dumps – Wireshark was used to show burstiness – Wireshark IO-graph with resolution < 1 ms

12. 12 /26 IPTV multicast from a network point of view Wireshark IO-graph ● Menu: Statistics → IO Graph → Tick Interval 0.001

13. 13 /26 IPTV multicast from a network point of view Bursts and switch buffer size ● From burst size – Directly calculate required switch buffer size – Fixed size packets of 1358 bytes on wire ● From burst size and interval – Calc the bandwidth usage during burst – Its easier to comprehend bandwidth bursts ● but the switch buffer size is key

14. 14 /26 IPTV multicast from a network point of view IPTV shaper project For multicasted MPEG2 Transport Streams: ● Milestone 1: Measure the drops – Implemented in Wirehark – Implemented in Kernel as iptables module ● Milestone 2: Measure the bursts – Plan to impl. RFC4445 definition of “Delay Factor” ● Milestone 3: Smooth out the bursts – Challenging due to variable bit-rate streams

15. 15 /26 IPTV multicast from a network point of view Wireshark MP2T drop detection ● MPEG2 TS drop detection in Wireshark ● Included in mainline since SVN rev. 27381 ● Use filter “mp2t.cc.drop” – Dumping 600Mbit/s traffic ● Standard tcpdump not fast enough – Causes “fake” packet drops (Func: “packet_rcv”) ● Use tcpdump/pcap with Mem Mapped IO – Function: “tpacket_rcv” ● See /proc/net/ptype for the function used – Not fast enough for realtime/continuous usage ● Cannot avoid disk writes, and mkfifo tricks not possible

16. 16 /26 IPTV multicast from a network point of view Kernel MP2T drop detection ● Implemented a iptables/netfilter module: "mp2t" – For realtime/continuous drop monitoring – Only uses 2% CPU with 600Mbit/s traffic – tcpdump solution used 100% and caused false drops ● Implemented with RCU locking – for parallel processing on each CPU ● Planned extensions – measurements of bursts – /proc/ file stats

17. 17 /26 IPTV multicast from a network point of view Conclusion: Watch out for bursts ● Packets bursts experiences – Surprisingly bursts do cause problems on 60% loaded links – Non-bursty streams are affected by bursty-streams – On Linux: Use Highres timers and no tick – Know you switch buffer sizes ● don't place low-buffer switches in backbone

18. 18 /26 IPTV multicast from a network point of view Almost: The End Goodbye and thank you for committing your patches... 81.161.128/0/18 195.135.216.0/22 87.72.0.0/16 82.211.224.0/1 9

19. 19 /26 IPTV multicast from a network point of view Upcoming projects 10 Gbit/s routing on standard hardware running Linux

20. 20 /26 IPTV multicast from a network point of view 10GbE routing on Linux ● Motivation: – Core routers are tooooo expensive! ● Linux routing experiences good – 4 years production usage of 1Gbit/s Linux routers ● Show stoppers: – Linux production routers don't route multicast – Need a multicast PIM-SM router daemon

21. 21 /26 IPTV multicast from a network point of view 10GbE Hardware evaluation ● With current hardware, – how close are we to handling 10Gbit/s ? – PCI-express shows great potential ● PCIe gen. 2 doubles the bandwidth – Starting with cheap overclocking/gaming style hardware. ● Currently: Core i7 and Phenom II X4. – End up with server class equipment. ● Network overclocking – Might help us understand the HW bottlenecks

22. 22 /26 IPTV multicast from a network point of view 10GbE port density ● PCI-express port density – cannot compete with slot based core routers – PCIe speeds can compete: (MB 48 lanes gen.2 = 192 Gbit/s) ● Per lane 2.5Gbit/s practical 2 Gbit/s, bidirectional ● PCIe gen.2 doubles BW, 5Gbit/s expected 4 Gbit/s ● Hotlava Systems – Producing a 6x 10GbE SFP+ PCI-express x16 NIC – PCIe gen.2, 4Gbit/s x16 = 64 Gbit/s, enough for 6x10GbE ● The question is – How many time 10GbE can a CPU really handle...

23. 23 /26 IPTV multicast from a network point of view Hotlava 6 x 10GbE SFP+ ● Hotlava Systems Inc. – Solved the port density issue – 6 times 10GbE one PCIe – Picture show 8x PCIe – It also comes in a 16x PCIe – Codename: ● Tambora

24. 24 /26 IPTV multicast from a network point of view Hotlava 12 x 1GbE ● Hotlava Systems Inc. – Also solved port density ● for 1Gbit/s NICs – 12 port NIC with ● Intel 82575 chips – I have this NIC in my testlab

25. 25 /26 IPTV multicast from a network point of view Current Network Cards ● Current NICs – 3x Sun Neptune 10GbE – 2x SMC 10GbE – 3x Intel 1G multi-queue 82575/82576 ● Upcoming from hotlava systems – Six port 10GbE SFP+ (PCIe x16 gen.2)

26. 26 /26 IPTV multicast from a network point of view Netfilter Workshop 2008 ● I presented scalability fixes for iptables rule loading ● POM was officially replaced by Xtables-addon – This out-of-tree stuff is maintained by Jan Engelhardt ● Patrick McHardy presented a iptables replacement ● DaveM presented his multiqueue work ● Eric Leblond and Pablo are working on conntrack tools – Eric now has commit access ● Tproxy was accepted into mainline ● Full updates see blog: http://nfws.inl.fr/en/http://nfws.inl.fr/en/

27. 27 /26 IPTV multicast from a network point of view Extra slides ● Bonus slides – if time permits – or funny questions arise

28. 28 /26 IPTV multicast from a network point of view Memory Latency ● Memory latency – is good indication for packet forwarding performance ● CPUs in my testlab – AMD Phenom X4 9950, 2MB L3-cache 4x512KB(L2) – AMD Phenom-II X4 940, 6MB L3-cache 4x512KB(L2) – Intel Core i7 920, 8MB L3-cache ● Following slides – lmbench-2.5 tool: lat_mem_rd ● varying stride size

29. 29 /26 IPTV multicast from a network point of view Stride 8

30. 30 /26 IPTV multicast from a network point of view Stride 64

31. 31 /26 IPTV multicast from a network point of view Stride 128

32. 32 /26 IPTV multicast from a network point of view Stride 256

33. 33 /26 IPTV multicast from a network point of view Route cache perf ● Improved route cache – Kernel 2.6.15 --> 2.6.25 ● Thanks to Eric Dumazet

34. 34 /26 IPTV multicast from a network point of view CPU util softirq ● Softirq CPU usage dropped – Kernel 2.6.15 --> 2.6.25 ● Patrick McHardy, improved conntrack locking