1. TNC 2007 Bandwidth-on-demand to reach the optimal throughput of media Brecht Vermeulen Stijn Eeckhaut, Stijn De Smet, Bruno Volckaert, Joachim Vermeir, Filip De Turck, Piet Demeester (Ghent University – IBBT) Ibrahim Habib, Zhaoming Li (City University of New York ) With acknowledgment to IBBT FIPA & GEISHA project, VRT, IBM & University of Antwerp

2. Brecht Vermeulen TNC 2007 p. 2 Broadcaster workflow

3. Brecht Vermeulen TNC 2007 p. 3 Tape based workflow Digital tapes Linear editing Rough cut Final editing Voice over Playout Non-Linear editing Local conversion to file

4. Brecht Vermeulen TNC 2007 p. 4 File based workflow Digital files Digital tape from camera or memory device Windows or Apple

5. Brecht Vermeulen TNC 2007 p. 5 Files: so ? Research issues: Optimal large file transfer: network & server performance Offsite transcoding/rendering farms (& editing & voice-over & subtitles,...) File-based archiving Disaster recovery

6. Brecht Vermeulen TNC 2007 p. 6 Contents Introduction Optimising server networking TCP/IP offloading vs. CPU based FTP vs. NFS vs. CIFS Network based vision Ongoing research Conclusion

7. Brecht Vermeulen TNC 2007 p. 7 TCP tuning options Adapt kernel TCP parameters (free) Bigger receive window: more data in-transit Important if bandwidth*delay is high Linux: rmem,wmem,tcp_rmem, tcp_wmem, mem,netdev_max_backlog Windows registry: Tcp1323Opts=3, GlobalMaxTcpWindowSize,TcpWindowSize,AFD DefaultReceive(Send)window e.g. Buffers and window on 4MB Jumbo frames ($) MTU 9000 bytes,..., 16000 bytes Not really a standard-> NICs, switches to be tested

8. Brecht Vermeulen TNC 2007 p. 8 TCP offloading TCP checksum & segmentation offload ($) Most modern good nics Works with standard kernel Warning: some cards say that they do offloading, but it is done in the driver software Full TCP offload ($$) Complete TCP/IP stack on the NIC (incl. retransmits, slow start...) TCP setup/teardown still by host  Webserver short connections vs. long transfers Problems with e.g. Bonding Kernel patch needed (linux)

9. Brecht Vermeulen TNC 2007 p. 9 TCP offloading Normal NICOffloading

10. Brecht Vermeulen TNC 2007 p. 10 TCP offloading tests Back-to-back tests between AMD dual Opteron systems (Opteron 246 @ 2GHz) Intel PRO/1000 NIC (4 x 1 Gbps)  TCP checksum & segm offload Chelsio T204 TOE (4 x 1 Gbps)  full TCP offload (= TCP Offload Engine) TCP throughput measured with Iperf Generates TCP streams on different interfaces Transfers are memory-to-memory Limitations PCI-X bus: 64 bit @ 133 MHz ~ 1GB/s  PCI-X is a half-duplex bus, PCI Express is a full-duplex point-to- point connection Maximal (unidir) TCP efficiency: 94.1% 941 Mbps per link  99% for 9000 byte MTU

11. Brecht Vermeulen TNC 2007 p. 11 Chelsio TOE vs. Intel Pro 1000 (MTU 1500) 4 links unidir: 3.7 Gb/s vs. Intel NIC 2.7 Gb/s 4 links bidir: 7 Gb/s vs. Intel NIC 3.2 Gb/s Jumbo frames on Intel: throughput +, CPU - TCP offloading results Intel Chelsio MTU 1500 MTU 9000 4 Gb/s 8 Gb/s 50% 100%

12. Brecht Vermeulen TNC 2007 p. 12 Protocol comparison setup Transfers between storage and memory GPFS fibre channel storage used 360MB/s write, 690MB/s read from one server 2.88Gb/s write, 5.52 Gb/s read

13. Brecht Vermeulen TNC 2007 p. 13 Protocol comparison FTP > NFS > CIFS for reads FTP > CIFS > NFS for writes FTP with chelsio close to GPFS performance

14. Brecht Vermeulen TNC 2007 p. 14 CIFS (synchr.) vs. Latency: model

15. Brecht Vermeulen TNC 2007 p. 15 Contents Introduction Optimising server networking Network based vision Broadcasters’ problems Media grid farms Archiving Disaster recovery Ongoing research Conclusion

16. Brecht Vermeulen TNC 2007 p. 16 Broadcasters’ problems Typically broadcasters work together with production houses, remote studios,... Storage and computing is not core business of broadcasters -> outsource to datacenters ? Networking seems THE solution, BUT... FTP > NFS > CIFS+delay issue: but remember windows editing clients -> CIFS HDCAM-SR: 440Mb/s video codec Storage bandwidth: both for archiving (and retrieve something from archive), disaster recovery Time-critical (journals)

17. Brecht Vermeulen TNC 2007 p. 17 Mediagrid farms Editing on standard definition, rendering on rendering farms on HD (editing effects, cuts,...) Problems: Standard grid infrastructure is more directed towards computing intensive vs. storage/dataset intensive tasks For broadcasters: guarantees are needed on bandwidth and computing availability Bandwidth to the rendering farms should be high, but can be by reservation (e.g. for non-live productions).

18. Brecht Vermeulen TNC 2007 p. 18 Archive: (S)ATA disk price evolution Source: own purchase prices 1999-2007

19. Brecht Vermeulen TNC 2007 p. 19 Archive Cheaper disks and tape library systems: Online/nearline file-based archiving Storage management

20. Brecht Vermeulen TNC 2007 p. 20 Archive providers

21. Brecht Vermeulen TNC 2007 p. 21 Archive needs Only high bandwidth when retrieving content Uploading of content may be slower Some content may be duplicated to two sites, other to only one site Reservations for guaranteed bandwidth ?

22. Brecht Vermeulen TNC 2007 p. 22 Disaster recovery Central storage is large Production is done on this Total restore = > 24 hours Solution: Working on remote copy ? Networking/server performance ? Client CIFS ? Bandwidth guarantees on-demand for this ?

23. Brecht Vermeulen TNC 2007 p. 23 Contents Introduction Optimising server networking Network based vision Ongoing research VPN between Gent and New York Conclusion

24. Brecht Vermeulen TNC 2007 p. 24 VPN: Gent – New York For now: only 100Mb/s Figure provided by Dante

25. Brecht Vermeulen TNC 2007 p. 25 CVLSR CHEETAH Virtual Label Switching Router Linux control PC with GMPLS engine Ethernet switch with bandwidth reservations Due to delay in setup and performance issues, research is still ongoing One possible way

26. Brecht Vermeulen TNC 2007 p. 26 Conclusions Demand from broadcasters: Bandwidth and remote storage/computing Large files Research: Optimal configuration and tuning of protocol parameters and servers to use the bandwidth Is bandwidth reservation a solution for network distribution of this functionality ?