ND The research group on Networks & Distributed systems

2 ND activities ICON – Interconnection Networks –Interconnection networks are tightly coupled/short distance networks with extreme demands on bandwidth, latency, and delivery –Problem areas: Effective routing/topologies, fault-tolerance/dynamic reconfiguration, and Quality of Service VINNER – End-to-end Internet communications –Problem area: Network resilience – as a set of methods and techniques that improve the user perception of network robustness and reliability.

3 ND activities QuA - Support of Quality of Service in component architectures –Problem area: How to develop applications that are sensitive to QoS on a component architecture platform and how dynamic QoS management and adaptation can be supported Relay – Resource utilization in time-dependent distributed systems –Problem area: Reduce the effects of resource limitations and geographical distances in interactive distributed applications – through a toolkit of kernel extensions, programmable subsystems, protocols and decision methods

Assessment of Data Path Implementations for Download and Streaming Pål Halvorsen 1,2, Tom Anders Dalseng 1 and Carsten Griwodz 1,2 1 Department of Informatics, University of Oslo, Norway 2 Simula Research Laboratory, Norway

5 Overview Motivation Existing mechanisms in Linux Possible enhancements Summary and Conclusions

6 Delivery Systems Network bus(es)

7 file system communication system application user space kernel space bus(es) Delivery Systems

8 Pentium 4 Processor registers cache(s) I/O controller hub memory controller hub RDRAM PCI slots network card disk file system communication system application file system communication system application disknetwork card Intel Hub Architecture  several in-memory data movements and context switches

9 Motivation Data copy operations are expensive –consume CPU, memory, hub, bus and interface resources (proportional to data size) –profiling shows that ~40% of CPU time is consumed by copying data between user and kernel space –gap between memory and CPU speeds increase –different access times to different banks System calls make a lot of switches between user and kernel space

10 file system communication system application user space kernel space bus(es) data_pointer Zero–Copy Data Paths

11 Motivation Data copy operations are expensive –consume CPU, memory, hub, bus and interface resources (proportional to data size) –profiling shows that ~40% of CPU time is consumed by copying data between user and kernel –gap between memory and CPU speeds increase –different access times to different banks System calls make a lot of switches between user and kernel space A lot of research has been performed in this area BUT, what is the status today of commodity operating systems?

Existing Linux Data Paths

13 file system communication system application user space kernel space bus(es) Content Download

14 Content Download: read / send application kernel page cache socket buffer application buffer read send copy DMA transfer  2n copy operations  2n system calls

15 Content Download: mmap / send application kernel page cache socket buffer mmap send copy DMA transfer  n copy operations  1 + n system calls

16 Content Download: sendfile application kernel page cache socket buffer sendfile gather DMA transfer append descriptor DMA transfer  0 copy operations  1 system calls

17 Content Download: Results UDPTCP Tested transfer of 1 GB file on Linux 2.6 Both UDP (with enhancements) and TCP

18 file system communication system application user space kernel space bus(es) Streaming

19 Streaming: read / send application kernel page cache socket buffer application buffer read send copy DMA transfer  2n copy operations  2n system calls

20 Streaming: read / writev application kernel page cache socket buffer application buffer read writev copy DMA transfer  3n copy operations  2n system calls copy  One copy more than previous solution

21 Streaming: mmap / send application kernel page cache socket buffer application buffer mmap uncork copy DMA transfer  2n copy operations  1 + 4n system calls copy send cork

22 Streaming: mmap / writev application kernel page cache socket buffer application buffer mmap writev copy DMA transfer  2n copy operations  1 + n system calls copy  Three calls less than previous solution

23 Streaming: sendfile application kernel page cache socket buffer application buffer DMA transfer  n copy operations  4n system calls gather DMA transfer append descriptor copy uncorksendfilesendcork

24 Streaming: Results Tested streaming of 1 GB file on Linux 2.6 RTP over UDP TCP sendfile (content download) Compared to not sending an RTP header over UDP, we get an increase of 29% (additional send call) More copy operations and system calls required  potential for improvements

Enhanced Streaming Data Paths

26 Enhanced Streaming: mmap / msend application kernel page cache socket buffer application buffer DMA transfer  n copy operations  1 + 4n system calls gather DMA transfer append descriptor copy msend allows to send data from an mmap ’ed file without copy mmap uncorksend cork msend copy DMA transfer  One copy less than previous solution

27 Enhanced Streaming: mmap / rtpmsend application kernel page cache socket buffer application buffer DMA transfer  n copy operations  1 + n system calls gather DMA transfer append descriptor copy mmap uncorksend cork rtpmsend RTP header copy integrated into msend system call  Three calls less than previous solution

28 Enhanced Streaming: mmap / krtpmsend application kernel page cache socket buffer application buffer DMA transfer  0 copy operations  1 system call gather DMA transfer append descriptor copy krtpmsend  One call less than previous solution An RTP engine in the kernel adds RTP headers rtpmsend RTP engine  One copy less than previous solution

29 Enhanced Streaming: rtpsendfile application kernel page cache socket buffer application buffer DMA transfer  n copy operations  n system calls gather DMA transfer append descriptor copy rtpsendfile  existing solution requires three more calls per packet uncorksendfilesendcork RTP header copy integrated into sendfile system call

30 Enhanced Streaming: krtpsendfile application kernel page cache socket buffer application buffer DMA transfer  0 copy operations  1 system call gather DMA transfer append descriptor copy krtpsendfile  One call less than previous solution An RTP engine in the kernel adds RTP headers rtpsendfile RTP engine  One copy less than previous solution

31 Enhanced Streaming: Results Tested streaming of 1 GB file on Linux 2.6 RTP over UDP TCP sendfile (content download) Existing mechanism (streaming) mmap based mechanisms sendfile based mechanisms ~27% improvement ~25% improvement

32 Conclusions Current commodity operating systems still pay a high price for streaming services However, small changes in the system call layer might be sufficient to remove most of the overhead Conclusively, commodity operating systems still have potential for improvement with respect to streaming support What can we hope to be supported? Road ahead: optimize the code, make patch and submit to kernel.org

33 Questions??