CS 443 Advanced OS Fabián E. Bustamante, Spring 2005 U-Net: A User-Level Network Interface for Parallel and Distributed Computing T. von Eicken, A. Basu, V. Buch and W. Vogels Cornell University Appears in SIGOPS 1995 Presented by: Joseph Paris

2 Introduction There has been a shift in local area network bottleneck –Traditionally, limited bandwidth –Now we see an issue in the message path through software Taking a look at the UNIX networking architecture –Message path through the kernel consists of Several Copies Crossing multiple levels of abstraction between device drivers and user applications Resulting in…. Overhead –We observe that the processing overheads limit the peak communication bandwidth and result in high latency –So, the upgrades in networking technology largely go unnoticed to the general user community Vendor supplied problem? –May think of large data-stream cases and less about per message overhead

3 Observation Most applications use relatively small messages and rely heavily on quick round-trip requests and replies –Distributed shared memory –Remote procedure calls –Remote object-oriented method invocations –Distributed cooperative file caches And, they could also benefit from more flexible interfaces to the network –Traditional architecture cannot easily support new protocols/interfaces –Integrating application specific information into protocol processing Higher efficiency Greater flexibility –I.e. Video, Audio, Transferring directly from data structures

4 Motivation Low end-to-end communication latencies Separating processing overhead from network latency –Distributed Systems Object-oriented Technology –Objects are generally small (100 bytes vs. Kbytes) Electronic workplace –Simple database servers that handle object naming, location, authentication, protection. (20-80bytes for requests, bytes for response) Cache Coherence –Keeping copies consistent introduces a large number of small coherence messages. Fault-tolerance Algorithms/Group Communication –Global locks, scheduling, coherence RPC’s, file systems, etc.

5 Motivation Small message Bandwidth –Same trends that demand low latencies also demand high bandwidth for small messages Object-oriented Technology, Electronic workplace, Cache Coherence, RPC’s, etc –Part of decreasing the overall end-to-end latency is having high-bandwidth technology for small messages –Basically, we want full network bandwidth with as small messages as possible Protocol Interface Flexibility –Traditionally protocol stacks are implemented as part of the kernel Lack of integration of kernel and application buffer management –Solution Remove the comm. Subsystem’s boundary with the application specific protocols Tight coupling between the comm. Protocol and the application

6 Solution - Unet Why? –Focus on low latency and high bandwidth using small messages –Emphasis on protocol design and integration flexibility –Desire to meet goals on widely available ‘off the shelf’ hardware How? –Simply, remove the kernel from the critical path of sending and receiving messages Eliminates the system call overhead Offers opportunities to streamline the buffer management –What’s required? Virtualizing the network interface among processes Protection such that processes using the network cannot interfere with each other –Message Multiplexing and De-Multiplexing Managing communication resources without the kernel Efficient and Versatile programming interface to the network

7 Design & Implementation of U-Net Virtualize the network interface in such a way that a combination of OS and hardware mechanisms can provide the illusion of owning the interface –In hardware Components manipulated by a process correspond to real hardware –In software Memory locations are interpreted by the OS –Both The Role of U-Net is limited to –Multiplexing the actual network interface among all processes –Enforcing protection boundaries –Enforcing consumption limits This leaves the process with control over –Contents of the message –Management of send and receive resources (such as buffers)

8 Design & Implementation of U-Net We have 3 main building blocks –Endpoints Serve as an applications handle into the network and contain… –Communication Segments Regions of memory that hold message data –Message Queues Holds descriptors for messages that are to be sent or have been received –Each process that wants to access the network Creates one or more endpoints Associates a communication Segment with the endpoint And a set of send, receive, free message queues

9 Design & Implementation of U-Net Sending –User process composes the data in the communication segment –Pushes a descriptor for the message onto the send queue –At this point the network interface is expected to pick the message up and insert it into the network If there is a back-up –Leave the descriptor in the queue –Eventually exert back-pressure to the user process when the queue becomes full Receiving –Messages are de-multiplexed based on their destination Data is transferred to the appropriate comm. Segment The message descriptor is pushed onto the corresponding receive queue –Receive model notification Polling –Blocks waiting for the next message to arrive via the UNIX select call Event Driven –Register an Up-Call »Signals the state of the receive queue that satisfies a certain condition –Only two conditions currently supported »Queue is non-empty »Queue is almost full –In order to keep performance high (and cost low) all messages can be consumed on a single up-call

10 Design and Implementation of U-Net Multiplexing and De-Multiplexing Messages –Uses a tag in each incoming message to determine destination endpoint Comm. Segment Message queue descriptor –Exact form of the message tag depends on the network substrate i.e. ATM uses virtual channel identifiers Getting the tag via an OS level service assists in –An application in determining the correct tag to use based on a specification of the destination process and the route between the two nodes route discovery Switch-path setup other signaling that is specific to the network technology –Authentication and authorization Performs checks to ensure that the application is allowed to access specific network resources Also checks to make sure there are no conflicts with other applications

11 Design and Implementation of U-Net Base-level Architecture –Hardware cannot support Direct-Access “True Zero-Copy” where data can be sent directly out of the applications data structures without intermediate buffering Requires special memory mapping to span the entire processes address space into the network interface –So we only get “Zero-Copy” support for now Which in reality requires a single copy, namely between the application’s data structures and a bugger in the communication segment –Queue based interface to the network Stages messages in a limited size comm. Segment on their way between application data structures and the network Send and Receive queues hold descriptors with information about the destination, origin, endpoints, length, as well as offsets within the comm. segment –Management of the send buffer is entirely up to the process »Must be properly aligned for the requirements of the network interface –Cannot control order in which messages are received into the Recv Buffer Free queues hold descriptors for free buggers that are made available to the network interface for storing arriving messages –Small Message Optimization Send and recv queues may hold entire messages in descriptors (instead of pointers to data) Avoids buffer management and can improve round-trip latency

12 Evaluation Two U-Net implementations –SBA-100 Non-programmable, completely done in software Performance sucks –33-40% increase in overhead due to ATM header CRC calculation being done in software –SBA-200 Programmable, custom firmware Reflects the base-level U-Net architecture in hardware Three tests –U-Net Active Messages Implementation (UAM) »Active messages is a mechanism that allows efficient over-lapping of communication with computation in multi-processors »Communication in form of requests and matching replies –Split-C »Parallel extension to C for programming distributed memory machines using a global address space abstraction »Comprises of one thread of control per process from a single code image and the threads interact through reads and writes on shared data »Implemented with U-Net Active Messages –TCP/UDP

13 Evaluation Active Message (UAM) U-Net round-trip time as a function of message size U-Net bandwidth as a function of message size

14 Evaluation Split-C Using UAM CPU and Network Breakdown for two applications Overall Execution Time

15 Evaluation TCP/UDP U-Net UDP Bandwidth as a function of Message Size U-Net TCP Bandwidth as a function of Message Size

16 Evaluation TCP/UDP Latency as a function of Message Size

17 Conclusion Processing overhead on messages has been minimized Latency experienced by the application is once again dominated by the actual message transmission time Simple networking interface that supports traditional inter-networking protocols and abstractions such as Active Messages Demonstrates that removing the kernel from the communication path can offer new flexibility in addition to high performance –TCP/UDP protocols achieve latencies and throughput close to the raw maximum