1. User-Level Interprocess Communication for Shared Memory Multiprocessors
Brian N. Bershad, Thomas E. Anderson, Edward D. Lazowska, and Henry M. Levy
Presented by Arthur Strutzenberg

2. Interprocess Communication
In the LRPC paper/presentation, it discussed the need for
Failure Isolation
Extensibility
Modularity
Usually a balance between the 3 needs and performance
This will is a central theme for this paper as well.

3. Interprocess Communication
Traditionally this is the responsibility of the Kernel
This suffers from two problems
Architectural Performance
Interaction between kernel based communication and user level threads
Generally designers use a pessimistic (non cooperative) approach
This begs the following question
“How can you have your cake and eat it too?”

4. Interprocess Communication
What if the Communication layer is extracted out of the kernel, and made part of the User level
This can increase performance by allowing
Messages sent between address spaces directly
Elimination of unnecessary processor reallocation
Amortization (processor reallocation (when needed) is spread over several independent calls)
Parallelism in message passing is exploited

5. User-Level Remote Procedure Call (URPC)
Allows communication between address spaces without kernel mediation
Isolates
Processor Reallocation
Thread Management
Data Transfer
Kernel is ONLY responsible for allocating processors to the address space

6. URPC & Communication Application OS Communication typically is
Narrow Channel (Ports)
Limited Number of Operations
Create
Send
Receive
Destroy
Most modern OS have support for RPC

7. URPC & Communication What does this buy URPC?
RPC is generally limited in definition about how the channels of communication operate
Also the definition generally does not specify how processor scheduling (reallocation) will interact with the data transfer

8. URPC & Communication URPC exploits this information by
Messages passed through logical channels are kept in memory that is shared between client and server
This memory once allocated is kept intact
Thread management is User Level (lightweight instead of “Kernel weight”)
(Haven’t we read this in another paper?)

9. URPC & Thread Management
There is less overhead involved in switching a processor to another thread in the same address space (context switching) versus reallocating it to another thread in a different address space (Processor Reallocation)
URPC uses this along with the user level scheduler to always give preference to threads within the same address space

10. URPC & Thread Management
Some numbers for comparison:
A context switch within the address space
15 microseconds
A processor reallocation
55 microseconds

11. URPC & Processor Allocation
What happens when a client invokes a procedure on a server process and the server has no processors allocated to it?
URPC calls this “underpowered”
The paper identifies this as a load balancing problem
The solution is reallocation from client to server
A client with an idle processor can elect to reallocate the idle processor to the server
This is not without cost, as this is expensive and requires a call to the kernel

12. Rationale for URPC
The design of the URPC package presented in this paper has three main components
Thread Management
Data Transfer
Processor Reallocation

13. Lets kill two birds with one stone
URPC uses an “optimistic reallocation policy” which makes the following assumptions
The Client will always have other work to do
The server will (soon) have a processor available to service messages
This leads to the “amortization of cost”
The cost of a processor reallocation is spread over several calls

15. Why the optimistic approach doesn’t always hold
This approach does not work as well when the application
Runs as a single thread
Is Real time
Has high latency I/O
Priority Invocations
URPC handles this by allowing the client’s address space to force a processor reallocation to the server’s even though there might still be work to do

16. The Kernel handles Processor Reallocation
URPC handles this through call called
“Processor.Donate”
This passes control of an idle processor down to the kernel, and then back up to a specified address in the receiving space

17. Voluntary Return of Processors
The policy of URPC on its server processors is
“…Upon receipt of a processor from a client address, return the processor when all outstanding messages from the client have generated replies, or when the server determines that the client has become ‘underpowered’….”

18. Parallels to the User Threads Paper
Even though URPC implement a policy/protocol, there is absolutely no way to enforce it. This has the potential to lead to some interesting side effects.
This is extremely similar to some of the problems discussed in the User Threads paper
For example, a server thread could conceivably continue to hold a donated processor and handle requests from other clients

19. What this leads to… One word: STARVATION
URPC handles this by only directly reallocating processors to load balance.
In other words, the system also needs the notion of preemptive reallocation
The Preemptive reallocation must also adhere to
No higher priority thread waits while a lower priority thread runs
No processor idles when there is work for it to do (even if the work is in another address space)

20. Controlling Channel Access
Data flows in URPC involving different address spaces use a bidirectional shared memory queue. The queues have a test and set lock on either end, which the papers specifically state must be NON SPINNING
The protocol is, if the lock is free, acquire it, otherwise go on and do something else
Remember this protocol operates under the assumption that there is always work to do!!

21. Data Transfer Using Shared Memory
There is still the risk of what the paper refers to as the “abusability factor” with RPC, where Clients & Servers can
Overload each other
Deny service
Provide bogus results
Violate communication protocols
URPC passes the responsibility to handle this off to the stubs.

22. Cross-Address Space Procedure Call and Thread Management
This section of the paper identifies that there is a correspondence between
Send  Receive
(messaging)
And
Start  Stop
(Threads)
Does this not remind everybody of a classic paper that we had to read?

23. Another link to the User Threads Paper
Additionally the paper identifies three arguments with the thread—message relationship
High performance thread management facilities are needed for fine-grained parallel programs
High performance can only be provided at the user level
The close interaction between communication and thread management can be exploited

24. URPC Performance Some comparisons: (values are in microseconds) Test
URPC Fast Threads
Taos Threads
Ratio of Taos Cost to URPC Cost
Procedure Call 7
1.0
Fork 43
1192
27.7
Fork;Join
102
1574
15.4
Yield 37 57
1.5
Acquire, Release 27
PingPong 53
271
5.1

25. (values are in microseconds)
URPC Performance
URPC can be broken down into 4 components
Send
Poll
Receive
Dispatch
(values are in microseconds)
Component
Client
Server
Poll 18 13
Send 6
Receive 10 9
Dispatch 20 25
Total 54 53

26. Call Latency and Throughput
Call Latency is the time from which a thread calls into the stub until control returns from the stub.
These are load dependent, and depend on
Number of Client Processors (C)
Number of Server Processors (S)
Number of runnable threads in the client’s Address Space (T)
The graphs measure how long it takes to make 100,000 “Null” procedure calls into the server in a “tight loop”

27. Call Latency and Throughput

28. Conclusions
In certain circumstances, it makes sense to move the Communication layer from the kernel to user space.
Most OS’s are designed for a uniprocessor system, and are ported over to an SMMP system.
URPC is one example of a system that is designed for SMMP directly, and takes direct advantage of the characteristics of the system

29. Conclusions
As a lead in to Professor Walpoles Discussion and Q&A, lets conclude by trying to fill out the following table:
RPC Type
Similarities
Differences
Generic RPC
LRPC
URPC