Reactor Design Pattern Lecture 13 Reactor Design Pattern

Overview Blocking sockets - impact on server scalability. Non-blocking IO in Java - java.nio package complications due to asynchronous nature impact on message parsing algorithms Reactor design pattern generic server more scalable than our earlier solutions …

Sockets Socket API: interaction with RTE when a process requires communication services. Three properties of Sockets: ServerSockets: accept incoming connections. OutputStream: send bytes through Socket InputStream: receive bytes through Socket

blocking IO operations Control does not return to calling thread until operation terminates. accept() - calling thread is blocked until new connection established. write(byte [] buffer)  - calling thread is blocked until all buffer sent to network. read(byte [] buffer)  - calling thread is blocked until buffer.length is received. This forces a server to have a thread per client (maybe from a “pool”). This poses scalability issues.

Non-blocking IO We would like to design scalable servers: Check if socket has some data available to read. Non-block read available data from socket. Return with any amount of data. Check if socket can send some data. Non-block write data to socket. Return immediately. Check if new connection is requested. If so, accept it, without blocking!

RTE perspective We partition the solution in two logical parts: Readiness notification Non-blocking input output. Modern RTEs supply both mechanisms: Is data available for read() in socket? Is socket ready to send some data? Is there new connection pending for socket? Non-blocking interface to read and write.

output buffer To understand how non-blocking operations work – we need to understand how RTE internally manages IO There’s a buffer associated with each socket. Write to socket - the RTE copies bytes to the internal buffer. RTE then sends bytes from the buffer to network.

Block vs. Non-block Write The network slower than process - output buffer fills more quickly than RTE sends bytes to network. If the output buffer is full - RTE blocks the process writes until output buffer has enough free space for data. Non-blocking write - RTE copies bytes from process as possible Notifies process how many bytes have been copied. If bytes need to be re-written, process responsible to re-invoke the write operation.

Block vs. Non-block Read RTE needs to receive bytes over the network, And deliver these bytes to our process. RTE buffer bytes until process actually requests them using a read() operation.

Block vs. Non-block Read input buffer: allocated to each socket RTE stores incoming bytes to input buffer When process read from socket, RTE copies the from socket's input buffer to process buffer. If process request more bytes than available, RTE blocks until enough bytes. non-block - RTE copy bytes available in socket's input buffer, and notify number of bytes copied.

Input Buffer Overflow Input buffer has limited space. Input overflowed - process reads data more slowly than data arrives from network. If input buffer full, RTE discard new data arriving from network sending side will retransmit the data later?

Java's NIO Package Java's interface to non-blocking IO and readiness notification services of RTE Provides wrapper classes for readiness notification and non-blocking IO.

Channels Connections to I/O entities Represent data source, destination Examples: ServerSocketChannel SocketChannel, FileChannel ServerSocketChannel provides  accept(), returns a SocketChannel  (similar to accept() method of  ServerSocket class). By default - new channels are in blocking mode Must be set manually to non-blocking mode.

Buffers A wrapper classes used by NIO to represent data interchanged through a Channel. Usually backed by array types. Example: SocketChannels use  ByteBuffer class for sending and receiving bytes. A buffer can be in “write mode” when: sock.read(buff). The socket reads from the stream and writes to buff. A buffer can be in “read mode” when: sock.write(buff). The socket reads from the buff and writes to stream. In between read and write: flip().

Selectors implements readiness notification. channels may be registered to a selector for specific readiness events read /write /accept selector can be polled to get list of ready channels Creating a Selector: Selector selector = Selector.open();

Selectors channel ready for read guarantees that a read operation will return some bytes. channel ready for write guarantees that a write operation will write some bytes channel ready for accept guarantees that accept() will result in a new connection. Example: channel.configureBlocking(false); SelectionKey key = channel.register(selector, SelectionKey.OP_READ);

Selection keys Four options for events SelectionKey.OP_CONNECT SelectionKey.OP_ACCEPT SelectionKey.OP_READ SelectionKey.OP_WRITE Selection keys hold The interest set (an int representing the events the selector listens to with respect to he channel). boolean isInterestedInRead = interestSet & SelectionKey.OP_READ; The ready set (same as interest set only for “ready”). selectionKey.isReadable(); The Channel The Selector An attached object (optional), e.g. an associated buffer.

Selectors Selector class abstracts a service given by OS under the system call select (or epoll). select() blocks until at least one channel is ready for the events you registered for. Returns the number of ready channels. After Select: Set<SelectionKey> selectedKeys = selector.selectedKeys();

Selector example Set<SelectionKey> selectedKeys = selector.selectedKeys(); Iterator<SelectionKey> keyIterator = selectedKeys.iterator(); while(keyIterator.hasNext()) { SelectionKey key = keyIterator.next(); if(key.isAcceptable()) { // a connection was accepted by a ServerSocketChannel. } else if (key.isConnectable()) { // a connection was established with a remote server. } else if (key.isReadable()) { // a channel is ready for reading } else if (key.isWritable()) { // a channel is ready for writing } keyIterator.remove();

Reactor Design Pattern solve scalability problems: task threads non-blocking IO channels manage socket communication: selector interrupt polls for readiness.

Reactor IO reactor is accepting new connections. If bytes ready to read from socket, reactor read bytes and transfer to protocol (previous lecture). if socket is ready for writing, reactor checks if there is a write request - if so, reactor sends data.

Reactor class (a server class) Has port. Has abstract protocol and message decoder. Holds a Thead pool and the main thread. Holds a selector. Holds a Task (“Runnable”) queue. It defines a NonBlockingConnectionHandler. Handles each client. The tasks of processing data are performed on a different thread.

Main Reactor thread (selectorThread) main reactor thread performs the following: Creates new thread pool (executor). Creates new ServerSocketChannel, bind to port. Creates new Selector. Registers ServerSocketChannel in Selector, asking for ACCEPT readiness. While(true) - wait for selector notifications For each each notification event check: Accept notification - server socket is ready to accept new connection - call accept. new socket created - register socket in  Selector. Write notification - socket ready for writing, if protocol ask to write - write bytes to socket Read notification - socket ready for reading, read bytes and pass them to protocol handler

pool thread actual work done by protocol will be achieved with the use of thread pool;  message processing is assigned as task for pool. event handling is done by two threads: ReactorThread pulls the bytes from socket and places them in a buffer. Thread-pool thread: processes bytes using a tokenizer and protocol. writes the protocol response back to the connection handler outgoing buffer

Difference between reactor and one-thread-per-connection ConnectionHandler: passive object (instead of active object) Methods are executed by main thread of  Reactor in reaction to events relayed by selector Methods don’t block –execute very fast - copying bytes from one buffer to another. Parsing/processing messages is delegated to active objects ProtocolTask submitted to thread pool executor

Cont’d There is a direct mapping between socket channels and their associated handlers. Selector class allows to attach and arbitrary object to a channel (in the SelectionKey), which can later be retrieved, we associate ConnectionHandler with socket created when accepting new connection.

We clear the selected keys set (“the ready set”) so we won’t have to handle those events again.

Here selectorThread changes the notifications of the listening keys. Here we put the connectionHandler as “attachment”, to a selectionKey. The handler is the state of the session. Here we submit the task to the thread pool. Following a read(), there might be a heavy task. Here selectorThread changes the notifications of the listening keys. See “undateInterestedOps()”.

Here we force that all the changes in the interestedSet will be performed by the selectorThread, to avoid concurrency issues regarding this set. SelectorTasks is protected since it is a “concurrentList”. Wakeup() wakes the selector from select().

Direct vs. non-direct buffers ByteBuffer.allocateDirect() A byte buffer is either direct or non-direct. Given a direct byte buffer, the Java virtual machine will make a best effort to perform native I/O operations directly upon it. That is, it will attempt to avoid copying the buffer's content to (or from) an intermediate buffer before (or after) each invocation of one of the underlying operating system's native I/O operations. The direct buffers typically have somewhat higher allocation and deallocation costs than non-direct buffers. It is therefore recommended that direct buffers be allocated primarily for large, long-lived buffers that are subject to the underlying system's native I/O operations.

Buffer Pool The connectionHandler uses a buffer pool. That is because DirectByteBuffers are expensive to allocate/deallocate. BufferPool in ConnectionHandler caches the already-used buffers. This is called the “Fly-weight design pattern”.

Concurrency Issues Reading tasks are performed by different threads. What about consecutive reads from the same client?? Assume a client that send two messages M1 and M2 to the server. The server then, create two tasks T1 and T2 corresponding to the messages. Since two different threads may handle the tasks concurrently, T2 may complete before T1. The response to M2 will be sent before the response to M1. The protocol may be broken.

Current code:

Naïve solution: queue of tasks for each connection handler.

Problems: The tasksQueue must be maintained (initialized and deleted). The Threads in the pool will block on tasks from the same connection instead of working on other connections.

The “Actor” thread pool. Like running actors in a play. one can submit new actions for actors, and the pool will make sure that each actor will run its actions in the order they were received while not blocking other threads.

execute

Some notes: WeakHashMap acts the ActorThreadPool uses WeakHashMap to hold the task queues of the actors. An entry in a WeakHashMap will automatically be removed when its key is no longer in ordinary use. The presence of a mapping for a given key will not prevent the key from being discarded by the garbage collector. When a key has been discarded its entry is effectively removed from the map. This class is not synchronized. And therefore we will guard access to it using the read-write lock.

Using ActorsPool in the reactor In order to not add two task of the same act to the pool it maintain the playingNow set.

Removing more sync? The only methods that now block and is executed by the pool is complete(). submit() also blocks (and performed by the server). Can we remove the synchronization completely? We will remove the playingNow list. We will wrap the Queue<Runnable> inside an Actor. Actor will also have an ExecutionState

Helper class 1 ExecutionState: holds the playingNow state…

Recall…