Industrial project Sergey Semenko & Ivan Nesmeyanov Under supervision of Eliezer Levy Erlang routing mesh overview and implementation details.

W W W W W W NM Virtual machine Node structure. User processes NM – Node Manager, process in charge of communicating with mesh and dispatching jobs to Workers. NM can play two different roles in mesh: Root and Leaf. More about this in mesh topology description W – Worker, process that executes jobs. Capable of executing one job at a time. When done, notifies NM and client.

Node structure. Supervision tree S – Supervisor, is a system process in charge of restarting crashed processes. Doesn’t implement user logic. NM and W are user processes NM w w w w w w

Node Manager’s roles. Leaf Leaf Node manager dispatches received jobs to local workers gets “done” notifications and notifies its root. Worker sends job result directly to client and notifies Node manager. W W W W W W NM job done result

Node manager’s roles. Root Roots are responsible for getting jobs from web server and forwarding them to the least occupied Leaf they have. Roots are also responsible for accepting join mesh requests from node managers and assigning roles to them. Roots do not execute jobs on their local workers. R LLL

Mesh topology overview Each HTTP request is forwarded by web server to a randomly chosen Root. Each Root forwards jobs to the least occupied Leafs registered at them Results are sent directly to web server from workers. Web Server R R R R R R R R R R HTTP requests Mesh job requests

Mesh topology. Join protocol Each root has at most MaxChildNum leaves. Root that has maximum children is called saturated. Root with less children is called hungry. There are always at least MinRootNum roots in operating mesh. If there are less, each new node is assigned a root role. If all roots are saturated, a new node is assigned to be a Root. Otherwise it becomes hungry root’s leaf.

Recovery protocol. When a leaf crashes, its root is notified and its jobs are reassigned. When root crashes, all its leafs perform a join request. If such a leaf gets root role, it reassigns its pending jobs. When both leaf and its root crash and there is no info about the job in mesh, web server resends the job after a timeout.

Sending job protocol Web server randomly chooses one of the roots and sends the request to it. If that root has no leaves registered, web server is notified and request is resent. Otherwise the job is forwarded to the least occupied leaf of the root. If all leaf’s workers are occupied the job is stored in the pending jobs list. When worker becomes available it is assigned one of the pending jobs

Implementation details. Process groups There are two registered pg2 groups: root_group and hungry_group. When a node manager becomes a root, it joins root_group and hungry_group. When root has maximum number of children, it leaves the hungry_group. When a saturated node looses a leaf, it rejoins hungry_group. Access to groups is synchronized by a global lock to avoid race conditions.

Locking clarification An Erlang way to synchronize access to a shared resource is by implementing a “resource manager” process that would get access requests and execute them. Due to requirement to have no single point of failure we decided not to implement such a process to sync access to root groups. Hence we were forced to implement a locking primitive which is not an Erlang way to solve the problem.

Implementation details. Monitoring Each leaf monitors its root (erlang:monitor() function). Each root monitors its leaves.

Implementation details. Node manager module Implements gen_server behavior. Role is preserved in the state. When the role is changed only the corresponding field in the state changes. Node manager is capable of processing other roles’ messages (this is useful when leaf turns into a root and might still get job_done messages from its workers)

Implementation details. Worker module Implements gen_server behavior. The only function is to execute jobs. Job execution is not of our interest. It is simulated by sleeping a certain amount of time.

Implementation details. Mesh module Provides interface for sending jobs protocol. Interface for joining mesh protocol.