Filterfresh COOTS’98 Department of Computer Science Courant Institute of Mathematical Sciences New York University Filterfresh: Hot Replication of Java RMI Server Objects Arash Baratloo, P. Emerald Chung, Yennun Huang, Sampath Rangarajan, and Shalini Yajnik Bell Laboratories Lucent Technologies

Filterfresh COOTS’98 Filterfresh Goals uSupport highly-available RMI services in presence of failures uHandle crash failures uTransparent failure masking uEasily integrate into Java RMI

Filterfresh COOTS’98 Roadmap 4 Goals uRMI Registry architecture & crash failures uRMI architecture & crash failures uProcess group approach to fault tolerance uHighly available registry service u“Reverse lookup” for masking (state-less) servers failures uTowards highly available servers uConclusions

Filterfresh COOTS’98 RMI in a nutshell uStep 1: a server object registers with the RMI registry running on the local host uSteps 2-3: Clients get server’s remote reference by performing a lookup operation at a known registry uStep 4: Given a remote reference, clients invoke server’s methods through RMI

Filterfresh COOTS’98 Limitations of RMI Registry uSingle point of failure uClients need to know a priori which registry to contact uDoes not allow multiple RMI servers to register under the same service name  Not suited for replicated highly-available RMI server objects

Filterfresh COOTS’98 Desirable properties of RMI Registry uDistributed to remove the single point of failure  Ability to dynamically add registries, and to detect and remove failed processes  Highly available uReplicated to remove the a priori requirement  Replication strategy to maintain a consistent global state uSupport for multiple RMI servers to register under the same service name uThus, to provide high-availability to RMI server objects we need a highly-available registry service!

Filterfresh COOTS’98 RMI Architecture uThe programmer writes the client and server application codes uThe RMI compiler (rmic) generates the client stub and server skeleton uThe RMI package implements the RRL and transport layers uTransparent masking of failures must occur below the stub/skeleton levels

Filterfresh COOTS’98 A unified solution uFault-tolerance based on process group approach  Non-faulty processes form a logical group  Members interact using a set of group primitives  Group primitives are guaranteed to be reliable -- all or nothing  Group primitives are guaranteed to be ordered  Group members have a consistent view of other group members uApplications built on process groups view events in a synchronous fashion:  The group view changes for all members as though it is instantaneous -- synchronous  Events (e.g, send & receive of multicasts) occur in a logical order, within the same view  Members have the same view of the group

Filterfresh COOTS’98 Strong Virtual Synchrony uProgress: a joining process will eventually become part of the group view (or be suspected of failures) uFailure detection: a crashed process will eventually be detected and removed form the group view uReliability: messages sent by a member that remains in the group view will be delivered by others uOrder: messages will be delivered by others in the view it was sent uConsistency: all surviving members of a view agree on the set of messages delivered within that view uSynchrony: between two consecutive views, no message is delivered

Filterfresh COOTS’98 Fortunately uProcess group approach is  Well studied  Well defined protocols uProcess group approach has been used in building general purpose fault-tolerant  Middle-ware systems, such as Horus/Ensemble, Transis, etc.  Services, such as FT directory and file servers  OO systems, such as ISIS+ORBIX, Electra, Orca  Java middle-ware systems such as iBus uSeems a good candidate for FT RMI services

Filterfresh COOTS’98 Unfortunately uProcess Group Membership is  As hard as distributed consensus  Impossible in purely asynchronous systems with crash failures uOur implementation  Based on the “timeout” assumption  Correctness is guaranteed once terminates  Ack-based protocol for simplicity

Filterfresh COOTS’98 Basis for process groups uA GroupManager Class  100% Pure Java  built on top of UDP/IP uImplements  Group creation  Join operation (with atomic state transfer)  Leave operation  Group multicast operation  Failure detection and recovery uAll events are reliable and totally ordered

Filterfresh COOTS’98 Performance of group multicast uPentiumPro 200, Linux 2.030, Fast Ethernet connected by a hub uJDK1.1.1 uThread and object serialization influenced the performance?

Filterfresh COOTS’98 Roadmap 4 Goals 4 RMI Registry architecture & crash failures 4 RMI architecture & crash failures 4 Process group approach to fault tolerance uHighly available registry service u“Reverse lookup” for masking (state-less) servers failures uTowards highly available servers uConclusions

Filterfresh COOTS’98 FT Registry architecture uEmbedded a GroupManager class to ensure reliable ordered events uReliable and ordered group operations ensure consistent state uReplicated registry service for high availability uSupports dynamic joins w/state transfer uDetects and removes failed registry servers

Filterfresh COOTS’98 Bind operation uBind operations are sent to every replica uReliable multicast ensures every replica receives the event uOrdered group operation ensures consistency even if a new replica joins

Filterfresh COOTS’98 Lookup operation uLookup operations are handled locally uProvides location transparency to clients  able to locate servers registered at unknown hosts  no need to have a priori knowledge of server’s host

Filterfresh COOTS’98 Performance of FT Registry uPentiumPro 200, Linux 2.030, Fast Ethernet connected by a hub uJDK1.1.1

Filterfresh COOTS’98 Roadmap 4 Goals 4 RMI Registry architecture & crash failures 4 RMI architecture & crash failures 4 Process group approach to fault tolerance 4 Highly available registry service u“Reverse lookup” for masking (state-less) servers failures uTowards highly available servers uConclusions

Filterfresh COOTS’98 RMI & FT Registry uSupports multiple replicated servers to register under the same service name uObject references remain valid after the associated object has failed

Filterfresh COOTS’98 In the event of server failure uThe failure is detected below the stub level, and...

Filterfresh COOTS’98 Failure recovery for state-less servers uA “reverse” lookup returns the name of a given wire connection uThe old connection is patched with a connection to a non-faulty server uThe operation is re-attempted uTransparent to the client: illusion of a valid object reference

Filterfresh COOTS’98 FT server Architecture uClient has the illusion of a single server uIn reality, a group of servers process clients requests uOperations are performed at each server, in the same order for consistency uReplicated servers for high availability

Filterfresh COOTS’98 Highly available server objects uGroupManager ensures reliable ordering of events across all servers uGuarantee consistent server state uAutomatic detection and removal of failed server objects uState transfer provide the ability to dynamically add new server objects uIn combination with FT Registry and “reverse lookup”, clients have the illusion of a single reliable server object

Filterfresh COOTS’98 Conclusions and future work uTo provide high availability there is need for  A reliable registry service  A reliable RMI architecture uShowed suitability of process group approach by  Transparently masking failures  Easily integrated our services into Java RMI uFuture work  Complete work on general-purpose FT services  Address nested RMI calls for replicated servers