Copyright Fujitsu & Savvion © 2000 ebXML Reliability Messaging A Proof of Concept Implementation Fujitsu Savvion Author: Jacques Durand

Copyright Fujitsu & Savvion © 2000 Buyer Business Process Buyer Message Service Handler Seller MSH Seller BP Purchase Order PO Reliability: Non-Failure Case Msg type: Ack Msg type: Normal Msg deliv.sem: OnceAndOnlyOnce Buyer Host Seller Host PO payload PO payload

Copyright Fujitsu & Savvion © 2000 timeout Buyer BP Buyer MSH Seller MSH Seller BP PO Reliability Use Case A “Normal message” failure

Copyright Fujitsu & Savvion © 2000 Buyer BP Buyer MSH Seller MSH Seller BP PO duplicate Does not transmit Reliability Use Case B Accidental message re-send

Copyright Fujitsu & Savvion © 2000 Buyer BP Buyer MSH Seller MSH Seller BP PO timeout resend Does not transmit (duplicate) Reliability Use Case C Acknowledgement failure

Copyright Fujitsu & Savvion © 2000 Reliability Messaging: A Service Architecture Message Service Handler (MSH) Business Process HTTP SMTP HTTP Reliability Service Sent messages Sender Counter data 1 sender RS-data set [Received messages?] Receiver Counter data 1 receiver RS-data set RS interface callback interface For each two-way connection

Copyright Fujitsu & Savvion © 2000 Reliability Service: connection-stateful Message Service Handler (MSH) Business Process A HTTP RS Message Service Handler (MSH) HTTP RS Message Service Handler (MSH) HTTP RS Business Process C Business Process B This RS -enabled MSH maintains 2 sender RS-data sets 2 receiver RS-data sets This RS-enabled MSH maintains 1 sender RS-data set 1 receiver RS-data set

Copyright Fujitsu & Savvion © 2000 Reliability Service Functions Before Sending a Reliable Normal Message (sender side) After Receiving a Reliable Normal Message (receiver side) After Receiving an Ack Message (sender side) get message out of store Callback sender MSH, with message to resend When a Msg TimeOut occurs Tell receiver MSH to send back an Ack msg check Seq Nbr: Duplicate ? Ignore msg OK (new) ? Pass to receiver MSH Store message copy generate new Seq Nbr update (and return to sender MSH) msg with modified header Remove acknowledge msg from sender store cancel timeout countdown MSH

Copyright Fujitsu & Savvion © 2000 time sender receiver time sender receiver time sender receiver ack msg Sender Blocking on Acks Sender Not blocking on Acks: the good and the bad Message sequences scenarios

Copyright Fujitsu & Savvion © 2000 Problem: The original Sequence Number Method was designed for purely sequential exchanges of messages (the sender is waiting for an acknowledgement from receiver before sending next message.) It is able to detect message duplicates only in this case. How to adapt it to enforce “Once And Only Once” when Sequencing is no longer guaranteed? The enhanced solution proposed here: Non-ordered Sequence Numbers Method allows for quick detection of message duplicates spanning an arbitrary long sequence of messages, without maintaining a database of billions of messages IDs on the receiver side...

Copyright Fujitsu & Savvion © 2000 time sender receiver Problem: how to detect duplicates when sequencing is no longer guaranteed? Solution: The receiver maintains: (a) a max Seq Nbr received (MSNR), that represents the max seq counter value received so far (e.g =4 at time t i ). (b) a list (union) of past Seq Numbers intervals (SN-intervals). Each interval represents a set of received messages with contiguous sequence numbers (no matter in which order they were received). [1:2] [4] [1:4] List of SN-Intervals at time t i When receiving mesg with seq nbr = 3 New list of SN-Intervals Time t i Simple version of algorithm: When receiving a message M: if (counter(M) > MSNR), then MSNR = counter(M), // it is not a duplicate, merge counter(M) with last SN-interval [or create a new one if needed] else if counter(M) belongs to any SN-interval, then it is a duplicate, ignore it else new union of SN-intervals = current union + counter(M) MSNR=4

Copyright Fujitsu & Savvion © 2000 Problem: how to detect a counter wrap-around on receiver side, since the maxSeqNbr (e.g. 999,999,999 ) may never be obtained (or too late ) due to message failure/delay? (same for the fisrt SeqNbr). Also, the receiver could get the message with SeqNbr=1 before receiving the SeqNbr=999,999,999. How to know whether this is a “new” message, or a duplicate of an (very) “old” one? SeqNbr=1 could also be delayed too, so receiver gets SeqNbr=2 or 3 instead. We could rely on a special “wrap-around” notification message to receiver, but: This message could be lost or delayed too, thus paralyzing the sender. Even if sender blocks on the Ack from such a notification message, this could hurt overall performance as normal messages would not be sent during this time. 1999,999,999 MSNR (Current max seq nbr received) Seq number range Most recent SN-intervals Most ancient SN-intervals Handling of Wrap-around Condition When wrapping-around the Sequence Number Counter, the sender re-generate sequence numbers from 1.

Copyright Fujitsu & Savvion © 2000 Solution: use a quantitative (yet safe) method … We define: A maximum contiguous loss (MCL), which represents the maximum number of contiguous messages that can be “missing” at any time on receiver side (i.e. delayed or lost). e.g. MCL = 1000 assumes that no more than 1000 contiguous messages are permitted to be “missing” at any time on receiver side, between two received messages. If that was about to be the case, the receiver would notify an Error to the sender, so that appropriate action be taken, e.g. stop sending until communication problem is solved. In other words, given an MSNR, it is not expected that the next MSNR value jumps to more than MSNR+MCL. Note that it is also the max distance (in nbr of messages) between the upper limit of an SN-interval, and the lower limit of the next one. A finish watermark (FW), which is the highest SeqNbr value (maxSeqNbr) minus the MCL (e.g. SeqNbr 999,999,999 minus 1000). We say we are in a finish condition if MSNR is past the finish watermark.

Copyright Fujitsu & Savvion © 2000 Note 1 : With the MCL, we can assume that before wrapping around the counter, the receiver has received at least one message beyond the FW mark. Extra care may be taken near the finish watermark to make sure the counter does not wrap around unexpectedly, in case of exceptional situation. Note 2 : The case where MCL = 0 is the traditional Sequence Number Method, which assumes all messages should be received in the same order they were sent. (No tolerance for “disorder”.) In such case, there is always only one large SN-interval, no fragmentation, and only the upper bound needs be checked. Note 3 : When incrementing the MSNR, the MCL assumes that the next “new” message may be in the range [MSNR, MSNR+MCL]. So, this range should not be searched for duplicates… this range must then be subtracted from the list of SN-intervals ,998, ,999,999 FW

Copyright Fujitsu & Savvion © 2000 When receiving a reliable normal message M: if (counter(M) <= MCL + MSNR - maxSeqNbr) then //we were in finish-condition and the counter just wrapped-around MSNR = counter(M), new union of SN-intervals = current union minus the interval [1, MSNR+MCL] else if (counter(M) <= MSNR) then //this is a message coming late - out of sequence if (counter(M) belongs to any SN-interval), then [it is a duplicate, ignore it] else new union of SN-intervals = current union + counter(M) endif else // it is a new message with higher Seq Nbr than previous ones MSNR = counter(M), new union of SN-intervals = current union minus interval [MSNR, MSNR +MCL] minus interval [1, MSNR+ MCL - maxSeqNbr] //in case of wrap around plus MSNR endif General un-ordered sequence number method algorithm with handling of counter wrap-around :

Copyright Fujitsu & Savvion © 2000 The un-ordered sequence number method algorithm takes advantage of the fact that the disorder of Sequence Numbers on the reception side, is local in time and space: After a reasonable amount of time, most old messages have arrived, thanks to re-send. Thus, the oldest the intervals the larger they are. The fragmentation of intervals will remain localised near the most recent sequence numbers. The overall number of intervals (of which depends the speed of the duplicate check) is expected to remain stable and manageable, no matter how big the max seq number (and in any case, much faster to use than a comparable database of message Ids). If some “holes” (missing messages) persist for a long time, the receiver could send some error message suggesting to the sender to either re-send, or reuse the missing numbers for new messages.