MQTT QoS2 Considerations Konstantin Dotchkoff

Challenges associated with implementing QoS 2 in large scale distributed systems Replication of QoS 2 messages and delivery state to other nodes is very expensive in terms of latency increases the network traffic impacts the overall system availability Network partitioning [split brain] invalidates QoS 2 behavior across nodes in different partitions Distributed systems typically designed for availability and resiliency through eventual consistency Common alternatives are use of idempotent operations E.g., for systems that exchange state changes “at least once” semantics is sufficient (assuming in order message delivery, unless the messages are commutative) de-duplication at the receiver

Specifics of the Azure Implementation Largely distributed environment with 64 or more nodes (i.e. MQTT Server side) – running clusters with 128 nodes is common Nodes may enter or leave the cluster at any time Clients may migrate between nodes Server to Client communication: PUBREL message state needs to be replicated across all nodes Received PUBCOMP message state needs to be replicated across all nodes Client to Server communication: State of received messages needs to be replicated across all nodes (or a messages in-flight list for all nodes needs to be managed centrally with concurrent access by all nodes) Received PUBREL messages need to be replicated across all nodes