1. P2P with non-centralized Why P2P?
Words written in front
P2P with non-centralized
As EMQ, Redis3.0, MongoDB, Cassandra, etc.
Why P2P?
Under the assumption that there does not exists a 100% reliable service, the DB or bridge would out of service and the system would block by these key service.
Advantage
Do not dependence on any node, make the system always available.
Disadvantage
Too much noise message inside the system.

2. Subscription broadcast
Sub / unsub
Subscription broadcast
Publish
TCP connection
Broker1
T7. PUBLISH
1.Publisher
T8. PUBLISH
T2. PrivateSub
T1. SUBSCRIBE
2.Subscriber
Broker2
Broker5
T2. PrivateSub
T4. Retain Msg
T3. PrivateRetain
T9. PUBLISH
3.Subscriber
T2. PrivateSub
T2. PrivateSub
T3. PrivateRetain
4.Publisher
Broker3
Broker4
Step1. UNSUBSCRIBE
5.Subscriber
While receive client subscriptin: Notification other brokers that the topic has been subscribed by me..
While receive client publish: route the message to the correct brokers which subscribed this topic..
System topic

3. Encapsulate message ClientID/Sub id/Topic … Client1 Client4 Client2
Sub / unsub
Encapsulate message
Publish
TCP connection
Client1
SUBSCRIBE
Client2
SUBSCRIBE
Client1
Client4
Broker1
Broker2
Client2
Client5
Client3
Private
SUB/PUB
ClientID/Sub id/Topic …
Client5
SUBSCRIBE
Client4
PUBLISH
There’s only one logical channel between brokers, in order to reuse this channel for multi clients, the private message has been introduced.
Include client id, subscription id, raw timestamp and etc. which help us make the correct route decision.

4. Basic message flow 1.Publisher 2.Subscriber 1.Subscriber 2.Subscriber
Messages
Context
is_node
Context
is_peer
TCP connection
Context
is_peer
1.Publisher
local
remote
PUBLISH: from client
2.Subscriber
1.Subscriber
CONNECT: version |= MOSQ_NODE_MEET/local time
CONNACK: this is a peer
SUBSCRIBE: for all topics
SUBSCRIBE: from client
SUBACK
2.Subscriber
Retain PUBLISH: from node
PINGREQ
PINGRESP
Private Subscribe: mid/topic/qos/client id/sub id
PUBLISH: unicast to client only
SUBACK
3.Publisher
Private Retain: topic/qos/mid/client id/sub id/ msg rcv time/payload
Traffic cycle avoidance
For internal PUBLISH messages,
send to only client
For SUB/UNSUB messages,
just add subscription.
PUBACK
PUBLISH: from client
PUBLISH: to whoever subscribed
PUBACK

5. Implementation 1 SUB/UNSUB Clean Session QoS
Save the SUB and UNSUB topic as reference counter, make the ref_cnt++ while receive a subscription, and broadcast each fresh subscription, make the ref_cnt-- for each non-duplicated UNSUB(or disconnect with clean session=true). Send the UNSUB while the topic no longger subscribed by any client.
Clean Session
For any client CONNECTION, broadcast SESSION_REQ(include client id and clean session flag) while there is no local client id exists. While receive a SESSION_REQ from remote broker, kick off this client if it exists in local DB, and send a SESSION_RESP with previous subscription and publishes with QoS>0 if clean session=false. Insert the subscriptions and publishes to local DB if receive the SESSION_RESP.
QoS
Send the PUBLISH messages with its original QoS to other brokers, send it with the min QoS between client subscription QoS and publish QoS. As expected in MQTT protocol.

6. Implementation 2 Retain Message Will Message
For any non-sys and non-duplicate client subscription, local broker would generate a subscription ID, then find the retain message inside the cluster. By send a PRIVATE_SUBSCRIBE which include topic, client ID, subscription ID, other broker would give a PRIVATE_RETAIN include the original receive time, client ID and subscription ID if they have the retain message. Local broker save these retain messages temporary, and choose the latest retain message after a frozen window, then send it to the client.
For any subscription from remote broker, just add it in the subscription tree, do not make any broadcast.
For any duplicate local subscription, do not broadcast the PRIVATE_SUBSCRIBE.
Will Message
As normal subscription.

7. Implementation 3 PRIVATE_SUBSCRIBE PRIVATE_RETAIN SESSION_REQ
Fix header( Remain Length) + Packet ID + Topic Length + Topic + clean session | QoS + Original Cliend ID length + Original Client ID + Sub ID (1111 will be use by MQTTv5 as AUTH msg)
PRIVATE_RETAIN
Fix header( Remain Length) + Topic Length + Topic + QoS + [Packet ID] + Original Cliend ID length + Original Client ID + Sub ID + Original Rcv Time Length + Original Retain Rcv Time + Payload
SESSION_REQ
Fix header( Remain Length) + Client ID length + Client ID + clean session
SESSION_RESP
Fix header( Remain Length) + Client ID length + Client ID + last mid + num of subs + sub1(topiclen/topic/qos) + sub2 + … +subN + num of pubs + pub1(topiclen/topic/flag/mid/payloadlen/payload) + pub2 + … + pubN, flag = state | dir | dup | qos

8. Implementation 4 Accelerate intra-cluster message processing
In order to improve the cluster efficiency, do not make any empty/validation/utf-8 checking.
Open a new TCP port for intra-cluster message if needed.
Set clean session to false for each brokers.
To be optimized
Send multi subscription in one PRIVATE_SUBSCRIBE and PRIVATE_RETAIN.

9. Implementation 5 Auto reconnect Crash detection Subscription recovery
Make a non-blocking TCP connect with other broker according to mosquitto.conf, check the connect result 3 seconds later, send MQTT CONNECTION if success, or try to reconnect some seconds(2/4/8/16/32/…) later according the errno of connect(EINPROGRESS/EHOSTUNREACH/ECONNREFUSED/…).
Besides, in order to accelerate cluster setup, connect immediately once receive a MQTT CONNECT from other broker.
Crash detection
For remote process crash, Kernel would close the socket,
For remote OS crash or network issue, check PINGRESP after half of the cluster keepalive time(e.g., 10s) PINGREQ sent.
Subscription recovery
Save each client subscription, and send them all to remote broker after MQTT CONNECT.

10. Implementation 6 Timer optimization Deployment
Use a min heap to manager all the remote brokers and clients, as the key last_msg_in + keepalive time.
Roll polling each remote broker’s health status at the beginning of each while loop.
Use epoll, set timeout with the gap between current and next timer.
Deployment
One instance per core under multi-core machine, listen with different TCP port.