Possible integer overflow in NetworkPublication

[ivanbogo]

We believe we've encountered a integer overflow bug in NetworkPublication#sendData, which can cause the publication to be stuck in the DRAINING state.

Details on our setup below.

We run a AeronCluster based service which publishes identical output from several replicas. The output is to an exclusive publication on a multicast
channel where each replica is configured to use the same session-id, stream-id and initial position.
The stream is also recorded by a local archive so late joiners can catch up using ReplayMerge.

We recently encountered an issue, whereby if a replica is >2GB behind the others (can happen on restart), it gets stuck and is not able to publish
to the live stream. Additionally, even when we stop the replica process, the media driver keeps the publication alive and does not
allow another instance to create the exclusive publication (because the conflicting publication is never reclaimed).

We believe the culprit is an integer overflow in NetworkPublication#sendData.
When the replica is far behind, the following code can cause an overflow:

    final int availableWindow = (int)(senderLimit.get() - senderPosition);

As a work around we start by publishing to a recorded ipc publication until close to the live stream position, and then switch to the multicast publication.

As far as I can tell, our setup is supported, but would appreciate guidance if there is something wrong:
(https://github.com/aeron-io/aeron/wiki/Transport-Protocol-Specification)

Note: Multiple Senders sending redundant data is supported. Can be as simple as having each use the same Session ID, Stream ID, Term ID, and term offset.

We managed to reproduce on both Linux and Windows and are using aeron 1.48.6.

[vyazelenko]

@ivanbogo If there are remote subscribers the sender limit is set based on the sender position plus the receiver window which is capped at max 1/2 the term length. The max supported term length in Aeron is 1GB. Therefore the max difference between sender limit and sender position can be at most 500MB (524'288'000 bytes) which fits comfortably in an integer.

If there are no remote subscribers and there are spies and spies simulate connection is set to true (ssc=true) then both the sender limit and sender position are set to max spy position. In which case the difference between the two will be zero.

NetworkPublication can stuck in the DRAINING state when:

1. The end of stream position (pub-pos) is greater then the sender position (snd-pos) and there are external receivers but they don't consume the stream, i.e. they never each the EOS position. Those could be slow/non-polling subscriptions or subscriptions that were closed without polling till the end of the stream.

2. There are active spies which do not poll till the end of the stream.

In both cases the subscribers must reach to the EOS position otherwise NetworkPublication will be stuck.

Can you share AeronStat output for the relevant stream from sender and receivers when NetworkPublication is being stuck?

[ivanbogo]

Hi @vyazelenko,

There are remote subscribers, but they drain the live channel published by the other replicas(see my comment above about: Multiple Senders sending redundant data).

I believe because a restarted replica is using the exact same combination of channel/stream-id/session-id as the other replicas, but is more than 2GB behind the others, the subscriber SM messages invalidate the invariant you mentioned:

the sender limit is set based on the sender position plus the receiver window which is capped at max 1/2 the term length

What I mean is that subscribers are happily draining where the other replicas are publishing, but our fresh replica is not allowed to catchup.

Here the relevant output of AeronStat from where the restarted replica tried to catchup:

------------------------------ media-driver-2
43:                    1 - snd-channel: aeron:udp?endpoint=224.0.0.1:10000|term-length=65536|init-term-id=0|term-id=0|term-offset=0|session-id=99999|linger=[1000000](tel:1000000)|eos=false|ssc=true 0.0.0.0:10000
44:                    1 - snd-local-sockaddr: 43 0.0.0.0:10000
45:               32,768 - pub-pos (sampled): 6 [99999 12345](tel:99999 12345) aeron:udp?endpoint=224.0.0.1:10000|term-length=65536|init-term-id=0|term-id=0|term-offset=0|session-id=99999|linger=[1000000](tel:1000000)|eos=false|ssc=true
46:               32,768 - pub-lmt: 6 [99999 12345](tel:99999 12345) aeron:udp?endpoint=224.0.0.1:10000|term-length=65536|init-term-id=0|term-id=0|term-offset=0|session-id=99999|linger=[1000000](tel:1000000)|eos=false|ssc=true
47:                    0 - snd-pos: 6 [99999 12345](tel:99999 12345) aeron:udp?endpoint=224.0.0.1:10000|term-length=65536|init-term-id=0|term-id=0|term-offset=0|session-id=99999|linger=[1000000](tel:1000000)|eos=false|ssc=true
48:        3,221,258,240 - snd-lmt: 6 [99999 12345](tel:99999 12345) aeron:udp?endpoint=224.0.0.1:10000|term-length=65536|init-term-id=0|term-id=0|term-offset=0|session-id=99999|linger=[1000000](tel:1000000)|eos=false|ssc=true
49:                    0 - snd-bpe: 6 [99999 12345](tel:99999 12345) aeron:udp?endpoint=224.0.0.1:10000|term-length=65536|init-term-id=0|term-id=0|term-offset=0|session-id=99999|linger=[1000000](tel:1000000)|eos=false|ssc=true
50:                    0 - snd-naks-received: 6 [99999 12345](tel:99999 12345) aeron:udp?endpoint=224.0.0.1:10000|term-length=65536|init-term-id=0|term-id=0|term-offset=0|session-id=99999|linger=[1000000](tel:1000000)|eos=false|ssc=true

And this is the same output from a healthy replica on a different media driver:

------------------------------ media-driver-1
44:                    1 - snd-local-sockaddr: 43 0.0.0.0:10000
45:        3,221,233,152 - pub-pos (sampled): 2 [99999 12345](tel:99999 12345) aeron:udp?endpoint=224.0.0.1:10000|term-length=65536|init-term-id=0|term-id=49152|term-offset=0|session-id=99999|linger=[1000000](tel:1000000)|eos=false|ssc=true
46:        3,221,265,920 - pub-lmt: 2 [99999 12345](tel:99999 12345) aeron:udp?endpoint=224.0.0.1:10000|term-length=65536|init-term-id=0|term-id=49152|term-offset=0|session-id=99999|linger=[1000000](tel:1000000)|eos=false|ssc=true
47:        3,221,233,152 - snd-pos: 2 [99999 12345](tel:99999 12345) aeron:udp?endpoint=224.0.0.1:10000|term-length=65536|init-term-id=0|term-id=49152|term-offset=0|session-id=99999|linger=[1000000](tel:1000000)|eos=false|ssc=true
48:        3,221,258,240 - snd-lmt: 2 [99999 12345](tel:99999 12345) aeron:udp?endpoint=224.0.0.1:10000|term-length=65536|init-term-id=0|term-id=49152|term-offset=0|session-id=99999|linger=[1000000](tel:1000000)|eos=false|ssc=true
49:                    0 - snd-bpe: 2 [99999 12345](tel:99999 12345) aeron:udp?endpoint=224.0.0.1:10000|term-length=65536|init-term-id=0|term-id=49152|term-offset=0|session-id=99999|linger=[1000000](tel:1000000)|eos=false|ssc=true
50:                    0 - snd-naks-received: 2 [99999 12345](tel:99999 12345) aeron:udp?endpoint=224.0.0.1:10000|term-length=65536|init-term-id=0|term-id=49152|term-offset=0|session-id=99999|linger=[1000000](tel:1000000)|eos=false|ssc=true

If needed I can share a short reproducer, which starts 3 media drivers and replicates the scenario I am describing.

[ivanbogo]

Just noticed my phone decided to add (tel:) when I pasted the AeronStat output. I will try to edit my comment once I am back at keyboard.

[vyazelenko]

Indeed, in this case the availableWindow will overflow given the difference >3GB.

However, I wonder why does the recovering replica trying to publish anything if it is more than 1/2 term behind live, i.e. those packets will be dropped as underflow anyway? A better solution would be to start publishing only when the Cluster node joined the live Raft stream, i.e. once ClusteredService#onRoleChange is called after ClusteredService#onStart finished executing.

[ivanbogo]

Agreed. Once we discovered the issue we’ve fixed our logic to start by publishing to an archived ipc publication and only switch to the multicast when caught up.

That being said, I wanted to confirm if my expectation about redundant publishers is correct, and also bring to your attention that the overflow can effectively put the publication into the DRAINING state forever.

[vyazelenko]

@ivanbogo I've merged PR #1971 which brings changes to the SM handling. The SMs that are out of bounds are now ignored/rejected, i.e. if the SM position is more that half a term before or after current sender position. As a result such SMs will not be treated as a connection, i.e. no data will be sent to the such subscribers.

In your setup that would mean that the publisher will be connected only to the local archive (due to ssc=true) but won't be sending any data via network unless it catches up with the live stream eventually. In which SM check will be satisfied and real network connection established.

Numeric overflow is fixed as well, although dropping SMs effectively solve this as well.

I wanted to confirm if my expectation about redundant publishers is correct

The assumption is correct but the approach is questionable. You could achieve the same by publishing only from the leader node but recording stream on all three nodes.

[ivanbogo]

@ivanbogo I've merged PR #1971 which brings changes to the SM handling. The SMs that are out of bounds are now ignored/rejected, i.e. if the SM position is more that half a term before or after current sender position. As a result such SMs will not be treated as a connection, i.e. no data will be sent to the such subscribers.

Awesome! I will get it built locally for a test drive. Do you have rough idea of when is next release planned for?

The assumption is correct but the approach is questionable. You could achieve the same by publishing only from the leader node but recording stream on all three nodes.

Thanks for the confirmation.
In an earlier version of the service, I explored publishing to the multicast only from the leader. My understanding is that this leaves a small window where one or more messages can be committed to the raft log before the ClusteredService.onSessionMessage callback gets invoked.
If the leader dies or gets isolated in this window, downstream subscribers will observe a gap.
The tradeoff was to implement some sort of backfill in the subscribers versus having the redundant publication, which can also rely on the retransmit mechanism already built into the protocol.
Incidentally, this gives a very small latency improvement for subscribers which are closer to a follower replica (same switch).

[vyazelenko]

Do you have rough idea of when is next release planned for?

The changes will be released in 1.51.0 due in about a month time.

My understanding is that this leaves a small window where one or more messages can be committed to the raft log before the ClusteredService.onSessionMessage callback gets invoked.

This is not how Cluster works. Leader will append ingress messages to the log creating a total order. In parallel it distributes the log to the follower nodes but limits log consumption based on the quorum commit position, i.e. value computed based on the append position (amount of log stored on disc) sent back by the followers. A node can process messages only if they were received by the majority of nodes, i.e. log consumption is capped by the commit position.

So in case a leader node appended some messages to its local disc and died there could be two possible outcomes:

• The data was replicated to other nodes and thus it will be committed and processed under the new leader.
• The data wasn't sent to other nodes. In which case these messages will be "lost", i.e. when the node comes back online it will join as a follower node and will have to truncate its log beyond the last valid position of the corresponding leaderhip term.

Incidentally, this gives a very small latency improvement for subscribers which are closer to a follower replica (same switch).

Is it really? The biggest cost in Cluster is persistence and achieving quorum for a given position in the log. Everything goes through the leader and it is the first one to advance the commit position and tell other nodes they are allowed to consume further in the log. I would expect leader to have a latency edge over the other nodes.