Slow data channels with big batch downloadSee original GitHub issue
Hi, I’ve found a CometD issue triggered by the combination of two factors:
- Slow network infrastructure / connection
- High volumes of subscribed data in a single batch
If the HTTP channel takes more than the configured client-side
maxNetworkDelay time to download a batch of data from the server, it can happen that the client code drops the connection with “Message Timeout” error message. (I see that
/meta/connect messages will wait more, and precisely (
maxNetworkDelay + timeout), but the channel drop can be caused even by other
/subscribe messages performed at the beginning of the batch subscription).
What is happening is that the websocket incoming channel will be clogged by a big amount of messages coming from the snapshot of the subscriptions. This causes the responses to
/meta/connect and to other
/subscription messages to be put in queue, reaching the client potentially after the
maxNetworkDelay time, causing the premature channel closure.
I understand that the very reason of such
maxNetworkDelay flag is to exactly cover this scenario.
However, since I cannot predict the amount of the data that should be downloaded in a batch, I’m forced to put there a high period to cover the worst case scenario (something like 60 seconds in some cases).
This however can hide other issues, in which some message response are really delayed too much when the system load is low (for instance after the batch download is finished).
I was wondering if it would be possible to implement a different logic to let the client to not drop the connection in case of a steady incoming data flow. I was trying to do some experiment in that direction, but I’ve found that in this case, it is the server that will terminate the session: the client will not be able to perform a new
/meta/channel request after the server ‘closed’ the previous request.
Is there some additional flag that can be used to cover this scenario, without the need to use huge values of
Perhaps, I’m just guessing, a good solution could be a sort of ‘heartbeat’ message that the client regularly send to the server, but not related to the
/meta/connect traffic, that can inform the server about the client download progress in case of slow channel/big amount of data. When the server receives the periodic hearbeat message (that can contains the message.id of the last message processed), it could adapt the session timeouts to take in account the global end-to-end delay.
I think the AckExtensions is not useful in this case, since it only transport information on the
What do you think?
- Created 2 years ago
- Comments:7 (5 by maintainers)
Top GitHub Comments
@lmartorella, after some discussion with colleagues, there may be a better solution - using an extension to dynamically increase the
maxNetworkDelaywithout having to change the CometD internals.
I’ll try to code one in the next days and you can try it out.