maxProcessing config parameter in ns instead of ms
See original GitHub issueIssue Description
When running CometD 7.0.5 in production, CometD was continually increasing its memory usage. From a heap dump, it appears that much of the memory usage was due to old ServerSessionImpl instances holding references to a large number of ServerMessageImpl instances and their related data. I thought that CometD would use the timeout
configuration parameter to remove old ServerSessionImpl instances, but it appears that this is not the case. The CometD documentation specifies a maxProcessing
setting at https://docs.cometd.org/current/reference/#_java_server_configuration which seems like it would resolve this apparent memory leak: “The maximum period of time, in milliseconds, that the server waits for the processing of a message before considering the session invalid and removing it.”
However, the actual implementation adds the value from maxProcessing setting to a time which is measured in nanoseconds: https://github.com/cometd/cometd/blob/d4536b9c4233e318cf40e1dc7e3cada9f92cfa33/cometd-java/cometd-java-server/cometd-java-server-common/src/main/java/org/cometd/server/ServerSessionImpl.java#L159 https://github.com/cometd/cometd/blob/d4536b9c4233e318cf40e1dc7e3cada9f92cfa33/cometd-java/cometd-java-server/cometd-java-server-common/src/main/java/org/cometd/server/BayeuxServerImpl.java#L1298 As a result the maxProcessing setting would also need to be defined in nanoseconds, not milliseconds as specified in the documentation.
Although ServerSessionImpl uses a long
for the _maxProcessing
variable, setting the maxProcessing config setting larger than 2^31 nanoseconds causes a NumberFormatException which causes CometD to not work at all.
Tested with:
- CometD 7.0.5
- Jetty 11.0.7
- openjdk 11.0.13
How to reproduce:
Set maxProcessing configuration setting to 2100000000 and verify that sessions are purged every 2.1 seconds:
<init-param>
<param-name>maxProcessing</param-name>
<param-value>2100000000</param-value>
</init-param>
Set maxProcessing configuration setting to 2200000000 and verify that clients are unable to connect at all:
<init-param>
<param-name>maxProcessing</param-name>
<param-value>2200000000</param-value>
</init-param>
Issue Analytics
- State:
- Created a year ago
- Comments:14 (7 by maintainers)
We are seeing similar issues after upgrading from 5.0.2 to 5.0.10. The symptoms are similar: after running a server for several days some ServerSessionImpl are not removed. It doesn’t happen to all sessions though. When we stop new incoming connections to the server the amount of server sessions doesn’t drop to zero even after many hours.
Our current way of coping with this is to set a max queue option so that we can clean at least some sessions.
Configuration:
addConvertor
(that is something I forgot to change)Clients run cometd 5.0.10 from npm.
We couldn’t find the root cause and couldn’t replicate the issue ourselves but we have an assumption: clients initiate a connection but then a page reload happens hence leaving session in an unfinished setup. There could be several consequent page reloads.
An example of a session that hit max queue limit despite us setting maxInterval (notice that lastConnected is null)
We will be interested to test the latest fixes once they are released to maven central.
In order to test the fixes I would also need a new version to be released to Maven Central.