Database connection timeouts after imposing some load on the node

Corda version 1.0

What I’m doing: Starting my custom cash payment flow 80 times, without any delay (using a 500ms delay will make this go away). My custom cash payment flow is basically copied from the standard net.corda.finance.flows.CashPaymentFlow except that it adds one more output state (not Cash) that is a QueryableState.

Now I just verified that the same thing happens when invoking the original CashPaymentFlow

What happens: Very reliably, the node throws an exception that it cannot connect to database (its internal H2 one) and then it tries to die (and fails to quit anyway). The H2 database is still accessible with external tools on the URL printed on node start up.

E 13:05:18+0200 [main] messaging.NodeMessagingClient.deliver - Caught exception whilst executing message handler for platform.session.0
 java.util.concurrent.ExecutionException: java.sql.SQLTransientConnectionException: HikariPool-1 - Connection is not available, request timed out after 30000ms.
	at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) ~[?:1.8.0_144]
	at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895) ~[?:1.8.0_144]
	at net.corda.node.utilities.AffinityExecutor$DefaultImpls.fetchFrom(AffinityExecutor.kt:44) ~[corda-node-1.0.0.jar:?]
	at net.corda.node.utilities.AffinityExecutor$ServiceAffinityExecutor.fetchFrom(AffinityExecutor.kt:56) ~[corda-node-1.0.0.jar:?]
	at net.corda.node.services.messaging.NodeMessagingClient.deliver(NodeMessagingClient.kt:420) [corda-node-1.0.0.jar:?]
	at net.corda.node.services.messaging.NodeMessagingClient.processMessage(NodeMessagingClient.kt:310) [corda-node-1.0.0.jar:?]
	at net.corda.node.services.messaging.NodeMessagingClient.runPostNetworkMap(NodeMessagingClient.kt:352) [corda-node-1.0.0.jar:?]
	at net.corda.node.services.messaging.NodeMessagingClient.run(NodeMessagingClient.kt:368) [corda-node-1.0.0.jar:?]
	at net.corda.node.internal.Node.run(Node.kt:358) [corda-node-1.0.0.jar:?]
	at net.corda.node.internal.NodeStartup.startNode(NodeStartup.kt:115) [corda-node-1.0.0.jar:?]
	at net.corda.node.internal.NodeStartup.run(NodeStartup.kt:74) [corda-node-1.0.0.jar:?]
	at net.corda.node.Corda.main(Corda.kt:11) [corda-node-1.0.0.jar:?]
Caused by: java.sql.SQLTransientConnectionException: HikariPool-1 - Connection is not available, request timed out after 30000ms.
	at com.zaxxer.hikari.pool.HikariPool.createTimeoutException(HikariPool.java:548) ~[HikariCP-2.5.1.jar:?]
	at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:186) ~[HikariCP-2.5.1.jar:?]
	at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:145) ~[HikariCP-2.5.1.jar:?]
	at com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:83) ~[HikariCP-2.5.1.jar:?]
	at net.corda.node.utilities.DatabaseTransaction$connection$2.invoke(DatabaseTransactionManager.kt:19) ~[corda-node-1.0.0.jar:?]
	at net.corda.node.utilities.DatabaseTransaction$connection$2.invoke(DatabaseTransactionManager.kt:12) ~[corda-node-1.0.0.jar:?]
	at kotlin.UnsafeLazyImpl.getValue(Lazy.kt:153) ~[kotlin-stdlib-1.1.4.jar:1.1.4]
	at net.corda.node.utilities.DatabaseTransaction.getConnection(DatabaseTransactionManager.kt) ~[corda-node-1.0.0.jar:?]
	at net.corda.node.utilities.DatabaseTransaction.close(DatabaseTransactionManager.kt:57) ~[corda-node-1.0.0.jar:?]
	at net.corda.node.utilities.CordaPersistence.inTopLevelTransaction(CordaPersistence.kt:100) ~[corda-node-1.0.0.jar:?]
	at net.corda.node.utilities.CordaPersistence.transaction(CordaPersistence.kt:75) ~[corda-node-1.0.0.jar:?]
	at net.corda.node.utilities.CordaPersistence.transaction(CordaPersistence.kt:65) ~[corda-node-1.0.0.jar:?]
	at net.corda.node.services.messaging.NodeMessagingClient$deliver$1.invoke(NodeMessagingClient.kt:421) ~[corda-node-1.0.0.jar:?]
	at net.corda.node.services.messaging.NodeMessagingClient$deliver$1.invoke(NodeMessagingClient.kt:76) ~[corda-node-1.0.0.jar:?]
	at net.corda.node.utilities.AffinityExecutor$fetchFrom$1.get(AffinityExecutor.kt:44) ~[corda-node-1.0.0.jar:?]
	at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1590) ~[?:1.8.0_144]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_144]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_144]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) ~[?:1.8.0_144]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) ~[?:1.8.0_144]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_144]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_144]
	at net.corda.node.utilities.AffinityExecutor$ServiceAffinityExecutor$1$thread$1.run(AffinityExecutor.kt:69) ~[corda-node-1.0.0.jar:?]

log file node-Clymene.local.log

[joeldudley]

@rampatra It’s an interval variable - see https://github.com/corda/corda/commit/3fdc69e54199b9eedddc82b597c95c73870cc998. I don’t believe you need to modify it to fix the issue.

[joeldudley]

@rampatra This is from 2017, so I assume it’ll be in all the Corda 4 releases.