Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

RPC deadlocks after a node restart

See original GitHub issue

With @Quiark, we observed this issue with Corda 3.1 and prior versions: when one is doing some operation with RPC (e.g. starting a flow) and at that time, a node (that a particular RPC Client is connected to) shuts down and then starts up again, RPCOps is stuck. There’s an exception thrown on one of the threads, something like:

E 12:23:05+0800 [Thread-1 (ActiveMQ-client-global-threads)] DefaultPromise.rejectedExecution.error - Failed to submit a listener notification task. Event loop shut down? {}
 java.util.concurrent.RejectedExecutionException: event executor terminated
    at io.netty.util.concurrent.SingleThreadEventExecutor.reject( ~[netty-all-4.1.9.Final.jar:4.1.9.Final]
    at io.netty.util.concurrent.SingleThreadEventExecutor.offerTask( ~[netty-all-4.1.9.Final.jar:4.1.9.Final]
    at io.netty.util.concurrent.SingleThreadEventExecutor.addTask( ~[netty-all-4.1.9.Final.jar:4.1.9.Final]
    at io.netty.util.concurrent.SingleThreadEventExecutor.execute( ~[netty-all-4.1.9.Final.jar:4.1.9.Final]
    at io.netty.util.concurrent.DefaultPromise.safeExecute( [netty-all-4.1.9.Final.jar:4.1.9.Final]
    at io.netty.util.concurrent.DefaultPromise.notifyListeners( [netty-all-4.1.9.Final.jar:4.1.9.Final]
    at io.netty.util.concurrent.DefaultPromise.setFailure( [netty-all-4.1.9.Final.jar:4.1.9.Final]
    at [netty-all-4.1.9.Final.jar:4.1.9.Final]
    at [netty-all-4.1.9.Final.jar:4.1.9.Final]
    at [netty-all-4.1.9.Final.jar:4.1.9.Final]
    at [netty-all-4.1.9.Final.jar:4.1.9.Final]
    at [netty-all-4.1.9.Final.jar:4.1.9.Final]
    at [netty-all-4.1.9.Final.jar:4.1.9.Final]
    at org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnection.closeSSLAndChannel( [artemis-core-client-2.2.0.jar:2.2.0]
    at org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnection.close( [artemis-core-client-2.2.0.jar:2.2.0]
    at org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.internalClose( [artemis-core-client-2.2.0.jar:2.2.0]
    at org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.destroy( [artemis-core-client-2.2.0.jar:2.2.0]
    at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect( [artemis-core-client-2.2.0.jar:2.2.0]
    at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure( [artemis-core-client-2.2.0.jar:2.2.0]
    at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure( [artemis-core-client-2.2.0.jar:2.2.0]
    at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.connectionException( [artemis-core-client-2.2.0.jar:2.2.0]
    at org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector$Listener$ [artemis-core-client-2.2.0.jar:2.2.0]
    at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask( [artemis-commons-2.2.0.jar:2.2.0]
    at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask( [artemis-commons-2.2.0.jar:2.2.0]
    at org.apache.activemq.artemis.utils.actors.ProcessorBase$ [artemis-commons-2.2.0.jar:2.2.0]
    at java.util.concurrent.ThreadPoolExecutor.runWorker( [?:1.8.0_144]
    at java.util.concurrent.ThreadPoolExecutor$ [?:1.8.0_144]
    at [?:1.8.0_144]

Issue Analytics

  • State:closed
  • Created 5 years ago
  • Comments:8 (6 by maintainers)

github_iconTop GitHub Comments

bpaunescucommented, Apr 13, 2018

@tomtau thank you for signalling this issue. It’s caused by Artemis not being too nice about non-durable queues and messages during failover. Specifically, messages don’t get re-sent if failover takes too long. Furthermore, there are RPCs that can’t really handle connection loss(the ones that return observables and send updates).

There was a pull request to address these issues. I believe it was done after 3.x was released. The behaviour at the moment is to throw an RPCException for any RPCs called during failover, and to also throw and clean-up ongoing RPCs. This will ensure no hanging in case server is unreachable. The above exception is thrown by the Artemis thread that handles reconnection. We’ll try to find a way to deal with it, but it doesn’t indicate anything wrong, really.

bpaunescucommented, Sep 10, 2018

Issue fixed and will be included in 3.3

Read more comments on GitHub >

github_iconTop Results From Across the Web

RPC deadlocks after a node restart - - Bountysource
With @Quiark, we observed this issue with Corda 3.1 and prior versions: when one is doing some operation with RPC (e.g. starting a...
Read more >
After logAssertFailed, restart gpfs, node showed as arbitrating, could not join back cluster. Cluster manager node have deadlock messages.
Read more >
How to avoid MySQL 'Deadlock found when trying to get lock
One easy trick that can help with most deadlocks is sorting the operations in a specific order. You get a deadlock when two...
Read more >
system node reboot - Product documentation - NetApp
The system node reboot command restarts a node. ... failover partner is not allowed to take over for the node when the node...
Read more >
NFSv4 server restarts causes long pause in NFS client when ...
During the grace period, the server must reject READ and WRITE operations and non-reclaim locking requests (i.e., other LOCK and OPEN operations) ...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Post

No results found

github_iconTop Related Hashnode Post

No results found