Failure creating a subscription on a (fairly) new connection

I’m encountering an exception when creating a subscription on a new connection. The new connection has only had a few messages passed through it based on the message stats. I’m working on isolating a use case and will add to this issue if successful.

The goal here is to have an application receive a message, and from within the callback create a new connection and subscribe. The connection used here and the attempt to subscribe were created/called from within the callback of a different connection.

            System.out.println("setupMessageHandling, conn="+ c);
            d = c.createDispatcher(this);
            d.subscribe("control.*");
            c.flush(Duration.ofSeconds(2));

What’s odd is that that if the connection has never been used to publish a message this does not occur.

The result:

NATS @ ~/Dropbox/go/src/github.com/ColinSullivan1/java-examples () $ ./publoss.sh 
setupMessageHandling, conn=io.nats.client.impl.NatsConnection@31221be2
Sending 100000 messages of 128 bytes on foo, server is nats://localhost:4222
Received control message on control.migrate.
newConn=io.nats.client.impl.NatsConnection@4e3e8e61
oldConn=io.nats.client.impl.NatsConnection@31221be2
publishing paused...
publishing paused...
NATS Connection Event: nats: connection closed
Done with migration.
Conn Statistics: 
### Connection ###
Reconnects:                      0
### Reader ###
Messages in:                     0
Bytes in:                        0
### Writer ###
Messages out:                    9
Bytes out:                       1,152
setupMessageHandling, conn=io.nats.client.impl.NatsConnection@4e3e8e61
Migration exception: Output queue is full 0
java.lang.IllegalStateException: Output queue is full 0
	at io.nats.client.impl.MessageQueue.push(MessageQueue.java:122)
	at io.nats.client.impl.MessageQueue.push(MessageQueue.java:108)
	at io.nats.client.impl.NatsConnectionWriter.queue(NatsConnectionWriter.java:191)
	at io.nats.client.impl.NatsConnection.queueOutgoing(NatsConnection.java:1316)
	at io.nats.client.impl.NatsConnection.sendSubscriptionMessage(NatsConnection.java:938)
	at io.nats.client.impl.NatsConnection.createSubscription(NatsConnection.java:908)
	at io.nats.client.impl.NatsDispatcher.subscribeImpl(NatsDispatcher.java:252)
	at io.nats.client.impl.NatsDispatcher.subscribe(NatsDispatcher.java:194)
	at io.nats.java.examples.ControlPlane.setupMessageHandling(ControlPlane.java:32)
	at io.nats.java.examples.ControlPlane.migrate(ControlPlane.java:63)
	at io.nats.java.examples.ControlPlane.onMessage(ControlPlane.java:77)
	at io.nats.client.impl.NatsDispatcher.run(NatsDispatcher.java:98)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

The problem appears to be in adding to an empty LinkedBlockingQueue.

https://github.com/nats-io/nats.java/blob/507490d9941a4dc5fe8333ad1931bd14818a66d2/src/main/java/io/nats/client/impl/MessageQueue.java#L121-L123

Environment

MacOS Catalina

$ java -version java version “1.8.0_231” Java™ SE Runtime Environment (build 1.8.0_231-b11) Java HotSpot™ 64-Bit Server VM (build 25.231-b11, mixed mode)

Any ideas? Thanks!

[ColinSullivan1]

Looks like it was introduced in 2.6.7…

Version 2.6.6
Done
^C=============
Version 2.6.7
Unable to stop reader thread
java.lang.IllegalStateException: Output queue is full 0
	at io.nats.client.impl.MessageQueue.push(MessageQueue.java:107)
	at io.nats.client.impl.NatsConnectionWriter.queue(NatsConnectionWriter.java:188)
	at io.nats.client.impl.NatsConnection.queueOutgoing(NatsConnection.java:1321)
	at io.nats.client.impl.NatsConnection.sendSubscriptionMessage(NatsConnection.java:943)
	at io.nats.client.impl.NatsConnection.createSubscription(NatsConnection.java:913)
	at io.nats.client.impl.NatsDispatcher.subscribeImpl(NatsDispatcher.java:252)
	at io.nats.client.impl.NatsDispatcher.subscribe(NatsDispatcher.java:194)
	at io.nats.java.examples.Example$1.onMessage(Example.java:32)
	at io.nats.client.impl.NatsDispatcher.run(NatsDispatcher.java:98)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Done
^C=============
Version 2.6.8
java.lang.IllegalStateException: Output queue is full 0
	at io.nats.client.impl.MessageQueue.push(MessageQueue.java:122)
	at io.nats.client.impl.MessageQueue.push(MessageQueue.java:108)
	at io.nats.client.impl.NatsConnectionWriter.queue(NatsConnectionWriter.java:191)
	at io.nats.client.impl.NatsConnection.queueOutgoing(NatsConnection.java:1316)
	at io.nats.client.impl.NatsConnection.sendSubscriptionMessage(NatsConnection.java:938)
	at io.nats.client.impl.NatsConnection.createSubscription(NatsConnection.java:908)
	at io.nats.client.impl.NatsDispatcher.subscribeImpl(NatsDispatcher.java:252)
	at io.nats.client.impl.NatsDispatcher.subscribe(NatsDispatcher.java:194)
	at io.nats.java.examples.Example$1.onMessage(Example.java:32)
	at io.nats.client.impl.NatsDispatcher.run(NatsDispatcher.java:98)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Done
^C=============

[ColinSullivan1]

Here’s test code that will reproduce this issue. Maybe I’m doing something wrong… If I comment out the close in the callback, it works. That being said, I don’t see how close on one connection should affect another.

Code

    static public void main(String argv[]) {
        try {
            // setup connection
            Connection nc = Nats.connect("localhost:4222");

            // setup a message handler that creates another connection
            // and closes the original connection.
            MessageHandler mh = new MessageHandler() {
                public void onMessage(Message msg) throws InterruptedException {
                    try {
                        // create a new connection here
                        Connection nc2 = Nats.connect("localhost:4222");
            
                        // close the original conn
                        try {
                          nc.close();
                        } catch (Exception e) {
                            // Doesn't get hit
                            e.printStackTrace();
                        }
            
                        // recreate subscription on the new connection
                        Dispatcher d2 = nc2.createDispatcher(this);
                        d2.subscribe("foo");  // <- exception here
                    } catch (Exception e) {
                        e.printStackTrace();
                    }
                } 
            };

            // setup handler and subscription
            Dispatcher d1 = nc.createDispatcher(mh);
            d1 .subscribe("foo");
            nc.flush(Duration.ofSeconds(2));

            // Invoke the callback from a completely different connection
            Nats.connect("localhost:4222").publish("foo", new byte[64]);

            Thread.sleep(2000);
        } catch (Exception e) {
            e.printStackTrace();
        }
        System.out.println("Done");
    }

Output

bash-3.2$  /Library/Java/JavaVirtualMachines/adoptopenjdk-8.jdk/Contents/Home/bin/java -Dfile.encoding=UTF-8 -cp /var/folders/2x/d1g9l2xd5xl17ktltf8pl7wh0000gn/T/cp_emg817tpcc6tlwijoobi02k26.jar io.nats.java.examples.Example 
java.lang.IllegalStateException: Output queue is full 0
        at io.nats.client.impl.MessageQueue.push(MessageQueue.java:122)
        at io.nats.client.impl.MessageQueue.push(MessageQueue.java:108)
        at io.nats.client.impl.NatsConnectionWriter.queue(NatsConnectionWriter.java:191)
        at io.nats.client.impl.NatsConnection.queueOutgoing(NatsConnection.java:1316)
        at io.nats.client.impl.NatsConnection.sendSubscriptionMessage(NatsConnection.java:938)
        at io.nats.client.impl.NatsConnection.createSubscription(NatsConnection.java:908)
        at io.nats.client.impl.NatsDispatcher.subscribeImpl(NatsDispatcher.java:252)
        at io.nats.client.impl.NatsDispatcher.subscribe(NatsDispatcher.java:194)
        at io.nats.java.examples.Example$1.onMessage(Example.java:32)
        at io.nats.client.impl.NatsDispatcher.run(NatsDispatcher.java:98)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Done