Connection issues in FT mode

Hi all 👋

I’m having an issue with the clients ability to establish an initial connection when running in fault tolerance (FT) mode within a k8s cluster.

Behavior

• Run Node.js server, nats client connects to FT leader
• Delete leader pod, client re-connects = expected/perfect
• Restart Node.js server

Result

• Client connection is closed on startup.
• error event fires with the following message:

{
    "name": "NatsError",
    "message": "stan: connect request timeout",
    "code": "stan: connect request timeout",
    "chainedError": {
        "name": "NatsError",
        "message": "The request timed out for subscription id: -1",
        "code": "REQ_TIMEOUT"
    }
}

• close event fires, process.exit() is called, server restarts over and over again repeating the above resulted steps until a connection can be established.

Expected Result Client connects first time without having to restart multiple times until a connection is established.

Nats Streaming Version Docker Image nats-streaming:latest Embedded Nats Server

Client Version latest (v0.3.3-0)

I’ve been combing through the docs, github issues, as well as experimenting with various settings on both the deployment and the client. Any ideas on how to solve this?

Thank you in advance for your help.

[kozlovic]

Interesting, I didn’t see that in the docker documentation. Good to know thank you for that.

That’s because this is more of a NATS Server configuration. NATS Streaming does not repeat all config that are available to NATS Server. Sorry about that.

cluster_advertise - Advertise the the available nats streaming pods from within the k8s cluster. This will allow other pods using the stan.js client to connect the Stan leader without the need for list of Stan pod ip’s when configuration the client.

Not really. Again, this is low-level NATS Server. What that does is that if a client connect to 1 server that is clustered (here server means either NATS Server or NATS Streaming Server in your case since you run the embedded version), that server will return to the client library the list of URLs of the other servers, so that if the client were to be disconnected, it could try to reconnect to any of the URLs. Once you have a cluster of servers, it doesn’t matter really which server a client connects to, servers will route the traffic appropriately. For sure, if a client connects directly to the Streaming active server, there is a bit of an advantage in reducing the number of hop the messages have to go through.

[aricart]

@demelvin current clients have started to support bare host:port for the URL - this makes sense (with the exception that url option is a misnomer). This may make it simpler and prevent that type of error.