nats.aio.errors.NatsError: nats: empty response from server when expecting INFO message

Hi

I am trying to understand why my nats setup is not working like it should.

Setup

NATS server

I am using the helm chart with image: nats:2.1.7-alpine3.11

values.yaml:

nats:
  image: nats:2.1.7-alpine3.11
  pullPolicy: IfNotPresent

  externalAccess: false
  advertise: true

  serviceAccount: "nats-server"

  connectRetries: 30

  pingInterval:

  limits:
    maxConnections:
    maxSubscriptions:
    maxControlLine: 512
    maxPayload: "1000000000"

    writeDeadline: "2s"
    maxPending:
    maxPings:
    lameDuckDuration:

  logging:
    debug: true
    trace: true
    logtime:
    connectErrorReports:
    reconnectErrorReports:

  tls:
    url: <DUMMYDATA>
    issuerName: letsencrypt-prod
    secret:
      name: nats-client-tls
    cert: "tls.crt"
    key: "tls.key"

nameOverride: ""
imagePullSecrets: []

securityContext: null
affinity: {}
podAnnotations: {}

nodeSelector:
  doks.digitalocean.com/node-pool: staging

cluster:
  enabled: true
  replicas: 3

leafnodes:
  enabled: false

gateway:
  enabled: false
  name: 'default'

bootconfig:
  image: connecteverything/nats-boot-config:0.5.2
  pullPolicy: IfNotPresent

reloader:
  enabled: true
  image: connecteverything/nats-server-config-reloader:0.6.0
  pullPolicy: IfNotPresent

exporter:
  enabled: true
  image: synadia/prometheus-nats-exporter:0.5.0
  pullPolicy: IfNotPresent

auth:
  enabled: true

  operatorjwt:
    configMap:
      name: operator-jwt-nats
      key: operator.jwt
      data: "DUMMYDATA"

  systemAccount: DUMMYDATA

  resolver:
    type: URL
    type: URL
    url: "http://nats-account-server:9090/jwt/v1/accounts/"

NATS is hosted on DigitalOcean and can be accessed via a LoadBalancer.

Nats client

requirements.txt

asyncio-nats-client[nkeys]==0.11.2
fastapi==0.61.0

#! /usr/bin/python

import asyncio
import logging
import ssl
from typing import Any

from fastapi import FastAPI
from nats.aio.client import Client as NATS
from pydantic import BaseModel

HOST_NATS = ""
NATS_CREDENTIALS_FILE = ""


class NatsClient(BaseModel):
    nc: Any = None

    @property
    def connected(self) -> bool:
        return self.nc.is_connected

    async def error_cb(self, e):
        logging.error("NATS Error:", e)

    async def closed_cb(self):
        logging.error("NATS CLOSED")

    async def reconnected_cb(self):
        logging.error("NATS RECONNECTED")

    async def setup(self) -> None:
        self.nc: NATS = NATS()

        context = ssl.create_default_context()
        await self.nc.connect(
            HOST_NATS,
            allow_reconnect=True,
            tls=context,
            user_credentials=NATS_CREDENTIALS_FILE,
            error_cb=self.error_cb,
            closed_cb=self.closed_cb,
            reconnected_cb=self.reconnected_cb,
            verbose=True,
            ping_interval=5,
        )

    async def teardown(self):

        await self.nc.close()


def create_app() -> "FastAPI":
    app = FastAPI()

    app.state.nats_client = NatsClient()

    @app.on_event("startup")
    async def setup() -> None:
        app.state.loop = asyncio.get_event_loop()
        await app.state.nats_client.setup()

    @app.on_event("shutdown")
    async def teardown() -> None:
        await app.state.nats_client.teardown()


app = create_app()

Problem

The problem I am having is that nats clients are not reconnecting properly and outputting these errors.

root 2020-09-26 13:31:43 ERROR (__init__-error_cb:23) >> NATS ERROR: 
root 2020-09-26 13:31:43 ERROR (__init__-error_cb:23) >> NATS ERROR: 
root 2020-09-26 13:31:43 ERROR (__init__-error_cb:23) >> NATS ERROR: nats: empty response from server when expecting INFO message
root 2020-09-26 13:31:43 ERROR (__init__-error_cb:23) >> NATS ERROR: nats: empty response from server when expecting INFO message
root 2020-09-26 13:31:43 ERROR (__init__-closed_cb:26) >> NATS CLOSED
root 2020-09-26 13:31:43 ERROR (__init__-closed_cb:26) >> NATS CLOSED

Is there something I do wrong here?

Greetings

Sebastiaan

[sevaho]

@wallyqs thank you very much, I’ve upgraded the server and updated the clients. I’ll wait until end of week to close this issue if you don’t mind just to be sure it works as expected.

greetings

sevaho

[sevaho]

Hi @wallyqs I don’t have the problem anymore so it looks like it worked 😃.